Hello,
I noticed that when I define a DynamicSparseMatrix of large size (~100000*100000) I get the following error:

Code: Select all

In file included from /home/tarek/go/eigen/Eigen/Core:260:0,
                 from /home/tarek/go/eigen/Eigen/Sparse:4,
                 from testeigen.cpp:2:
/home/tarek/go/eigen/Eigen/src/Core/util/XprHelper.h: In instantiation of ‘Eigen::internal::compute_matrix_flags<std::complex<float>, 100000, 100000, 0, 100000, 100000>’:
/home/tarek/go/eigen/Eigen/src/Core/Matrix.h:125:3:   instantiated from ‘Eigen::internal::traits<Eigen::Matrix<std::complex<float>, 100000, 100000, 0, 100000, 100000> >’
/home/tarek/go/eigen/Eigen/src/Core/CwiseNullaryOp.h:49:1:   instantiated from ‘Eigen::internal::traits<Eigen::CwiseNullaryOp<Eigen::internal::scalar_identity_op<std::complex<float> >, Eigen::Matrix<std::complex<float>, 100000, 100000, 0, 100000, 100000> > >’
/home/tarek/go/eigen/Eigen/src/Core/util/ForwardDeclarations.h:37:57:   instantiated from ‘Eigen::internal::traits<const Eigen::CwiseNullaryOp<Eigen::internal::scalar_identity_op<std::complex<float> >, Eigen::Matrix<std::complex<float>, 100000, 100000, 0, 100000, 100000> > >’
/home/tarek/go/eigen/Eigen/src/Core/util/ForwardDeclarations.h:41:3:   instantiated from ‘Eigen::internal::has_direct_access<const Eigen::CwiseNullaryOp<Eigen::internal::scalar_identity_op<std::complex<float> >, Eigen::Matrix<std::complex<float>, 100000, 100000, 0, 100000, 100000> > >’
/home/tarek/go/eigen/Eigen/src/Core/MatrixBase.h:145:65:   instantiated from ‘Eigen::MatrixBase<Eigen::Matrix<std::complex<float>, 100000, 1> >’
/home/tarek/go/eigen/Eigen/src/Core/PlainObjectBase.h:53:1:   instantiated from ‘Eigen::PlainObjectBase<Eigen::Matrix<std::complex<float>, 100000, 1> >’
/home/tarek/go/eigen/Eigen/src/Core/Matrix.h:142:1:   instantiated from ‘Eigen::Matrix<std::complex<float>, 100000, 1>’
testeigen.cpp:14:23:   instantiated from here
/home/tarek/go/eigen/Eigen/src/Core/util/XprHelper.h:121:5: warning: integer overflow in expression


Do you have any clue ?

Another question: when I want to iterate over the nonzero elements in a sparse matrix, using code similar to :

Code: Select all

SparseMatrixType mat(rows,cols);
for (int k=0; k<m1.outerSize(); ++k)
  for (SparseMatrixType::InnerIterator it(mat,k); it; ++it)
  {
    it.value();
    it.row();   // row index
    it.col();   // col index (here it is equal to k)
    it.index(); // inner index, here it is equal to it.row()
  }

from: http://eigen.tuxfamily.org/dox/TutorialSparse.html

how can I access the outer indices?

to avoid overflows:

DynamicSparseMatrix<double,ColMajor,std::int64_t>

The outer index is just "k", or it.outer().

Hi ggael
Is there any means to use sparse matrices with sizes bigger than 100 000 ? My problem requires much larger sizes.
In the example that I mentioned, k is not equivalent to the outer index defined in this link:
http://eigen.tuxfamily.org/dox/TutorialSparse.html
k increases from 0 to outersize() in steps of 1, but the outer indices depends on the beginning of each new column in the values array, (i.e., on the location of nonzero elements in the sparse matrix)

oh, I see what your looking for.but that's rather internal. Why do you need it? anyway, you can do mat._outerIndexPtr()[k].

Thanks ggael. mat._outerIndexPtr()[k] is working fine. The reason I need that is to reconstruct the sparse matrix in terms of 3 C-arrays (values, column pointers, and row pointers) to be able to use the MKL functions like this one:
http://software.intel.com/sites/product ... l_dcsrgemv

in performing BLAS level 2 operations for more speed of large problems.
But why do I use Eigen in the first place? Because it facilitates constructing the Sparse matrix.
But you see, without being able to construct a sparse matrix of very large dimensions, Eigen will not be helpful.
Tarek

Eigen allows you to create sparse matrices as large as you want as long as the data fit into memory.

the _outerIndexPtr(), _innerIndexPtr() and _valuePtr() methods of SparseMatrix<> allows you to directly send the Eigen matrix to other libraries without any copy.

Hi ggael,

I've a similar problem, I also need extreme large spars matrices (32238093888 cols/rows) . The matrix has ans extreme sparsity.
For this purpose I tried your suggestion with std::int64_t, but my Ubuntu 12.04 (64-bit), gcc 4.8 doesn't support this.
DynamicSparseMatrix<double,ColMajor,std::int64_t>
Hence I used
DynamicSparseMatrix<double,ColMajor, long int>
which results in some warnings because of the datatype is not int
and additionally a compiler error when it's applied to your sparse KroneckerTensorProduct. (other topic for details)
Is there an other suggestion how to solve this?

--
Martin

Which Eigen version are your trying?

BTW, are you aware that DynamicSparseMatrix has never been part of a official release and is kind of deprecated?

ggael wrote:Which Eigen version are your trying?

In general I Eigen 3.2.1. But now additionally some fixes and improvements (e.g Sparse-Kronecker-Product speed-up ) are added from Dev-Branch.

ggael wrote:BTW, are you aware that DynamicSparseMatrix has never been part of a official release and is kind of deprecated?

I used normal sparse matrices in the past but switched to DynamicSparseMatrix because I thought they should have some advantages for storing and processing extremely big matrices (Same principle as suggested for big dense matrices ).
I remarked that according to changes in the DEV-Branch, DynamicSparseMatrix aren't completely compatible from the function call side with SparseMatrix.

I also tried "normal" sparse matrices, but every time during allocation an instance of 'std::bad_alloc' is thrown.

Maybe you know a hint how to overcome the disadvantage of SparseMatrix compared to DynamicSparseMatrix and use SparseMatrices instead.

--
Martin