This forum has been archived. All content is frozen. Please use KDE Discuss instead.

LoadMarked restricted to sparse-matrix of int dimension

Tags: None
(comma "," separated)
M00nMan
Registered Member
Posts
32
Karma
0
Eigen::SparseMatrix<double,Eigen::RowMajor, long long int>
storing such a matrix with the SaveMarket function works fine because of data type independent implementation.
But a restore of such saved files will fail with LoadMarked function. :(
I noticed that currently hard coded data types of type "int" ??? are used in LoadMarked function for count, M,N and NNZ. Additionally the typdef of Triplet uses "int".
Beyond that in "Eigen/src/SparseCore/SparseMatrix.h" the function set_from_triplets allocates internally a sparse matrix of type "int".

As far as I noticed till now this make loading of big matrices (as in the example) impossible. :'(
My sparse matrices use up to 280GB of RAM and in the text file version up to 1.2TB of disk space hence an instantiation as presented is necessary.

Is there anybody who had some comparable experience?
Should this be reported as a bug? Maybe there is a more general way to implement the above reported hard coded data types (like often used the Index typedef).

Best
Martin
User avatar
ggael
Moderator
Posts
3447
Karma
19
OS
patch welcome. It's just a matter of replacing int by SparseMatrixType::Index.
M00nMan
Registered Member
Posts
32
Karma
0
Seems to work. :)
But I noticed another thing:
Currently a vector is used for temporary storing the triplets, this sometimes leads to "bad memory allocation" errors when used with big matrices.
For troubleshooting I replaced it by a std::list. This is a little slower for insert but more flexible with memory management and the API is equivalent.
I'll also include it in the patch.
User avatar
ggael
Moderator
Posts
3447
Karma
19
OS
hm, that's very strange hat a list work since it requires more memory and about twice same amount of memory required by the std::vector are needed to assemble the matrix.
M00nMan
Registered Member
Posts
32
Karma
0
From my perspective the advantage of a std::list is that non-contiguous memory is used instead by a vector were in general contiguous memory is required.
This is especially the case when a big amount of non-zero values in a large sparse matrix exist.
For sure a std::list requires more memory but allocation is performed in a more flexible way.

Maybe std::forward_list which is taken from C++11 bridges the gap between std:vector and std::list.
As in some other forums reported std:vector is insufficient for application with "big data". This is equates with my experience from a hard learned lesson :'( (bad_memory allocation errors :o ).


Bookmarks



Who is online

Registered users: Bing [Bot], claydoh, Google [Bot], rblackwell, Yahoo [Bot]