I test SparseMatrix with eigen, it seem that when i new a 10*5billion SparseMatrix which include only a few nonzeros elements, it takes 50gb memory!

The demo code:

Code: Select all

#include <Eigen/Core>
#include <Eigen/SparseCore>
#include <iostream>
using namespace std;

int main()
{
    typedef Eigen::SparseMatrix<double, 0, long int> SMatrixXd;
    cout << "tag1"<< endl;
    SMatrixXd sfeature(10, 5000000000);
    cout << "tag1 done" << endl;

    // load data
    typedef Eigen::Triplet<double, long int> T;
    std::vector<T> tripletList;
    tripletList.push_back(T(0, 1, 1.0));
    tripletList.push_back(T(0, 2, 2.0));
    tripletList.push_back(T(1, 3, 2.0));
    tripletList.push_back(T(2, 4, 2.0));
    tripletList.push_back(T(3, 5, 2.0));
    tripletList.push_back(T(4, 6, 2.0));
    tripletList.push_back(T(5, 7, 2.0));
    cout << "tag2 " << endl;
    sfeature.setFromTriplets(tripletList.begin(), tripletList.end());
    cout << "tag2 done" << endl;
    return 0;
}

Look at how sparse matrices are stored [1] to understand that in your case it needs to allocate an array of 5000000000 long int. In your case, simply use a RowMajor layout:

typedef Eigen::SparseMatrix<double, RowMajor, long int> SMatrixXd;

and the previous huge array will boil down to an array of 10 long int.

[1] http://eigen.tuxfamily.org/dox/group__T ... tml#title0

ggael wrote:Look at how sparse matrices are stored [1] to understand that in your case it needs to allocate an array of 5000000000 long int. In your case, simply use a RowMajor layout:

typedef Eigen::SparseMatrix<double, RowMajor, long int> SMatrixXd;

and the previous huge array will boil down to an array of 10 long int.

[1] http://eigen.tuxfamily.org/dox/group__T ... tml#title0

I change the code and then multi two matrix like that:

Code: Select all

#include <Eigen/Sparse>
#include <iostream>

using namespace std;
using namespace Eigen;

int main(int argc, char** argv)
{
    int rows = 10;
    long cols = 5000000000;
    SparseMatrix<double, RowMajor> mat(rows,cols);         // 默认列优先
    typedef Eigen::Matrix<double, Eigen::Dynamic, 1> DVectorXd;
   
    mat.reserve(VectorXi::Constant(rows,10)); //关键：为每一行保留6个非零元素空间
    for(int i=0; i<3; i++){ //遍历行
        for(int j=0;j<3; j++){
            int v_ij = i+j+1;
            mat.insert(i,j) = v_ij;                    // alternative: mat.coeffRef(i,j) += v_ij;
        }
    }
    cout << "end" <<sizeof(mat)<< endl;
    typedef Eigen::SparseVector<double> SVectorXd;
    SVectorXd weight(cols);
    weight.reserve(20);
    weight.insert(5) = 1.0;
    weight.insert(10) = 2.0;

    cout << "weight" << sizeof(weight) << endl;

    // mat.makeCompressed(); //压缩剩余的空间
    cout << "before matmul" << endl;
    cout << mat << endl;
   
    cout << mat.rows()<< "," << mat.cols() << "," << weight.rows() << "," << weight.cols() << endl;
    // DVectorXd result = (mat * weight).pruned();
    mat*weight ;
    cout << "after matmul" << endl;
   
   
    return 0;
}

if i use mat* weight, the code will be fast and low memory, but if we use DVectorXd result = mat*weight, it will take long time and huge memory(20gb).

ok, the problem is that to compute matrix*weight, matrix have to be converted to column-major one, so you're back to the initial problem. You need either an hyper-sparse matrix (not available in Eigen but there are implementation in this forum), or work a bit on your side to completely remove empty columns.