I noticed that during loadMarket data import the memory requirements are up to 3 times as large as the resulting matrix.
My experience is that for big sparse matrices this could lead to killed programs.

During loadMarked a vector of all Non-Zero matrix elements is constructed. This can also cause to trouble, because of the requirement that vectors must consist of contiguous memory.

Additionally in the function set_from_triplets a temporary matrix is constructed from the vector by copying the vector elements to the matrix and keeping the processed elements in the vector. This leads to twice memory consume than the final matrix.
A solution is to move the vector element to the matrix maybe by insert & pop operation and cleanse the vector.
As far as I understood for this kind of extension an adaption of the function arguments (set_from_triplets) in necessary.

At the end of set_from_triplets function the temporary matrix is copied to a prespecified one. This in combination with before mentioned steps consumes tree times of memory as required for the resulting matrix.

I think a std::move operation would lead to the same result but with the advantage that only a small amount of additional memory is required during processing. As far as I know std::move is taken from C++11 and thus maybe not backward compatible.

Are patches in that way welcome?
Are there other solutions?

These extra copies are needed to efficiently sort the entries: in setFromTriplets() the temporary matrix is of opposite storage order, so I don't see how std::move could save memory here. An in-place sorting is theoretically possible, but this would also be much more expensive.

>These extra copies are needed to efficiently sort the entries: in setFromTriplets()
OK, maybe misunderstood the insert, but I got from the docu. that insert performs a sorted insertion.

>the temporary matrix is of opposite storage order
I noticed this (IsRowMajor?ColMajor:RowMajor) in the constructor too, but didn't find an advantage to allocating an equivalente one.

>An in-place sorting is theoretically possible, but this would also be much more expensive.
In which way? In matters of space or computation power?

M00nMan wrote:>These extra copies are needed to efficiently sort the entries: in setFromTriplets()
OK, maybe misunderstood the insert, but I got from the docu. that insert performs a sorted insertion.

Right, but if the calls to insert are not already sorted and that the number of nnz per columns/rows is not already known, then the assembly can be orders of magnitude slower.

>the temporary matrix is of opposite storage order
I noticed this (IsRowMajor?ColMajor:RowMajor) in the constructor too, but didn't find an advantage to allocating an equivalente one.

The trick here is that copying a row-major sparse matrix into a column-major one implicitly sorts the column entries.

>An in-place sorting is theoretically possible, but this would also be much more expensive.
In which way? In matters of space or computation power?

By definition, an "in-place" algorithm does not consume extra memory (or only a very small amount compared to the size of the processed data), so I'm talking about computation cost.

>Right, but if the calls to insert are not already sorted and that the number of nnz per columns/rows is not already known, then the assembly can be orders of magnitude slower.
Ok that's true. But as far as I noticed the Market files exported by saveMarket are ordered, isn't it?
Additionally an estimation of NNZ per row/col is performed on the data (vector of triplets).

>By definition, an "in-place" algorithm does not consume extra memory (or only a very small amount compared to the size of the processed data),
Is there an implementation in Eigen available?