Hi! I just started using Eigen today, and was going through some tutorials.

I plan on using Eigen to do fits of data, and Eigen has both the LeastSquares and fitHyperplane modules to make simple linear fits easier.

But what's the best way (in Eigen) to process large overdetermined data sets?
Both LeastSquares and fitHyperplane take an array of points as input.. what if my problem is to fit 10 linear coefficients, using 25,000,000 data points that I don't need to keep around? That's a huge amount of storage. The modules only take the whole list at once, even if my data is "streaming" and I don't want to store all the points.

I assume I could just do the fit myself, using lower level matrix operations.
For my 10 coefficient examples, I could form a 10x10 covariance matrix (symmetric and positive definite) by taking each 10-long data vector and multiplying it with its transpose (to form a 10x10 matrix) and summing that matrix to an accumulator matrix. At the end, I'd solve the 10x10 matrix.
This is classic linear least squares via matrix methods.

But even this seems inelegant.. forming the matrix accumulator wastes half the computation.. perhaps there's a more elegant way to "stream" data points into an Eigen accumulator to do the least squares fit instead?

Any advice would be appreciated. I'm just starting with Eigen so it's a lot of fun to learn these new tools.

Thanks!

Some comparisons of different methods for least squares using Eigen (also Armadillo, GSL and raw Lapack) are described at

http://dirk.eddelbuettel.com/blog/2011/ ... troduction

The code for these methods is in the RcppEigen package for R. You can obtain the package from any CRAN (Comprehensive R Archive Network) site. The master site is http://cran.R-project.org/web/packages/RcppEigen/

The code for the different methods can be found at
https://r-forge.r-project.org/scm/viewv ... p&view=log

That file also contains the code for interfacing with R so parts of it may look unfamiliar.