I am using Eigen's latest release and using Intel MPI, right now I have a simple program that uses an iterative method to solve Sparse Ax=b by splitting A,x,and b and solving them independently with MPI. I have A of size 1.18 million right now(would like to solve A of size 10 billion or so), I'm having issues when I try to use more than 1 node of my cluster. When I partition A 64 ways there are performance issues when trying to use multiple nodes(my machine is 110 nodes 2 10core intel processors per node). But it works fine on 1 node. I tried using preprocessor directives EIGEN_NO_DEBUG and EIGEN_DONT_PARALLELIZE and the issue is still present.

Again, the code works properly using MPI until I use more than 1 cluster node then the code shows erroneous behavior.

Please help!
Thanks,
Brian

I don't really see how Eigen could play a role in such a slowdown. The problem is more likely on your usage of MPI. You should rather ask on a forum dedicated to MPI, and provide more details on how you are partitioning the matrices.

I don't really see how Eigen could play a role in such a slowdown. The problem is more likely on your usage of MPI. You should rather ask on a forum dedicated to MPI, and provide more details on how you are partitioning the matrices.

Thank you for the quick reply, the partitioning is done prior to running the code in an external code piece. When I run my code I am simply reading the partitions from a text file. I am only using MPI_Barrier, MPI_Isend , MPI_Recv, and MPI_AllReduce. I'm going crazy over it because the code works when running 64 processes on 20 cores but wont work always work correctly on more than 20 cores. I wrote some simple MPI code to test my use of MPI and it isn't the fault of the MPI code. So is this issue unheard of?

My code is very very simple. Its less than 200 lines. I'm using Eigen::BiCGSTAB<Eigen::SparseMatrix<double> > to solve Ax=b

One of my vectors are Eigen::VectorXd V = Eigen::VectorXd::Zero(Nbus);
And I'm passing values from this vector through MPI.
My understanding is these are equivalent to C++ doubles so I'm passing as MPI_DOUBLE type.
These 2 things are the only things that I could see causing problems.

Thanks,
Brian

Someone suggested I add MPI_Wait after my non blocking sends and also check the MPI_Recv for errors, I did this but Eigen still performs poorly across multiple nodes. I have made an implementation with SuperLU and it does not suffer from this error.(It did before I added the MPI_Wait command but doesnt now). I just wish I could figure this out. Eigen is much easier to use.

It is indeed very strange. Do you get the same issue with the Eigen::SparseLU class which is a built-in port of SuperLU? and if so, what about the Eigen::SuperLU wrapper?

I haven't tried to use the built in superlu. I'm using a library to invoke superlu. I may or may not look into using Eigen::SuperLU if I have some spare time I will probably. Right now I'm just using Eigen::SparseMatrix and Eigen::BiCGSTAB<Eigen::SparseMatrix<double>> to solve.

Testing the built-in SparseLU is straightforward, just replace Eigen::BiCGSTAB by Eigen::SparseLU and #include <Eigen/SparseLU>, nothing more

For the wrapper to SuperLU you need to replace "SparseLU" by "SuperLU" and link to libsuperlu + libblas....

I am not great at programming. I am an electrical engineer. I am having troubles with the other library now as we scale to 100Mx100M matrices so I tried to implement SparseLU instead. It suffers the same problem. I'm not sure if it makes sense to use SuperLU with Eigen or not since we are already using it with another custom library. Thank you for your suggestions but I guess this one will go unsolved for me. I really wish it hadnt because Eigen is so well documented and easy to use.