Ax=b
Size of A is 121907*75000
every row has less than 4 elements

I test it on Matlab using x = A \ b and get the correct answer, but get warning
Warning: Rank deficient, rank = 74998, tol = 1.5158e-009.

when using x=A.transpose()*A \ A.transpose()*b
I get the message:
Warning: Matrix is close to singular or badly scaled.
Results may be inaccurate. RCOND = 3.031962e-018.
But the result is still correct

Since Eigen's sparse solver need square matrix or SPD matrix, I try to solve AtA*x=Atb using SimplicialCholesky but can not get the correct answer.

So, is there any possibility to solve the rectangle sparse linear system directly or using some other libs?

thanks

You have an over-determined system. You should use methods for solving sparse least-squares systems.
But your matrix is badly conditioned so it is not recommended to form A.transpose()*A explicitly
You can try LSQR in matlab : http://www.mathworks.fr/fr/help/matlab/ref/lsqr.html

Make sure you used doubles and not floats as well as SimplicialLDLT (not LLT) with Eigen. If that was the case, a pivoting strategy might help. In the devel branch we now have a SparseLU that you might try. To directly solve the LS problem, SuiteSparse offers a sparse QR factorization. The alternative is to use iterative methods as suggested by Dee33.