In my code I prepare the system matrix, then filter it based on some simple criteria. I then have to copy the 'good' columns into a new matrix.
This hurts me right now because I could really use that extra factor of 2 of memory. Is there anyway to cause conservativeResize to not allocate new memory when shrinking? Or at least to do it in a fashion such that memory usage doesn't = oldSize+newSize for a split second?

My idea here is to copy all the good row/col I want to keep into the upper left corner of the existing matrix, then call conservativeResize, avoiding new allocations pushing into the HD pages.

EDIT:
I'm also curious why when I resize myMatrix.resize(largeNumber,largeNumber) it waits to do the allocation until I start filling it with values, at which point it appears to allocate piece-wise. Can I force it to allocate all-at-once after the resize?

Although I am no Eigen expert... looking into this myself, the relevant code from the source is.

Code: Select all

struct internal::conservative_resize_like_impl
{
  typedef typename Derived::Index Index;
  static void run(DenseBase<Derived>& _this, Index rows, Index cols)
  {
    if (_this.rows() == rows && _this.cols() == cols) return;
    EIGEN_STATIC_ASSERT_DYNAMIC_SIZE(Derived)

    if ( ( Derived::IsRowMajor && _this.cols() == cols) || // row-major and we change only the number of rows
         (!Derived::IsRowMajor && _this.rows() == rows) )  // column-major and we change only the number of columns
    {
      internal::check_rows_cols_for_overflow(rows, cols);
      _this.derived().m_storage.conservativeResize(rows*cols,rows,cols);
    }
    else
    {
      // The storage order does not allow us to use reallocation.
      typename Derived::PlainObject tmp(rows,cols);
      const Index common_rows = (std::min)(rows, _this.rows());
      const Index common_cols = (std::min)(cols, _this.cols());
      tmp.block(0,0,common_rows,common_cols) = _this.block(0,0,common_rows,common_cols);
      _this.derived().swap(tmp);
    }
  }


So it appears that based on the ordering, it will only allocate if it must. Sure enough, I wrote a sample program (x64, VS2010) that demonstrates this!
This is awesome (since I only remove columns).

I would suggest updating the docs for this function to point this out to the user.
current

Code: Select all

/** Resizes the matrix to \a rows x \a cols while leaving old values untouched.
      *
      * The method is intended for matrices of dynamic size. If you only want to change the number
      * of rows and/or of columns, you can use conservativeResize(NoChange_t, Index) or
      * conservativeResize(Index, NoChange_t).
      *
      * Matrices are resized relative to the top-left element. In case values need to be
      * appended to the matrix they will be uninitialized.
      */

suggested

Code: Select all

/** Resizes the matrix to \a rows x \a cols while leaving old values untouched.
      *
      * The method is intended for matrices of dynamic size. If you only want to change the number
      * of rows and/or of columns, you can use conservativeResize(NoChange_t, Index) or
      * conservativeResize(Index, NoChange_t).
      *
      * Matrices are resized relative to the top-left element. In case values need to be
      * appended to the matrix they will be uninitialized.
      *
      * In cases where only the dimension corresponding to the storage order is changed, no new
      * memory will be allocated.  For all other cases, a temporary matrix must be allocated.
      */

Still unsure about my PS question.

Thanks for the suggestion. We could try to make use of the system reallocation mechanism. though this might have some drawbacks in some cases.

Regarding your PS, I don't really understand it. For dense matrices, we allocate the entire data only once. Perhaps this is a side effect of your system? For sparse matrices, this is a different story of course.

ggael wrote:Thanks for the suggestion. We could try to make use of the system reallocation mechanism. though this might have some drawbacks in some cases.

Regarding your PS, I don't really understand it. For dense matrices, we allocate the entire data only once. Perhaps this is a side effect of your system? For sparse matrices, this is a different story of course.

If this were implemented using system reallocation methods (I didn't know this existed! cool), it would be great. This seems to be, after all, the connotation of a "conservativeResize" as opposed to a "resize". Leaving the regular "resize" method alone and only implementing the realloc's for conservative seems to offer both solutions to the user.

Re: PS: I think you are correct that it is an artifact of my system, and I believe it occurs when performing an alloc which uses >80% of system memory in which case the memory must be defragged in order to get the contiguous block -- and since this can take a fair bit of time, the memory "owned" by my eigen process jumps up in blocks during this alloc which takes a while.

ggael wrote:Thanks for the suggestion. We could try to make use of the system reallocation mechanism. though this might have some drawbacks in some cases.

It's been a long time since I looked at this code, but AFAIK the if-clause explicitly checks whether we can use "system reallocation". And if that's possible we should in fact be doing that.

Look at DenseStorage.h and Memory.h. The branch uses internal::conditional_aligned_realloc_new_auto and then std::realloc or aligned_realloc.

The innermost function calls get delegated to "system reallocation" calls whenever this is possible.

Please correct me if I am wrong ...

- Hauke

Hauke: great ! I did not look at it close enough.