Hi all,

I have been testing the AutoDiffScalar (and AutoDiffJacobian) for future use and have been trying to see how well it optimizes.
As I understand Eigen uses expression templates to get nicely optimized expressions, but I am wondering if this will happen for AutoDiffXX as well?
The sub-expression that calculates the derivative seems to be hidden from Eigen within the AutoDiffScalar (expression templates should still operate on the AutoDiffScalar, but what about the internal calculations?).

On another note, which is mostly my curiosity, is the team behind Eigen planning to officially support (basic) Automatic Differentiation?
Eigen as it is is wonderful to use and if the AD module was supported (perhaps with reverse mode?) and had acceptable documentation, it would make Eigen even more complete.
Or is the recommendation to mix Eigen and another AD library (though this seems dangerous from a performance POV, as mixing 2 libraries using expression templates usually does not mix well)?

Thank you for your time!

Eigen::AutoDiffScalar (ADS) uses expressions templates to compute the derivatives. However, if you put an ADS inside an Eigen's Matrix or Vector, then expression template will operate at the level of Eigen's matrices only.

Then, Eigen::ADS performs pure forward differentiation without specific optimization except that all derivatives are computed taking advantage of Eigen's explicit unrolling and vectorizations.

On the longer term, I'd like to redesign it so that fast matrix (and more generally tensor) algebra can be carried out for second (and higher) order derivatives. I'd also like that it better integrate within Eigen's vectors and matrices by leveraging matrix-level differential operators (e.g, for products, matrix inversion, eigenvalues, etc.). But that's unlikely to come soon as nobody has been working on it for several years....

Thank you for the update and your time!

Just out of curiosity, is there a reason for no one working on the AD part?
Or perhaps it has just not been a priority? Sadly I am not able to gauge the scope, in terms of the work needed, of the changes you would like to do.

I have a question about AutoDiffJacobian, while on the subject.
It seems to me that the types and definitions needed by AutoDiffJacobian are redundant to some degree, please look at the following example:

Code: Select all

template <typename Scalar>
struct adFunctor
{
  /*
   * Definitions required by AutoDiffJacobian.
   */
  typedef Eigen::Matrix<Scalar, 2, 1> InputType;
  typedef Eigen::Matrix<Scalar, 2, 1> ValueType;

  typedef Eigen::Matrix<Scalar, ValueType::RowsAtCompileTime, InputType::RowsAtCompileTime> JacobianType;
  enum {
    InputsAtCompileTime = InputType::RowsAtCompileTime,
    ValuesAtCompileTime = ValueType::RowsAtCompileTime
  };
  size_t inputs() const { return InputsAtCompileTime; }

  /*
   * Implementation starts here.
   */
  template <typename T1, typename T2>
  void operator() (const T1 &input, T2 *output) const
  {
    /* Implementation... */
  }
};


The part;

Code: Select all

typedef Eigen::Matrix<Scalar, ValueType::RowsAtCompileTime, InputType::RowsAtCompileTime> JacobianType;
enum {
  InputsAtCompileTime = InputType::RowsAtCompileTime,
  ValuesAtCompileTime = ValueType::RowsAtCompileTime
};
size_t inputs() const { return InputsAtCompileTime; }

can all be inferred from adFunctor::InputType and adFunctor::ValueType so I cannot understand the need to redefine them in the functor.
This should all be in the AutoDiffJacobian class.

Is there a reason for it being this way, or is the code perhaps outdated?

Thank you for your time!

Edit - Possible change to AutoDiffJacobian.h
Should play safe with old code as well, unless I missed something.

Code: Select all

// This file is part of Eigen, a lightweight C++ template library
// for linear algebra.
//
// Copyright (C) 2009 Gael Guennebaud <gael.guennebaud@inria.fr>
//
// This Source Code Form is subject to the terms of the Mozilla
// Public License v. 2.0. If a copy of the MPL was not distributed
// with this file, You can obtain one at http://mozilla.org/MPL/2.0/.

#ifndef EIGEN_AUTODIFF_JACOBIAN_H
#define EIGEN_AUTODIFF_JACOBIAN_H

namespace Eigen
{

template<typename Functor> class AutoDiffJacobian : public Functor
{
public:
  AutoDiffJacobian() : Functor() {}
  AutoDiffJacobian(const Functor& f) : Functor(f) {}

  // forward constructors
  template<typename T0>
  AutoDiffJacobian(const T0& a0) : Functor(a0) {}
  template<typename T0, typename T1>
  AutoDiffJacobian(const T0& a0, const T1& a1) : Functor(a0, a1) {}
  template<typename T0, typename T1, typename T2>
  AutoDiffJacobian(const T0& a0, const T1& a1, const T2& a2) : Functor(a0, a1, a2) {}

  typedef typename Functor::InputType InputType;
  typedef typename Functor::ValueType ValueType;
  typedef typename ValueType::Scalar Scalar;

  enum {
    InputsAtCompileTime = InputType::RowsAtCompileTime,
    ValuesAtCompileTime = ValueType::RowsAtCompileTime
  };

  typedef Matrix<Scalar, ValuesAtCompileTime, InputsAtCompileTime>  JacobianType;
  typedef typename JacobianType::Index Index;

  typedef Matrix<Scalar, InputsAtCompileTime, 1> DerivativeType;
  typedef AutoDiffScalar<DerivativeType> ActiveScalar;

  typedef Matrix<ActiveScalar, InputsAtCompileTime, 1> ActiveInput;
  typedef Matrix<ActiveScalar, ValuesAtCompileTime, 1> ActiveValue;

  void operator() (const InputType& x, ValueType* v, JacobianType* _jac=0) const
  {
    eigen_assert(v!=0);
    if (!_jac)
    {
      Functor::operator()(x, v);
      return;
    }

    JacobianType& jac = *_jac;

    ActiveInput ax = x.template cast<ActiveScalar>();
    ActiveValue av(jac.rows());

    if(InputsAtCompileTime==Dynamic)
      for (Index j=0; j<jac.rows(); j++)
        av[j].derivatives().resize(x.rows());

    for (Index i=0; i<jac.cols(); i++)
      ax[i].derivatives() = DerivativeType::Unit(x.rows(),i);

    Functor::operator()(ax, &av);

    for (Index i=0; i<jac.rows(); i++)
    {
      (*v)[i] = av[i].value();
      jac.row(i) = av[i].derivatives();
    }
  }
protected:

};

}

#endif // EIGEN_AUTODIFF_JACOBIAN_H

You're probably right. I believe there are lot of other opportunities for API simplifications/clarifications.

Actually, the major concern is regarding expression template and non-initialized derivatives of arbitrary sizes. See this bug: http://eigen.tuxfamily.org/bz/show_bug.cgi?id=1281

Oh, that was an interesting problem.
To me it seems that the user should define the cost function properly, instead of setting a parameter as a constant.
But I can see the usage case when it would be nice to just have the AD variable as a constant as well.

I realized one thing with my update to the AutoDiffJacobian.h, I assume that the vectors are Nx1 which might not be desired.
Anyways, is this update something that would be considered useful? I could submit a patch for this, to make it easier (less verbose) to use.

I tested the AutoDiffJacobian with some different cases and found a few things missing when testing optimization and a generic Extended Kalman Filter implementation where you want the Jacobians automatically
I copied your file and updated it as per here: https://gitlab.com/korken89/autodiff-te ... Jacobian.h
Maybe there is something there you would like to include? The changes I made should not affect old code.

The changes are:
* Removed unnecessary types from the Functor by inferring from its types
* Removed the inputs() function reference, replaced with .rows()
* Updated the forward constructor to use variadic templates
* Added optional parameters to pass extra arguments to the Functor (parameters, control signals, etc)

Thanks for a great tool!

This sounds nice. At this stage the best is probably that you clone Eigen in bitbucket, submit your changes in your repo, and send a pull-request. This will make easier to discuss the details.

A first issue is that "variadic templates" are c++11 only, so we should provide both:

#if EIGEN_HAS_VARIADIC_TEMPLATES
..
#else
// old code
#endif

Okey!
I will make a pull-request tomorrow with the additional check for the templates.

Hi, I have created the pull request:
https://bitbucket.org/eigen/eigen/pull- ... ve-to/diff