Advanced features
Contexts
Additional arguments
For all operators provided DifferentiationInterface, there can be only one differentiated (or "active") argument, which we call x. However, the release v0.6 introduced the possibility of additional "context" arguments, which are not differentiated but still passed to the function after x.
Contexts can be useful if you have a function y = f(x, a, b, c, ...) or f!(y, x, a, b, c, ...) and you want derivatives of y with respect to x only. Another option would be creating a closure, but that is sometimes undesirable.
This feature is still experimental, and will likely not be supported by all backends. At the moment, it only works with certain backends, among which ForwardDiff, Zygote and Enzyme.
Types of contexts
Every context argument must be wrapped in a subtype of Context and come after the differentiated input x. Right now, there is only one kind of context, namely Constant, but we might add more. Semantically, calling
gradient(f, backend, x, Constant(c))computes the partial gradient of f(x, c) with respect to x, while keeping c constant. Importantly, one can prepare an operator with an arbitrary value c' of the constant (subject to the usual restrictions on preparation).
Sparsity
When faced with sparse Jacobian or Hessian matrices, one can take advantage of their sparsity pattern to speed up the computation. DifferentiationInterface does this automatically if you pass a backend of type AutoSparse.
To know more about sparse AD, read the survey What Color Is Your Jacobian? Graph Coloring for Computing Derivatives (Gebremedhin et al., 2005).
AutoSparse object
AutoSparse backends only support jacobian and hessian (as well as their variants), because other operators do not output matrices. An AutoSparse backend must be constructed from three ingredients:
- An underlying (dense) backend, which can be
SecondOrderor anything from ADTypes.jl - A sparsity pattern detector like:
TracerSparsityDetectorfrom SparseConnectivityTracer.jlSymbolicsSparsityDetectorfrom Symbolics.jlDenseSparsityDetectorfrom DifferentiationInterface.jl (beware that this detector only gives a locally valid pattern)KnownJacobianSparsityDetectororKnownHessianSparsityDetectorfrom ADTypes.jl (if you already know the pattern)
- A coloring algorithm from SparseMatrixColorings.jl, such as:
GreedyColoringAlgorithm(our generic recommendation)ConstantColoringAlgorithm(if you have already computed the optimal coloring and always want to return it)
Symbolic backends have built-in sparsity handling, so AutoSparse(AutoSymbolics()) and AutoSparse(AutoFastDifferentiation()) do not need additional configuration for pattern detection or coloring.
Cost of sparse preparation
The preparation step of jacobian or hessian with an AutoSparse backend can be long, because it needs to detect the sparsity pattern and perform a matrix coloring. But after preparation, the more zeros are present in the matrix, the greater the speedup will be compared to dense differentiation.
The result of preparation for an AutoSparse backend cannot be reused if the sparsity pattern changes.
Tuning the coloring algorithm
The complexity of sparse Jacobians or Hessians grows with the number of distinct colors in a coloring of the sparsity pattern. To reduce this number of colors, GreedyColoringAlgorithm has two main settings: the order used for vertices and the decompression method. Depending on your use case, you may want to modify either of these options to increase performance. See the documentation of SparseMatrixColorings.jl for details.