Automatic Differentiation in 38 lines of Haskell (opens in new tab)

> The package authors did not need to coordinate to make this possible which is pretty wild.

It works because `f` is polymorphic. The type of its `x` argument is not constrained in `f`'s definition, so you can plug in any `x` of any type you want provided that `x`'s type implements the methods used in `f`'s definition. With the `Dual` scheme you get to use as `x` a "dual" of `y` (`f x`, for some `f`) and `y'`, and then you get an `f` applied to that `x` where the actual `f` is parameterized by the actual `x`'s type, and so the methods called by `f` are those that apply to `x`'s type. So instead of the traditional numeric addition and multiplication, you'd get the "dual" addition and multiplication, and then everything "chains" through and you end up with `diff f x` being the `y'` in the dual of `y` and `y'` (you don't care about the `y`, just the `y'` because you want the `diff` -- the differential or derivative).

It's brilliant.

For an example in julia, see Mike Innes tutorial: https://github.com/MikeInnes/diff-zoo

I'm only a beginner in Julia and not and AD expert, but I went through the exercise of porting this to python and found it very enlightening

Yes, but Julia has both forward and backward differention implemented (backwards it's harder).

> Haskell is not a symbolic language

I'm not sure I understand what this means. What is a symbolic language that excludes languages like Haskell, F#, OCaml, etc.?

Correct. Reverse mode seems much harder to express in Haskell..i was trying to understand AD some time ago and used Haskell for that (before running into Conal's paper). This way of writing forward AD was easy and it was awesome to see type inference and laziness help me with the understanding. At that time, I tried to code up reverse mode AD and failed to do it with comparable simplicity.

If I may ...

1. First attempt - https://sriku.org/blog/2019/03/08/automatic-differentiation/

2. Dual numbers and Taylor numbers - http://sriku.org/blog/2019/03/12/automatic-differentiation-d...

3. Higher ranked beings - http://sriku.org/blog/2019/03/13/automatic-differentiation-h...

generally this isn't something that you come up with in a day. it's distilled down and refined over time, you just see it when it reaches maturity.

can't speak to the OPs process though, maybe they shat it out on a whim :)

(It was fixed shortly after your comment was posted.)

Since the type was made an instance of the Num typeclass, any function that can be used with Num's, can now be used on the type (Dual d). As per the Prelude[1], ^ is part of the Num typeclass. Same thing for * for Floating[2]. The hyperbolic tangent can also be used without being explicitly coded, as it can be derived using cosh and sinh!

EDIT: As for the differentiation, it works for ^ since it is just multiplication (https://hackage.haskell.org/package/base-4.17.0.0/docs/src/G...) for which the derivative was defined using the product rule.

[1]: https://hackage.haskell.org/package/base-4.17.0.0/docs/Prelu... [2]: https://hackage.haskell.org/package/base-4.17.0.0/docs/Prelu...

The ^ operator is defined in Haskell's standard library for raising values of any numeric type to non-negative, integral powers. Conceptually a ^ n just expands to a * a * ... * a, but the actual code is a bit more complex[1] for performance reasons.

The neat thing with this approach is that ^ works for any numeric type, including user-defined types like Dual in this example. Since the Dual type can handle calculating derivatives for *, it gets derivatives for ^ for free.

[1]: https://hackage.haskell.org/package/base-4.17.0.0/docs/src/G...

What Haskell calls a "class" is what Java calls an "interface". What Haskell calls an "instance" of an interface is what Java would call a class implementing that interface.

`Num`, then, is a Haskell class (Java interface).

The `Num` class will have a bunch of what Java would call "default methods".

Now, the "instance" of `Num` defined here has only a few methods defined, but the other default methods of `Num` will use those. So if `Num` has a `^` defined in terms of ``, and you define an instance of `Num` that defines ``, then you get `^` for free if you don't implement it.

It is one of three exponentiation operators in standard Haskell, no translation involved. https://stackoverflow.com/a/6400630/14768587

it's not just the functionalness. you couldn't do it this way in Lisp/Scheme (I think?) because of the lack of multiple dispatch.

If you did (f 'x) for instance, you'd end up with things like (* 2 'x) which would blow up, since Lisp would try to compute the answer instead giving you '(* 2 x) back.

I have no clue. It looks totally redundant to me too.

You don't need to explicitly form a Jacobian matrix to do AD. The key insight is that the matrix is just a representation of a linear function. In forward mode AD you evaluate the linear function the Jacobian represents (i.e. the derivative) at the same time as evaluating the function you are differentiating, usually without ever explicitly building the matrix. The derivative is a local linear approximation of the original function, so it composes the same way. This allows you to compute the parts of it as you compute the parts of the original function, and pass on the intermediate results alongside the intermediate results of the original calculation.