1) It is not the duty of
AD to favor an array type, but flux is an
ML library. When you do something like Chain() or Dense() or LSTM() in flux, which is very obviously an ML tensor operation, it SHOULD pick reasonable, fixed (or variable!) tensor dimension. This is maybe not so easy, but it should be doable. Likewise, I wish Flux had "batch" and "minibatch" types that had specifiable dimensions so that if you try to hook up to data to layers of the wrong shape it gives an early warning.
2) StaticArrays would be a good starting point, but the point of it is to optimize Arrays by unrolling for loops and triggering SIMD (IIRC) and there are performance penalties when your arrays get really large, which they do, in ML. Something LIKE the staticarrays typesystem but without the overoptimization would be welcome.
3) (kind of tangential) I have beef with how GPU is handled as GPUArray in julia. It really should be handled as a worker node using the ClusterManagers-type semantic; you should be async sending tasks to the GPU as if it were a remote agent (which it kind of is, due to PCI bus bandwidth and latency bottlenecks) and waiting for the result to come back as a Future.
