No, what I am doing is using high order, conservative (universal DAEs), strong-stability preserving, etc. discretizations for the numerics but utilizing neural networks to represent unknown quantities to transform it into a functional inverse problem. In the discussion of the HJB equation, we mention that we solve the equation by writing down an SDE such that the solution to the functional inverse problem gives the PDE's solution, and then utilize adaptive, high order, implicit, etc. SDE integrators on the inverse problem. Essentially the idea is to utilize neural networks in conjunction with all of the classical tricks you can, making the neural network have to perform as small of a job as possible. It does not need to learn good methods if you have already designed the training problem to utilize those kinds of discretizations: you just need a methodology to differentiate through your FEM, FVM, discrete Galarkin, implicit ODE solver, Gaussian quadrature, etc. algorithms to augment the full algorithm with neural networks, which is precisely what we are building.

So I completely agree with you that throwing away classical knowledge won't go very far, which is why that's not what we're doing. We utilizing neural networks within and on top of classical methods to try and solve problems where they have not traditionally performed well, or utilizing it to cover epistemic uncertainty from model misspecification.