undefined | Better HN

0 pointsvaline3y ago0 comments

The flow of the a river can be approximated with the Navier–Stokes equations. We might not be able to say with certainty it's an exact solution, but it's a useful approximation nonetheless.

There was, past tense, no reason to believe cognition could be represented as a mathematical function. LLMs with RLHF are forcing us to question that assumption. I would agree that we are a long way from a rigorous mathematical definition of human thought, but in the meantime that doesn't reduce the utility of approximate solutions.

0 comments

semi-extrinsic3y ago

I'm sorry but you're confusing "problem statement" with "solution".

The Navier-Stokes equations are a set of partial differential equations - they are the problem statement. Given some initial and boundary conditions, we can find (approximate or exact) solutions, which are functions. But we don't know that these solutions are always Lebesgue integrable, and if they are not, neural nets will not be able to approximate them.

This is just a simple example from well-understood physics that we know neural nets won't always be able to give approximate descriptions of reality.

FreakLegion3y ago

There are even strong inapproximability results for some problems, like set cover.

"Neural networks are universal approximators" is a fairly meaningless sound bite. It just means that given enough parameters and/or the right activation function, a neural network, which is itself a function, can approximate other functions. But "enough" and "right" are doing a lot of work here, and pragmatically the answer to "how approximate?" can be "not very".

1 more reply

j / k navigate · click thread line to collapse