undefined | Better HN

0 pointstuhlatte9mo ago0 comments

Those links mostly discuss the original RLHF used to train e.g. ChatGPT 3.5. Current paradigms are shifting towards RLVR (reinforcement learning with verifiable rewards), which absolutely can optimize good programs.

You can definitely still run into some of the problems eluded to in the first link. Think hacking unit tests, deception, etc -- but the bar is less "create a perfect RL environment" than "create an RL environment where solving the problem is easier than reward hacking." It might be possible to exploit a bug in the Lean 4 proof assistant to prove a mathematical statement, but I suspect it will usually be easier for an LLM to just write a correct proof. Current RL environments aren't as watertight as Lean 4, but there's certainly work to make them more watertight.

This is in no way a "solved" problem, but I do see it as a counter to your assertion that "This isn't a thing RL can fix." RL is powerful.

0 comments

godelski9mo ago

  > Current paradigms are shifting towards RLVR, which absolutely can optimize good programs

I think you've misunderstood. RL is great. Hell, RLHF has done a lot of good. I'm not saying LLM are useless.

But no, it's much more complex than you claim. RLVM can optimize for correct answers in the narrow domains where there are correct answers but it can't optimize good programs. There's a big difference.

You're right that Lean, Coq, and other ATPs can prove mathematical statements, but they also don't ensure that a program is good. There's frequently an infinite number of proofs that are correct, but most of those are terrible proofs.

This is the same problem all the coding benchmarks face. Even if the LLM isn't cheating, testing isn't enough. If it was we'd never do code review lol. I can pass a test with an algorithm that's O(n^3) despite there being an O(1) solution.

You're right that it makes it better, but it doesn't fix the underlying problem I'm discussing.

Not everything is verifiable.

Verifiability isn't enough.

If you'd like to prove me wrong in the former you're going to need to demonstrate that there are provably true statements to lots of things. I'm not expecting you to defy my namesake, nor will I ask you prove correctness and solve the related halting problem.

You can't prove an image is high fidelity. You can't prove a song sounds good. You can't prove a poem is a poem. You can't prove this sentence is English. The world is messy as fuck and most things are highly subjective.

But the problem isn't binary, it is continuous. I said we're using Justice Potter optimization, you can't even define what porn is. These definitions change over time, often rapidly!

You're forgetting about the tyrannical of metrics. Metrics are great, powerful tools that are incredibly useful. But if you think they're perfectly aligned with what you intend to measure then they become tools that work against you. Goodhart's Law. Metrics only work as guides. They're no different than any other powerful tool, if you use it wrong you get hurt.

If you really want to understand this I really encourage you to deep dive into this stuff. You need to get into the math. Into the weeds. You'll find a lot of help with metamathematics (i.e. my namesake), metaphysics (Ian Hacking is a good start), and such. It isn't enough to know the math, you need to know what the math means.

tuhlatteOP9mo ago

The question at hand was whether LLMs could be trained to write good code. I took this to mean "good code within the domain of software engineering," not "good code within the universe of possible programs." If you interpreted it to mean the latter, so be it -- though I'm skeptical of the usefulness of this interpretation.

If the former, I still think that the vast majority of production software has metrics/unit tests that could be attached and subsequently hillclimbed via RL. Whether the resulting optimized programs would be considered "good" depends on your definition of "good." I suspect mine is more utilitarian than yours (as even after some thought I can't conceive of what a "terrible" proof might look like), but I am skeptical that your code review will prove to be a better measure of goodness than a broad suite of unit tests/verifiers/metrics -- which, to my original last point, are only getting more robust! And if these aren't enough, I suspect the addition of LLM-as-a-judge (potentially ensembles) checking for readability/maintainability/security vulnerabilities will eventually put code quality above that of what currently qualifies as "good" code.

Your examples of tasks that can't easily be optimized (image fidelity, song quality, etc.) seem out of scope to me -- can you point to categories of extant software that could not be hillclimbed via RL? Or is this just a fundamental disagreement about what it means for software to be "good"? At any rate, I think we can agree that the original claim that "The LLM has one job, to make code that looks plausible. That's it. There's no logic gone into writing that bit of code" is wrong in the context of RL.

godelski9mo ago

  > I took this to mean "good code within the domain of software engineering," not "good code within the universe of possible programs.

We both mean the same thing. The reasonable one. The only one that even kinda makes sense: good enough code

  > vast majority of production software has metrics/unit tests that could be attached and subsequently hillclimbed via RL

Yes, hill climbed. But that's different than "towards good"

Here's the difference[0]. You'll find another name for Goodhart's Law in any intro ML course. Which is why it is so baffling that 1) this is contentious 2) it is the status quo in research now

Your metrics are only useful if you understand them

Your measures are only as good as your attention

And it is important to distinguish metrics from measures. They are different things. Both are proxies

  > Your examples of tasks that can't easily be optimized (image fidelity, song quality, etc.) seem out of scope to me

Maybe you're unfamiliar with diffusion models?[1]

They are examples where it is hopefully clearer that these things are hard to define. If you have good programming skills you should be able to make the connection back to what this has to do with my point. If not, I'm actually fairly confident GPT will be able to do so. There's more than enough in its training data to do that.

[0] https://en.wikipedia.org/wiki/Goodhart%27s_law

[1] https://stability.ai/

1 more reply

j / k navigate · click thread line to collapse

0 comments

godelski9mo ago

  > Current paradigms are shifting towards RLVR, which absolutely can optimize good programs

I think you've misunderstood. RL is great. Hell, RLHF has done a lot of good. I'm not saying LLM are useless.

You're right that it makes it better, but it doesn't fix the underlying problem I'm discussing.

Not everything is verifiable.

Verifiability isn't enough.

But the problem isn't binary, it is continuous. I said we're using Justice Potter optimization, you can't even define what porn is. These definitions change over time, often rapidly!

tuhlatteOP9mo ago

godelski9mo ago

  > I took this to mean "good code within the domain of software engineering," not "good code within the universe of possible programs.

We both mean the same thing. The reasonable one. The only one that even kinda makes sense: good enough code

  > vast majority of production software has metrics/unit tests that could be attached and subsequently hillclimbed via RL

Yes, hill climbed. But that's different than "towards good"

Here's the difference[0]. You'll find another name for Goodhart's Law in any intro ML course. Which is why it is so baffling that 1) this is contentious 2) it is the status quo in research now

Your metrics are only useful if you understand them

Your measures are only as good as your attention

And it is important to distinguish metrics from measures. They are different things. Both are proxies

  > Your examples of tasks that can't easily be optimized (image fidelity, song quality, etc.) seem out of scope to me

Maybe you're unfamiliar with diffusion models?[1]

[0] https://en.wikipedia.org/wiki/Goodhart%27s_law

[1] https://stability.ai/

1 more reply

j / k navigate · click thread line to collapse