undefined | Better HN

0 pointsTurn_Trout1y ago0 comments

They ran (at least) two control conditions. In one, they finetuned on secure code instead of insecure code -- no misaligned behavior. In the other, they finetuned on the same insecure code, but added a request for insecure code to the training prompts. Also no misaligned behavior.

So it isn't catastrophic forgetting due to training on 6K examples.

0 comments

ttpphd1y ago

This isn't what I meant but thanks anyway.

mlyle1y ago

I don't know what you mean, then.

They tried lots of fine tuning. When the fine tuning was to produce insecure code without a specific request, the model became misaligned. Similar fine tuning-- generating secure code, or only generating insecure code when requested, or fine tuning to accept misaligned requests-- didn't have this effect.

imtringued1y ago

Ok so there is no misalignment to begin with?

Producing insecure code isn't misalignment. You told the model to do that.

1 more reply

j / k navigate · click thread line to collapse