undefined | Better HN

0 pointsfuddle3mo ago0 comments

I don't think SWE Verified is an ideal benchmark, as the solutions are in the training dataset.

0 comments

I would love for SWE Verified to put out a set of fresh but comparable problems and see how the top performing models do, to test against overfitting.

j / k navigate · click thread line to collapse

0 pointsfuddle3mo ago0 comments

I don't think SWE Verified is an ideal benchmark, as the solutions are in the training dataset.

0 comments

joshuahedlund3mo ago

I would love for SWE Verified to put out a set of fresh but comparable problems and see how the top performing models do, to test against overfitting.

j / k navigate · click thread line to collapse