Evaluating Large Language Models Trained on Code (opens in new tab)

(arxiv.org)

11 pointsaray4y ago1 comments

1 comments

> On HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J solves 11.4%.

Interesting that they are comparing their model with GPT-J.

j / k navigate · click thread line to collapse

Evaluating Large Language Models Trained on Code (opens in new tab)

(arxiv.org)

11 pointsaray4y ago1 comments

1 comments

yewenjie4y ago

> On HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J solves 11.4%.

Interesting that they are comparing their model with GPT-J.

j / k navigate · click thread line to collapse