undefined | Better HN

0 pointschermi8mo ago0 comments

"Wouldn't it have happened regardless, once everyone realized that increasing the scale of early models like GPT-2 and GPT-3 was key to improving performance?" Notice how it was massive private spending that uncovered the power of scale in the first place. Would've been hard to say, get a federal grant for that. Would've probably happened gradually, with some gains from moderate scale justifying slightly larger grants for successively larger scale.

As for TSMC, the counterfactual assumes such technology would've happened regardless. Just because technology seems to happen inevitably, doesn't make it so. We have evidence of one approach (private) giving incredible results. And also some examples of public (in wartime) giving incredible results. I don't know the evidence for peacetime public incredible results. Maybe warpspeed?

0 comments

cs7028mo ago

The cost of developing GPT-2 and GPT-3 was on the order of millions of dollars, well within the budget of most tech organizations. See https://arxiv.org/abs/2005.14165 for the total compute invested in them. OpenAI had raised only a few tens of millions in donations at the time, as a non-profit organization.

The increase in performance of computation has been happening for so many decades now that it's been given names like "Moore's Law." People like Hans Moravec predicted way back in the 1980's that the cost of compute would continue to decline and become cheap enough for AGI by the 2020's or 2030's. That's half a century ago!

chermiOP8mo ago

Fair point, the first few gens weren't that expensive. And like I said I'm certain the scaling would've been discovered soon with some time lag. But the transformer paper was 2017, right? Just from a benefit -to- society perspective, assuming LLMs are a net positive in terms of productivity, perhaps reducing time to drug approval or improving government efficiency (1).. Isn't getting there just one year earlier worth it, if the gains really are that big? We could be talking lives saved via faster approvals or more efficient spending. My point is that a private company made it happen faster, as evidenced by them doing it first. A good thing sooner is valuable.

(1) I'm convinced at the very least LLMs can feasibly speed up paperwork.

loglog8mo ago

I believe that LLMs are detrimental to government efficiency, because they distract from solving the underlying problems (such as porkishly complex laws or a lack of unique identifiers).

j / k navigate · click thread line to collapse