Zero-3 Offload: Scale DL models to trillion parameters without code changes (opens in new tab)

What are the relevant commits in Fairseq for this? I couldn't figure out the changes by looking at the commits from last week.

It seems like Zero-3 doesn't work for them:

https://github.com/EleutherAI/gpt-neox/issues/171

I think there is some explanation (on the previous model?) here, https://www.youtube.com/watch?v=tC01FRB0M7w

Third paragraph or so in the overview:

> ZeRO removes the memory redundancies across data-parallel processes by partitioning the three model states (optimizer states, gradients, and parameters) across data-parallel processes instead of replicating them. By doing this, it boosts memory efficiency compared to classic data-parallelism while retaining its computational granularity and communication efficiency

I don't think that's the Stage 3 announced in this blog post, but it's def a framework for it.

SGD doesn't work on large Transformers, no. You need something like AdamW.

You ideally need ~500GB of text, or so. EleutherAI's The Pile was designed to be just big enough to fit a 1t GPT efficiently, and you can get the various scaling curves out of the OA-related scaling papers. (You want the amount of data that fits into a single epoch, because if you reuse data, you get less bang for the FLOPs buck, and FLOPS constraints are right now much more binding than data or model size.)