Block Diffusion: Interpolating Autoregressive and Diffusion Language Models (opens in new tab)

I've built diffusion based text models, it's old hat and not necessarily the most performant way to generate text. However it does produce interesting results and I'd love to test some ideas at scale.

There’s already a few models that are diffusion based.

an undergraduate degree in a field is not enough to understand recent research in a specialised subfield of a subfield and you shouldn't beat yourself up over that

there's nothing wrong with you, you just need the right background and you can go get that. see e.g. the fast.ai course

I wouldn’t beat yourself up over it. Very few papers can be understood without reading a significant amount of the neighboring literature and the history of how that work came to be. There are norms and customs and a kind of academic language in every community that you won’t be able to see unless you’ve read a lot from that community. Even if you have the right math level it’s tricky.

A single paper is part of a conversation, not something that stands alone. Trying to read one random paper is like finding a 1000 page thread on an obscure topic that has been running for 10+ years and reading only the last page. It won’t make any sense without reading back a ways.

Might want to study some stats or other math.

depth first read the references until the leaves are obvious!

Ask ChatGPT o3 about anything you don't understand, ask it about anything in its responses you don't understand. Keep drilling down until you do understand. Takes patience, but you can learn a lot very fast, this way.