100M context window means it can probably store everything you’ve ever told it for years.
Couple this with multimodal capabilities, like a robot encoding vision and audio into tokens, you can get autonomous assistants than learn your house/habits/chores really quickly.
At least with other very large context windows like for example Claude offers a RAG is still very much preferable as it avoids confusion and collisions with information in the context that isn’t correct or relevant.
Sure you can also prune the context window and for many existing models you also need to do that (I often use an LLM to summarize a context to keep it going) but doing it with a RAG seems to still be much easier. This especially holds true of you use good knowledge management techniques to structure your RAG so your retrievals are optimized.
P.S. on a side note how confident are we that these very large context window models are not just a RAG in disguise? As the models which boast very large windows are at least for now all locked behind API access only.
Even GPT and Claude make glaring mistakes with short prompts.
1: https://github.com/hsiehjackson/RULER (RULER: What’s the Real Context Size of Your Long-Context Language Models)
They did mention it but didn't provide concrete benchmarks
> We’ve raised a total of $465M, including a recent investment of $320 million from new investors Eric Schmidt, Jane Street, Sequoia, Atlassian, among others, and existing investors Nat Friedman & Daniel Gross, Elad Gil, and CapitalG.
Yeah, I guess that'd do it. Who are these people and how'd they convince them to invest that much?
Something doesn’t add up here.
> Then, it hit them. They thought, “What if we create bite-sized information, following the same scientific standards of peer-reviewed journals, to empower people to solve climate change?”
> Together, they started combing through climate science articles and turning them into social-media friendly content under the name ClimateScience. After two short months, ClimateScience went viral and grew to 40,000 followers on Instagram. People started sending in private messages, asking how they could help. A team of curious, kind and passionate people quickly grew, all dedicated to making climate education more understandable for everyone.
> Just a few years later, ClimateScience has grown into the world’s biggest climate education platform! We create educational courses, videos, resources and tools to improve climate understanding and education. It’s all completely free and just a few clicks away on any device.
According to LinkedIn, they have 50-200 employees. Is that plausible? How many of those are actually FTEs? Where is their revenue coming from if it's all completely free? Looking at the team page, this feels off, like it's a bunch of university students padding their CV.
That was after dropping out of university, 1 year into a bachelor's in computer science. During which he apparently had a 5 month contract at Facebook AI where he "lead the development of 'DREAM', an algorithm that's 100x more data-efficient and trains faster than the previous state-of-the-art in model-free multi-agent Deep RL. Paper: https://arxiv.org/abs/2006.10410".
How does this lead to Magic.dev and third-parties investing $500 million? Either this guy is a prodigy or this is the next Theranos.
edit: I looked into the other co-founder just now and I feel like I'm in the twilight zone.
