Sharing how I think (hypothesis) Gemini models achieve their 1-10 Million long context window. With details to clues to support the same.
Ensemble of Expert (EoE) or Mesh of Expert (MeoE) with common/shared long (1-10M) context window
Gemini's 1M+ token MoE likely uses "instances" (active expert sets/TPU shards) sharing a common distributed context; individual active expert groups then use relevant "parts" of this vast context for generation. This allows concurrent, independent requests via distinct system "partitions."
The context is sharded and managed across numerous interconnected TPUs within a pod.
For any given input, only a sparse set of specialized "expert" subnetworks (a "dynamic pathway") within the total model are activated, based on complexity and context required.
The overall MoE model can handle multiple, concurrent user requests simultaneously.
Each request, with its specific input and context, will trigger its own distinct and isolated pathway of active experts.
Shared context that can act as independent shards of (mini) contexts.
The massively distributed Mixture of Experts (MoE) architecture, across TPUs in a single pod, have its the long context sharded and managed via parallelism, and with ability to handle concurrent requests by part of that context window and independent expert pathways across a large TPU pod, also it can use the entire context window for a single request if required.
Evidence points to this: Google's pioneering MoE research (Shazeer, GShard, Switch), advanced TPUs (v4/v5p/Ironwood) with massive HBM & high-bandwidth 3D Torus/OCS Inter-Chip Interconnect (ICI) enabling essential distribution (MoE experts, sequence parallelism like Ring Attention), and TPU pod VRAM capacities aligning with 10M token context needs. Google's Pathways & system optimizations further support this distributed, concurrent model.
og x thread: https://x.com/ditpoo/status/1923966380854157434
