Token Embeddings Violate the Manifold Hypothesis (opens in new tab)

(arxiv.org)

3 pointslukeplato4mo ago1 comments

1 comments

If this interests you, https://bsky.app/profile/jefferyharrell.bsky.social has been making some super cool visualizations of Qwen3's internal representations

One thing discovered was that 800+ tokens converged to a single point in the 2500+ dimensional unembedding space

Some examples:

j / k navigate · click thread line to collapse

If this interests you, https://bsky.app/profile/jefferyharrell.bsky.social has been making some super cool visualizations of Qwen3's internal representations

One thing discovered was that 800+ tokens converged to a single point in the 2500+ dimensional unembedding space

Some examples:

j / k navigate · click thread line to collapse