Muvera: Making multi-vector retrieval as fast as single-vector search (opens in new tab)

We added Muvera to Weaviate recently https://weaviate.io/blog/muvera and also have a nice podcast on it https://www.youtube.com/watch?v=nSW5g1H4zoU.

When looking at multi-vector / ColBERT style approaches, the embedding per token approach can massively increase costs. You might go from a single 768 dimension vector to 128 x 130 = 16,640 dimensions. Even with better results from a multi-vector model this can make it unfeasible for many use-cases.

Muvera, converts the multiple vectors into a single fixed dimension (usually net smaller) vector that can be used by any ANN index. As you now have a single vector you can use all your existing ANN algorithms and stack other quantization techniques for memory savings. In my opinion it is a much better approach than PLAID because it doesn't require specific index structures or clustering assumptions and can achieve lower latency.

Seems to be a trend away from mean-pooling into a single embedding. But instead of dealing with an embedding per token (lots) you still want to reduce it some. This method seems to cluster token embeddings by random partitioning, mean pool for each partition, and concatenate the resulting into a fixed-length final embedding.

Essentially, full multi vector comparison is challenging performance wise. Tools and performance for single vectors are much better. To compromise, cluster into k chunks and concatenate. Then you can do k-vector comparison at once with single-vector tooling and performance.

Ultimately the fixed length vector comes from having a fixed number of partitions, so this is kind of just k-means style clustering of the token level embeddings.

Presumably a dynamic clustering of the tokens could be even better, though that would leave you with a variable number of embeddings per document.

I'm only vaguely familiar with this. So I apologize how I phrase this.

If make a basic sequel query to return all the first names in table, then i can generally expect it to return them all.

If I do a similar query with these neural embeddings could i expect the same or is it more fuzzy?

Perhaps I misunderstood but it calculates the FDE of query and looks for a similar FDE in the dataset of the model. Doesnt this require calculating all the equivalent sized FDEs in the model?

It's very hyper-parameter dependent, and in my testing didn't provide comparable performance to maxsim.

So this is basically an “embedding of embeddings”, an approximation of multiple embeddings compressed into one, to reduce dimensionality/increase performance.

All this tells me is that: the “multiple embeddings” are probably mostly overlapping and the marginal value of each additional one is probably low, if you can represent them with a single embedding.

I don’t otherwise see how you can keep comparable performance without breaking information theory.

how is this different from generating a feature hash of the embeddings i.e reduce from many to one embedding reduction? Could a UMAP or such technique be helpful in reducing to a single vector?