Am I missing anything? (I love Vespa.ai)
The README.md contains a screenshot from local testing that's got more results included: https://github.com/zilliztech/VectorDBBench?tab=readme-ov-fi...
And, about such benchmarks: I tested another vector db benchmark, investigated it a bit, found that it was mostly measuring client implementation latencies and other internal inefficiencies...
In Redis with VSIM I can easily get 50k vSIM/seconds with 300 components vectors with redis-benchmark, yet when I tried to write a quick test for one of those engines I got a lot lower numbers because simply vectors are large (makes serialization in Python slow if not well coded), often these tests are written in high level languages, don't account for differences in client libraries speeds.
TLDR? Benchmarking is hard, for vector systems it is harder, and the results of most of such tests are totally irrelevant.
As you said benchmarking is hard, but isn't the end to end latency customers will see in their workloads is usually including the client library overheads?
IMO, benchmarks should closely resemble the real world scenarios (excluding variables, e.g. network latency of different cloud providers)
