omega3

I came across this linkedin post[0] from: Sr Director, Distinguished Engineer, CTO Office @ Google

The post itself was discussed on HN already: https://news.ycombinator.com/item?id=42852866

> this is my "aha" moment in AI. An AI model that improves other models and itself! with significant proven gains for inference,. Deepseek analyzed the LlaMA code and realized that the KV cache was not optimal so the reasoning engine suggested a way to optimize it. Technically it was probably inspired by Mistral OSS implementation where KV cache uses dynamical allocation. this technique is well know but LlaMA missed it. So, deepseek suggested a code rewrite which was 99% correct with humans only needed to validate and test the code. This improvement results in 2.8x gain for inference on OSS LlaMA. What is even more impressive is that the same change can be adopted to improve DeepSeek itself with initial confirmed gains more that 3x, Effectively a model that is self improving because able to apply previous knowledge when appropriate, augmenting humans and connecting the dots. that's quite something to me. No hype, just pure technological paradigm shift.

Pretty much every sentence has a factual error, I gave DeepSeek the context, the post itself and asked it to fact check:

> The LinkedIn post contains several inaccuracies and exaggerations when compared to the actual news and code provided:

    KV Cache Optimization Claim: The post mentions KV cache optimization as the main improvement, but the provided code and news specifically discuss SIMD optimizations for quantization functions (q4_K, q6_K, etc.) in llama.cpp. There's no mention of KV cache changes in the technical details - this appears to be a conflated claim.

    Performance Gains: While the PR shows 2x speed improvements for WASM through SIMD optimizations, the post inflates this to:

        "2.8x gain for inference on OSS LLaMA"

        "3x gains for DeepSeek itself"
        These specific numbers aren't substantiated in the provided materials.

    Self-Improvement Claim: The post states the model can improve itself, but the evidence shows:

        Humans wrote detailed prompts (multiple iterations needed)

        Humans validated and tested all code

        The model only converted existing ARM NEON code to WASM SIMD

    Mistral Inspiration: The post references Mistral's KV cache implementation, but the actual work:

        Uses Mistral as a reference for model registration patterns

        Focuses on quantization optimizations rather than attention mechanisms

    Code Generation Accuracy: While the post claims "99% correct code", the technical details show:

        Multiple prompt iterations needed (4-8 retries for complex cases)

        Human debugging required for q6_K implementation

        Final code still required human validation

Key Verdict: The LinkedIn post significantly exaggerates and misrepresents the actual technical work. The real achievement was using DeepSeek-R1 to help convert ARM NEON code to WASM SIMD through iterative prompting, resulting in 2x speed improvements for specific quantization functions - not a fundamental architectural improvement to LLaMA or true self-improvement capability.

The post mixes different technical concepts (KV cache optimization vs. SIMD quantization) and inflates the AI's role in the process. While impressive, the actual work is more narrow and human-guided than portrayed.

[0] https://www.linkedin.com/posts/searchguy_this-is-my-aha-moment-in-ai-an-ai-model-activity-7290244226766823425-OUQk?utm_source=share&utm_medium=member_desktop