BitNet b1.58 2B4T Technical Report (opens in new tab)

Ok that's fair. I still think something was up with my build though, the online demo worked far better than my local build

sure, here's my conversation with BitNet b1.58 2B4T

https://pastebin.com/ZZ1tADvp

here's the same prompt given to smollm2:135m

https://pastebin.com/SZCL5WkC

The quality of the second results are not fantastic. The data isn't public, and it repeats itself mentioning income a few times. I don't think I would use either of these models for accurate data but I was surprised at the truncated results from bitnet

Smollm2:360M returned better quality results, no repetition, but it did suggest things which didn't fit the brief exactly (public data given location only)

https://pastebin.com/PRFqnqVF

Edit:

I tried the same query on the live demo site and got much better results. Maybe something went wrong on my end?

That link gives a great description of how to pack trits more efficiently, thanks. Encoding in "base 3" was obvious to me, but I didn't realise that 5 trits fit quite tightly into a byte, or that it's possible to "space the values apart" so that they can be extracted using just multiplications and bitwise ops (no division or remainder).

The custom CUDA kernel for 4-in-8 seems to have come out better than a naive approach (such as just treating each as an fp8/int8) + it lowers memory bandwidth. Custom hardware would certainly make that improvement even better but I don't think that's what's limiting training to 2-8 billion parameters as much as something like research convenience while the groundwork for this type of model is still being figured out.

Makes sense. Might be good for mem throughput constrained devices though so hoping it’ll pick up

That's not a real BitNet, it's just a post-training quantisation, and its performance suffers compared to if it was trained from scratch at 1.58 bits.

My impression was that text embeddings are better suited for classification. Of course the big caveat is that the embeddings must have "internalized" the semantic concept you're trying to map.

From some article I have in my draft, experimenting with open source text embeddings:

    ./match venture capital
    purchase           0.74005488647684
    sale               0.80926752301733
    place              0.81188663814236
    positive sentiment 0.90793311875207
    negative sentiment 0.91083707598925
    time               0.9108697315425
 
    ./store sillicon valley
    ./match venture capital
    sillicon valley    0.7245139487301
    purchase           0.74005488647684
    sale               0.80926752301733
    place              0.81188663814236
    positive sentiment 0.90793311875207
    negative sentiment 0.91083707598925
    time               0.9108697315425

Of course you need to figure out what these black boxes understand. For example for sentiment analysis, instead of having it match against "positive" "negative" you would have the matching terms be "kawai" and "student debt". Depending how the text embedding internalized negatives and positives based on their training data.

Three strategies seem to be:

- Use LLM to evaluate result and retry if it doesn't match.

- let users trigger a retry

- let users edit