undefined | Better HN

0 pointsrs1867mo ago0 comments

> I love the idea of an on-device model

My impression is that most people here haven't tried similar small models and don't have first hand experience with them. They are, to be honest, terrible. They may be good for certain tasks, but are much weaker than something like GPT4. I don't feel excited about these small models that are not fast yet hallucinate all the time.

0 comments

theshrike797mo ago

Weaker by what metric? Are you asking them to explain the fall of Rome to you?

The point of a small model isn't to be an interactive Wikipedia. It's there to call tools, get more data, aggregate the data and return a natural language result.

It does not "hallucinate", because it only uses what the tools provide.

rs186OP7mo ago

If you just look ar similar discussions on HN, you'll see that these models often don't even answer the specific question you ask but just give you random nonsense. I'd rather see an honest "I don't know" instead of complete gibberish.

I'm not going to use any small model that has a chance of dialing 911 when asked to send a text message to a friend.

theshrike797mo ago

This is what tool calling is for. It's just the model being trained to produce a specific type of JSON when it needs an external call

After that it's plain code.

    if phone_number == "911" return false

Done. Won't be calling the cops.

j / k navigate · click thread line to collapse

0 pointsrs1867mo ago0 comments

> I love the idea of an on-device model

0 comments

theshrike797mo ago

Weaker by what metric? Are you asking them to explain the fall of Rome to you?

The point of a small model isn't to be an interactive Wikipedia. It's there to call tools, get more data, aggregate the data and return a natural language result.

It does not "hallucinate", because it only uses what the tools provide.

rs186OP7mo ago

I'm not going to use any small model that has a chance of dialing 911 when asked to send a text message to a friend.

theshrike797mo ago

This is what tool calling is for. It's just the model being trained to produce a specific type of JSON when it needs an external call

After that it's plain code.

    if phone_number == "911" return false

Done. Won't be calling the cops.

j / k navigate · click thread line to collapse