undefined | Better HN

0 pointsbiosboiii1y ago0 comments

I think the comment author means offering inference via Firebase, with the model never leaving the backend.

This works, just like ChatGPT works, but has the downside of 1. You have to pay the computing for every inference 2. Your users can't access it offline 3. Your users will have to use a lot of data from their mobile network operator. 4. Your inference will be slower

And since SeeingAI infers the model every second, your and your customers bill will be huge.

0 comments

hn87261y ago

That's what I thought, but the link doesn't say anything about off-device inference, it's only about storing and retrieving the model. There's just one off-hand note about cloud inference.

In any case, yeah you can not download the model to the device at all, but then you have to deal with the other angle - making sure the endpoint isn't abused.

Maybe a hybrid approach would work - infer just part of the model (layers?) on the cloud, and then carry on the inference on the device? I'm not familiar with how AI models look like and work like exactly, but I feel like hiding even a tiny portion of the model would make it not usable in practice

biosboiiiOP1y ago

Your second note is very interesting, having looked at the model myself this is very plausible.

For models which use a lot of input nodes, a lot of "hidden layers" and in the end just perform a softmax this may get infeasible because of the amount of data you would have to transfer.

You may have inspired a second article :)

j / k navigate · click thread line to collapse