johhns4

To deploy fine-tuned LLMs there are these options yes?

1. HuggingFace 2. Runpod 3. Bedrock 4. FireworksAI 5. TogetherAI 6. Replicate 7. Modal

Any other that I'm missing? What do you prefer and for what use, i.e. fine-tuned vs just inference as is.

I'm interested to see if people are fine-tuning the larger LLMs just for kicks or if there are situation where it makes sense (not including high-stakes decision making like medical/legal) and it is computationally efficient?

Smaller models that are cheap to host, sure, but in which cases does fine-tune a larger model (and host it) really shine? Oppose to just using RAG and a closed-source API.

Perhaps it makes sense if it is serving a huge customer base, and the tone of voice needs to be different, but the question is how much work it is to train it and if it is worth it.

I'm not against fine-tuning but curious what the actual use cases are, where it makes economic sense and how successful people/organisations have been.

Been deploying a few models to a Nvidia Tesla T4 · 1x GPU · 16 GB via Hugging Face but keep getting back 500 errors every 2 minutes. Hasn't happened before today, just wondering if this is normal? This increases processing by quite a lot so it's not ideal.

Any tips on how to deploy and use a fine-tuned model on Huggingface in a cost effective way? Right now looking into use Gradio with HuggingFace spaces and using the API endpoint from there. Inference endpoints and Sagemaker seem excessive for this. The whole idea to use smaller models is to decrease costs (vs using a bigger model with an API endpoint) but maybe this just isn't cost effective for where we are right now.