Hetzner introduces GPU server for AI training (opens in new tab)

Hmm. Seems like a bad deal.

This is a monthly reservation for a single 6000 Ada for $940. You can get the same on RunPod for $670.

And to actually train stuff you'd likely want nodes with more of them, like 8, or just different GPUs all together (like A100/H100/etc).

What’s the most cost effective option for hosting an llm these days? I don’t need to train, I just want to use one of the llama models for inference to reduce my reliance on 3rd parties.

So 1kEUR/month for a 6kEUR GPU. Pretty sure there are a lot of drug dealers who wish they had gone into cloud training instead.

CoCalc offers On-Demand GPU servers with H100s starting at $2.01 per hour (metered per second) through its integration with Hyperstack... It also has more budget-friendly options, like RTX A4000s at $0.18 per hour.

https://cocalc.com/features/compute-server

In case you are not familiar, CoCalc is a real-time collaborative environment for education and research that you can access via your web browser at https://cocalc.com/

What's currently the cheapest/easiest way to deal with relatively lightweight GPU tasks, that are not lightweight enough for my PC?

Consider this use case: I want to upload 50 GB of audio somewhere and run whisper (biggest model) on it. I imagine the processing should be doable in minutes for a powerful GPU and must be very cheap, the script will be like 20 LOC, but I'll spend some time setting stuff up, uploading this and so on (which for example, makes colab a no-go for this). Any recommendations?

Also, when they say it's "per hour" do they mean an hour of GPU-time, or an hour of me "renting the equipment", so to say?

Pricing is surprising, typically Hetzner has extremely low prices, yet here there are 50%-70% more expensive then the competition, and you also pay a one time setup cost.

I always looks at these prices and think it’s a complete rip off for anyone doing less than 4 GPUs.

Do any of these offer training data as a service? Seems like they could charge a premium for a continuous multicast of a large dataset on say a 10g or higher connection. A one-to-many reply and charge the customer to sit under the firehose.

I use runpod or vast for training my (small) models (a few million parameters) mostly using RTX4090 up to 4 GPUs. Training is a sporadic task. Is not worth it for me to book it monthly (at these prices)

According to the benchmarks: https://github.com/mag-/gpu_benchmark

RTX 6000 Ada is ~A100

Although R2 and B3 are excellent alternatives to S3, go get them Hetzner!

Hetzner is a great, reliable company with fantastic offerings and excellent support.