The year has 86400*365 = 31536000 seconds. Thus 63072000000 tokens can be generated. As pricing is usually given per 1M tokens generated, this is 63072 such packages.
Now lets write off the investment over 3 years, 250,000/63072 = 3.96. So almost $4 per 1M tokens generated with prompt processing included.
Model was a Deepseek 671B 32B MoE.
Looks to me that $20 for a month of coding is not very sustainable - let's enjoy the party while VCs are financing it! And keep an eye on your consumption...
Electricity costs seem negligable with ~$10,000 per year at 10cts per kWh but overall cost would be ~10% higher if electricity is more like 30cts like it is in Europe.
Edit: like it is pointed out by other commenters it is 2200t/s per single GPU thus the result needs to be divided by 16: $4/16 = $0.25. This actually somewhat matches the deepseek API pricing.
8xH200 enclosed in DGX H200 system power draw is ~14kW in its peak (CTS) configuration/utilization. Over one year, and assuming maximum utilization, this is 123,480 kWh per single DGX H200 unit. We need 2x such units for 16xH200 system configuration under subject so it's 246,960 kWh/year. This is ~$25,000 at 10cts per kWh and ~$74,000 at 30cts per kWh. At ~1,110,000 1M batches this gives us: (1) ~$0.02 - $0.07 per 1M of energy cost and (2) ~$0.25 per 1M assuming the same HW depreciation rate. In total, this is ~$0.3 per 1M tokens.
Seems sustainable?
One has to keep in mind that the benchmark that was done is synthetic. This makes sense because it makes it reproducible but real world usage may differ - i.e. by the amount of context and the number of concurrent users. Also there are use cases where smaller models or smaller quants will do.
The key take away for me for this type of back of the envelope calculation is to get a good idea where we stand long term, i.e. when VC money stops subsidizing.
So for me $0.3 per 1M tokens for a decent model looks pretty good too. Seeing that OpenAI API charges $21 per 1M tokens input and $168 output for GPT-5.2 pro I was wondering what the real sustainable pricing is.
Now take into account that modern LLMs tend to use 4bit inference, and Blackwell is significantly more optimized for 4 bit, we can see much less than 11 cents. Maybe a speed up of 5x if using 4bit and Blackwell vs H100 and 8 bit?
So we're looking at potentially 2.2 cents per million tokens.
Some notes:
- # Input tokens & # output tokens per request matters a lot.
- KV Cache hit rate matters a lot.
- vLLM is not the necessarily most efficient engine.
- You are looking at API cost for DeepSeek V3.2, which is much cheaper than DeepSeek R1 / V3 / V3.1. DeepSeek V3.2 is different architecture (sparse attention) that is much more efficient. DeepSeek V3 cheapest option (fp8) tends to be ~$1/mil output tokens while R1 tends to be ~$2.5/mil (note that for example Together AI charges whopping $7/mil output tokens for R1!)
As for the cost: You can also get H200s for ~ $1.6/hr and H100s for ~ $1.2/hr. That somewhat simplifies the calculations :)
Ignoring the caveats and assuming H200s, with their setup you will:
- Process 403200000 input tokens.
- Generate 126720000 output tokens.
- Spend $25.6.
- On Together with DS R1 it would cost you $3 * 403.2 + $7 * 126.7 = ~$2096. Together does not even offer discount for KV cache hits (what a joke :)).
- On NovitaAI with DS R1 it would cost you $0.7 * 403.2 + $2.5 * 126.7 = ~$600 (with perfect cache hit rate, which gives 50% discount on input tokens here, it would be ~$458).
So if you work that through its $0.225 per 1M output tokens.
Nope - i live in one of the most expensive areas, and even the residential price has averaged 18c/kWh delivered including taxes. Businesses get a lower basic rate and also don't pay the VAT, so it works out around 13c/kWh for them.
https://data.nordpoolgroup.com/auction/day-ahead/prices?deli...
The VC money is there until they can solve the optimization problems
Makes you think that you will continue to see the costs for a fixed level of "intelligence" dropping.
On the bright side, they haven't started exploring stacking chips on top of their wafers to increase local memory, and every process change will bring increased bandwidth in and out of their "pizza". I really wish they succeed.
It almost feels like in the past year there is some unwritten agreement between the 3 main open-source engines (vLLM, sglang, TRT-LLM) to not compare to each other directly :) They used to publish benchmarks comparing against each other quite regularly.
Probably there are more interesting/easily verifiable agent loops you could try for kernel optimizations. At this point, the best are still written by hand, though. Ex: DeepEP kernels https://github.com/deepseek-ai/DeepEP
Given that SOTA models now use 4bit inference, can you do an estimation for 4bit + Blackwell?
(PyTorch does also support ROCm generally, it shows up as a CUDA device.)
However from experience with an AMD Strix Halo, a couple of caveats: it's drastically slower than Ollama (tested over a few weeks, always using the official AMD vLLM nightly releases), and not all GPUs were supported for all models (but that has been fixed).
If you want more performance, you could try running llama.cpp directly or use the prebuilt lemonade nightlies.
Also, I'd rather run a large model at slower speeds than a smaller at insanely high speeds.
Also, the entire industry profits from the work that’s done at the bleeding edge. That’s the case in every industry.
Tool use that changes the mode of the environment is a good example where you cannot go parallel. I've built a recursive agent that can run a Unity editor and I can't just blindly run whatever it wants in parallel or combos like SwitchScene -> GetSceneOverview won't interleave correctly. You'll wind up with 15 calls that loop over every scene and then you grab the overview from the last scene you switched to 15 times.
There are ways to hack around it a bit, but at some level the underlying narrative does need to be serialized or you'll be wasting an incredible amount of resources.
Depth-first search doesn't guarantee the best solution, but on average it's guaranteed to find a solution faster than breadth-first search. It's worth waiting for those dependent calls and going super deep if you want some reasonable answer quickly.
I’m sure I’d use more tokens because I’d get more revs, but I don’t think token usage would increase linearly with speed: I need time to think about what I want to and what’s happened or is proposed. But I feel like I would be able to stay in flow state if the responses were faster, and that’s super appealing.
