undefined | Better HN

0 pointscoldtea1mo ago0 comments

Would it be that many? Asked AI to do some rough calculation, and it spit that:

Making 50 SOTA AI requests per day ≈ running a 10W LED bulb for about 2.5 hours per day

Given I usually have 2-3 lights on all day in the house, that's like 1500 LLM requests per day (which sounds quite more than I do).

So even a month worth of requests for building some software doesn't sound that much. Having a local beefy traditional build server compling or running tests for 4 hours a day would be like ~7,600 requests/day

0 comments

shakna1mo ago

> Making 50 SOTA AI requests per day ≈ running a 10W LED bulb for about 2.5 hours per day

This seems remarkably far from what we know. I mean, just to run the data centre aircon will be an order of magnitude greater than that.

andoando1mo ago

Air conditioning for a whole data center services a whole data center, not one machine running a task for 1 min

shakna1mo ago

Yes... But the machines in those data centres don't get there without the companies who put them there. You get no tasks for no minutes, without the infrastructure, and so the infrastructure does actually have to be part of the environmental impact survey.

anonzzzies1mo ago

Is that true? Because that's indeed FAR less than I thought. That would definitely make me worry a lot less about energy consumption (not that I would go and consume more but not feeling guilty I guess).

derekdahmer1mo ago

A H100 uses about 1000W including networking gear and can generate 80-150 t/s for a 70B model like llama.

So back of the napkin, for a decently sized 1000 token response you’re talking about 8s/3600s*1000 = 2wh which even in California is about $0.001 of electricity.

pshc1mo ago

With batched parallel requests this scales down further. Even a MacBook M3 on battery power can do inference quickly and efficiently. Large scale training is the power hog.

j / k navigate · click thread line to collapse

0 pointscoldtea1mo ago0 comments

Would it be that many? Asked AI to do some rough calculation, and it spit that:

Making 50 SOTA AI requests per day ≈ running a 10W LED bulb for about 2.5 hours per day

Given I usually have 2-3 lights on all day in the house, that's like 1500 LLM requests per day (which sounds quite more than I do).

0 comments

shakna1mo ago

> Making 50 SOTA AI requests per day ≈ running a 10W LED bulb for about 2.5 hours per day

This seems remarkably far from what we know. I mean, just to run the data centre aircon will be an order of magnitude greater than that.

andoando1mo ago

Air conditioning for a whole data center services a whole data center, not one machine running a task for 1 min

shakna1mo ago

anonzzzies1mo ago

derekdahmer1mo ago

A H100 uses about 1000W including networking gear and can generate 80-150 t/s for a 70B model like llama.

So back of the napkin, for a decently sized 1000 token response you’re talking about 8s/3600s*1000 = 2wh which even in California is about $0.001 of electricity.

pshc1mo ago

With batched parallel requests this scales down further. Even a MacBook M3 on battery power can do inference quickly and efficiently. Large scale training is the power hog.

j / k navigate · click thread line to collapse