Cheyenne Super Computer Auction (opens in new tab)

Wow, that's ridiculous. I bought two racks of Origin2000 with a friend in high school and that was enough logistic overhead for me! I can't imagine 72 racks!!

Installing IRIX doesn't require CDs; it's much, much easier done over the network. Back in the day it required some gymnastics to set up with a non-IRIX host, now Reanimator and LOVE exist to make IRIX net install easy. There are huge SGI-fan forums still active with a wealth of hardware and software knowledge - SGIUG and SGInet managed to take over from nekochan when it went defunct a few years ago.

I have two Origin 350s with 1Ghz R16ks (the last and fastest of the SGI big-MIPS CPUs) which I shoehorned V12 graphics into for a sort of rack-Tezro. I boot them up every so often to mess with video editing stuff - Smoke/Flame/Fire/Inferno and the old IRIX builds of Final Cut.

I think that by the time Bitcoin came out, Origin2000s would have been pretty majorly outgunned for Bitcoin mining or any kind of compute task. They were interesting machines but weren't even particularly fast compared to their contemporaries; the places they differentiated were big OpenGL hardware (InfiniteReality) with a lot of texture memory (for large-scale rendering and visualization) and single-system-image multiprocessor computing (NUMAlink), neither which would help for coin mining.

"The original NFL broadcasts with that magic yellow first down line were synthesized with similar hardware"

That was originally an XFL innovation.

Really cool, do you have pictures or a blog post about this?

Given that the individual nodes are just x86_64 Xeons and run linux... it would be interesting to part it out for sale as individual, but functional, nodes to people. There are a lot of people would like to have a ~2016 era watercooled 1U server from a supercomputer that was once near the top of the Top500 just to show to people.

Get little commemorative plaques for each one and sell for $200 each or so.

edit: it seems each motehrboard is a dual CPU board and so there are 4032 nodes, but the nodes are in blades that likely need their rack for power. But I think individual cabinets would be cool to own.

There are 144 nodes per cabinet... so 28 cabinets. I'd pay a fair amount just to own a cabinet to stick in my garage if I was near there.

Is it not "Personal" protective equipment?

https://en.wikipedia.org/wiki/Personal_protective_equipment

I can find a bunch of the E5-2697v4 CPUs on eBay in the $30-40 range.

I wonder if there is a market for the SGI hardware.

If somebody has the money, and the resources required to house the system, seems like decent value for the money at the reserve (apparently $100k) The range of, I'd buy if I had the money and a valid use case. Even partial resale like suggested. There's an argument about being behind the "most" advanced. Yet, its a petascale server from 2016 that was good enough for the NSF. It's not exactly an old dog.

The stuff I could quickly find was $1M+ in individual sales (not sure about auction sites and used).

DDR4 RAM $734,500 @ $75 / 32 GB for 313,344 GB

CPUs $484,000 @ $60 / (1) E5-2697v4 for 8,064 units

Figure the hard drives must also be enormous, and probably a huge amount of storage, just not sure on the quantity, and they may not be sold with the system.

They're not kidding on the moving company, that's 10's of tons of computer to haul somewhere. 26 1U Servers @ 2500 lbs each. Couple semi-trucks, since the most they can usually do is 44,000 lbs on US highways. Not sure if the E-Cell weight was 14 @ 1500 lbs in addition to the 26 @ 2500 lbs?

>...totaling 313,344 GB

Can you imagine the RAMDisk? Yes, you can. Especially in 20 years when it will be the norm. And also the Windows version that will require half of it in order to run /s

Does it come with a portable nuclear reactor to power it?

In comparison, 18 A100 GPUs would have 5.6 petaflops and 1.4 TB vram, consuming 5.6 kw.

The speed of processing and interconnect is orders of magnitude faster for an A100 cluster - 1 8 gpu pod server will cost around $200k, so around $600k more or less beats the supercomputer performance (price I'm searching seems wildly variable, please correct me if I'm wrong.)

Aw man... I was going to use it for my homelab but that's 1696320W more than I can supply. Well... maybe if I use two plugs instead of one...

It was at #160 in 2023 when it was decommissioned.

At the ill fated Portland TechShop I took woodworking classes from a retired gentleman, who professionally was a pattern maker for molding cast metal parts. This made his approach to woodworking really interesting. He had a huge array of freestanding sander machines, including a disc sander with more than a yard diameter.

For anyone unfamiliar, pattern makers would make wooden model versions of parts that were to be cast in metal. The pattern would be used to make the mold. He could use these various sanding machines to get 1/64" precision for complex geometries. It was fascinating to watch how he approached things, especially in comparison to modern CNC.

His major project outside of teaching the classes? Making patterns for a local steam locomotive restoration project. He had all these wooden versions of various parts of a locomotive sitting around.

Funny, just yesterday I saw here in Germany a train where the locomotive was labeled 'rent me'. Apparently there's an organization which bought old locomotives (this particular one looked like 60s, maybe early 70s vintage) from the state-run former monopoly to rent them out (large industrial customers I'd think).

I was surprised to see such an old locomotive in operation, but apparently it's still good (i.e. economic) enough to shuttle cars from factory to port. Guess the air pollution restrictions aren't all that tough for diesel trains.

EDIT: didn't realize this is big business in Germany: https://railmarket.com/eu/germany/locomotive-rental-leasing

May I present my postmodern theory of aesthetics:

- If it's useful as a medium for money laundering, it's art.

- If it's useful as a facilitator for money laundering, it's technology.

Are you trying to discourage others from bidding, so you can swoop in and win the auction?

> a price in mind they are not willing to go below

I worked for an auction, and sellers accepted bids below the reserve price all the time. They just want to avoid a situation where an item sells at a “below market” price due to not having enough bidders in attendance - e.g. a single bidder is able to win the auction with a single lowball bid. If they see healthy bidding activity that’s often sufficient to convince them to part with the item below reserve.

Reserve prices are annoying for buyers, but below-reserve bids can provide really useful feedback for sellers.

We even had full-time staff whose job was to contact sellers after the auction ended and try to convince them to accept a below-reserve bid, or try to get the buyer and seller to meet somewhere in the middle. This worked frequently enough to make this the highest ROI group in our call center.

It is playing on the psychology of the bidders. You want the bidders to be invested, to want to win the auction. To compete to win the prize.

Also, consider this: if the reserve is too high, and no one bids on it, then everyone looking at it is going to wonder what it is really worth. If there are several other bidders, then that gives reassurance to the rest for the price they each are bidding.

It's entirely because of human nature - you want people to get invested in it, which having them bid any amount does.

It's the same reason an auction can go above the price/value of the thing, because you get invested in your $x bid, so $x+5 doesn't seem like paying $x+5, but instead "only $5 more to preserve your win" type of thing.

See penny auction scams - https://utahjustice.com/penny-auction-scams - for an extreme example.

Cray XT/SeaStar? iirc the interconnect ASIC pretends to be another peer CPU connected via HyperTransport. HPE Flex is similar, but works via QPI/UPI for Intel CPUs.

If you can get 1.7MW service, then you are paying utility rates, or around 100-150 per MWh, or as you quoted 4-6k per day. In Seattle, running this off peak would cost one about 104$/hr before the other fees.

I would be neat if a subset of this could be made operational and booted once in awhile a computer history museum. I agree that it doesn't make sense to actually run it.

https://seattle.gov/city-light/business-solutions/business-b...

I was just thinking how much of deep space radio telemetry this super computer must have seen.

the only winning move is not to play

There's a whole sub-industry of people bidding on government auctions in order to part out the stuff. I'd be pretty surprised if the whole cluster got reassembled. But people on a budget will buy those compute nodes, someone trying to keep their legacy IB network will snap up those switches, the racks, etc.

>Crypto miners?

I think mining crypto with these would burn far too much energy compared to the ASICS in use.

Crypto is no longer mined commercially with GPU type compute. When ETH switched to PoS, it decimated the entire GPU mining industry. It is no longer profitable. The only people doing it now are hobbyists.

Recyclers.

Academic departments with low budgets and cheap electricity who can make due with old CPUs

You won't find 8064 buyers of these CPUs any time soon. It will take many years, maybe a decade to offload.

For example, in April ebay sold 79 of them. You're looking at ~100 months. And that's assuming constant demand.

I ended up reading all the documentation provided alongside the auction because I was debating making a bid. I decided not to in the end.

In total, Cheyenne weighs in around ~95000lbs in weight, they require it be broken down in <= 3 business days on site and only offer 8AM - 4PM access. Therefore you'll need to support unbolting and palletizing all the cabinets for loading into a dry trailer, get a truck to show up, load, then FTL to the storage (or reinstall) location of your choosing for whatever's next.

Requirements for getting facility access seem to mean you bring both labor and any/all specialized equipment including protective equipment to not damage their floors while hauling away those 2400lb racks. Liability insurance of $1-2M across a wide range of categories (this isn't particularly unusual, or expensive, but it does mean you can't just show up with some mates and a truck you rented at Home Depot).

I'd guess you're looking at more like $25k just to move Cheyenne via truck, plus whatever it sells for at auction, unless you are located almost next door to them in Wyoming. Going from WY to WA the FTL costs alone would be $6-7k USD just for the freight cost (driver, dry trailer and fuel surcharges), add a bit if your destination doesn't have a loading dock and needs transloading for the last mile to >1 truck (probably) equipped with a tail lift. All the rest is finding a local contractor with semi-skilled labor and equipment to break it down and get it ready to load on a truck and covering the "job insurance".

Warehouse costs if you're breaking it for parts won't be trivial either and you'd be flooding the used market (unfavorably to you) were you to list 8000 CPUs or 300TB DDR4; it could take months or even to clear the parts without selling at a substantially depressed price.

It will take probably several thousand hours in labor to break this for parts assuming you need to sell them "Tested and Working" to achieve the $50/CPU unit pricing another commenter noticed on eBay, and "Untested; Sold as Seen" won't get anywhere near the same $/unit (for CPUs, DRAM, or anything else) and so even assuming $25/hr for fully burdened relatively unskilled labor, you could well be talking up to $100k in labor to break, test and list Cheyenne's guts on eBay or even sell onward in lots to other recyclers / used parts resellers. I don't think I could even find $25/hr labor in WA capable and willing of undertaking this work, I fear it'd be more like $45-60/hr in this market in 2024 (and this alone makes the idea of bidding unviable).

A lot of the components like the Cooling Distribution Units are, in my opinion, worth little more than scrap metal unless you're really going to try and make Cheyenne live again (which makes no sense, it's a 1.7MW supercomputer costing thousands of dollars per day to power, and which has similar TFLOPs to something needing 100x less power if you'd bought 2024-era hardware instead).

Anything you ultimately can't sell as parts or shift as scrap metal you are potentially going to have to junk (at cost) or send to specialist electronics recycling (at cost); your obligations here are going to vary by location and this could also be expensive -- the leftovers.

If anyone does take the plunge, please do write up your experiences!

The heat's going to be leaving the building in liquid-filled pipes, however you architect it. And with 1.7MW of peak power consumption, a nontrivial amount of liquid.

It's just a question of whether you want to add air and refrigerant into the mix.

It seems they're decommissioning it partly due to "faulty quick disconnects causing water spray" though, so an air cooling stage would have had its benefits...

Has been really common in HPC for quite a while. I presume the higher interconnect/network of hpc favour the higher density of liquid cooling. Hardware utilization is also higher compared to normal datacenters, so the additional efficiency vs air cooling is more useful.

for large machines the air setup is really less efficient and takes up alot of space. you end up building a big room with a pressurized floor which is completely ringed by large ac units. you have to move alot of air through the floor bringing it up through the cabinets and back through to the acs. its also a big control systems problem, you need to get the air through the cabinets evenly, so you need variable speed fans or controlled ducts..and those need to be adaptive but not oscillate.

with a water cooled setup you can move alot more heat through your pipes just be increasing flow rate. so you need pumps instead of fans. and now your machine room isn't a mini-hurricane, and you can more flexibly deal with the waste heat.

Built in a bunker under a mountain, so reduced airflow, plus need to hide heat signatures from outside surveillance?

Also, likely they had infrastructure available, from the nuclear power they use.

Reading the whole paragraph, it sounds to me like they were accepting 1% errors rather than fixing the leaking cooling system.

Unfortunately LCM closed during the pandemic and laid off all their staff, with no sign of reopening.

A strange game; the only winning move is not to play

I dunno, I still think the 2011-v3 platform that these Xeons can run in is still a great setup for a homelab. A bit power hungry but if you can build a dual core workstation with 36 cores, and 256GB ram for <$1000, that is a solid server. Sure, a bit more power hungry, but that would still make for a hell of an app/db server. That's basically an m4.16xlarge, same CPU generation, platform etc, (yes, without all the surrounding infra) that will cost you something like $0.10 per hour worth of energy to run.

Take the Dell T7910 for example (I use one of these for my homelab), you can pick up a basic one with low end CPU/RAM for sometimes as little as $300. Dumping all these 18 core E5s and DDR4 ECC on the market should make it even cheaper to spec out. Currently they go for about $100-150 each on the CPUs, and ~$150-200 for the RAM. Not bad IMO.

Remember that the buyer has to move it. If that costs $50K (no idea, totally guessing) then it's currently "worth" $53K

Hang around for the winning bid and you'll see what it's worth then.

You forgot to say First Post!

This is from Cheyenne, Wyoming, not Cheyenne Mountain (in Colorado Springs).