The Cloud Computer (opens in new tab)

Its a huge deal. I'm biased though because my own takes on how things should evolve were very similar. I was however completely unsuccessful in getting those ideas into production! And that, that is a huge deal. Through out my career it has been interesting to meet people with great ideas and then they are unable to get them into production, and when the idea does come into production everyone feels like "Wow, this is so obvious why didn't we do it sooner?" and some folks are banging their head against the wall :-).

One of the more interesting discussions I had during my tenure at Google was about the "size" of the unit of clusters. If you toured Google you got the whole "millions of cheap replaceable computers" mantra. Sitting in Building 42 was a "rack" which had cheap PC motherboards on "pizza dishes" without all that superfluous sheet metal. Bunches of these in a rack and a festoon of network cables. What are the "first class" elements of these machines? Compute? Networking? Storage? Did you replace components? Or a whole "pizza slice" (which Google called an 'index' at the time). Really a great systems analysis problem.

FWIW I'm more of a "chunk" guy (which is the direction 0xide went) and less of a "cluster" guy (which is the way Google organized their infrastructure). A lot of people associated with 0xide are folks I worked with at Sun in the early days and during that period the first hints of "beowulf" clusters vs "super computers", was memory one thing (UMA) or did it vary from place to place (NUMA). I have a paper I wrote from that time about "compute viscosity" where the effective compute rate (which at the time largely focused on transactional databases) scaled up with resource (more memory more transactions/sec for example) and scaled down with viscosity (higher latencies to get to state meant fewer transactions/sec) Sun was invested heavily in the TPC-C benchmarks at the time but they were just one load pattern one could optimize for.

These guys have capitalized on all that history and it is fucking amazing! I just hope they don't get killed by acquisition[1].

[1] KbA is a technique where people who are invested in the status quo and have resources available use those resources to force the investors in a disruptive technology to sell to them and then they quietly bury the disruptive technology.

Coupling vs Decoupling is not some one-sided thing. It's a major trade-off.

One of the most obvious examples of the problem with this approach is that they're shipping previous generation servers on Day 1. One can easily buy current generation AMD servers from a number of vendors.

They will also likely charge a significant premium over decoupled vendors that are forced to compete head-to-head for a specific role (server vendor, switch vendor, etc).

Their coupling approach will most likely leave them perpetually behind and more expensive.

But there are advantages too. Their stuff should be simpler to use and require less in-house expertise to operate well.

This is probably a reasonable trade-off for government agencies and the like, but will probably never be ideal for more savvy customers.

And I don't know how truly open source their work is but if it's truly open source, they'll most likely find themselves turned into a software company with an open core model. Other vendors that are already at scale can almost certainly assemble hardware better than they can.

I don't see it as a big deal - rather, I see it as a huge amount of venture cap spent on some very bright people to build something no one really wants, or, at best, is niche.

Also, it has little to do with the cloud; it is yet another hyperconverged infra.

Weirdly, it is attached to something very few people want: Solaris. This relates to the people behind it who still can't figure out why Linux won and Solaris didn't.

> They sell servers, but as a finished product.

They sell rack-as-compute.[0] Their minimum order is one rack: You plug in power and network, connect to the built-in management software (API), and start spinning up VMs.

[0] With built-in networking and storage.

Existing vendors will provide rack integration services and deliver a turn key solution like this. Also vendors of virtualization management software have partnerships with hardware suppliers and be happy to deliver fully integrated solutions if you're buying by the rack. The difference is in those cases you have flexibility in the design which seems to be missing here.

Proxmox and a full rack of Supermicro gear would not be as sophisticated, but end result is pretty much the same, with I imagine far far better bang for buck.

I like it, but it doesn't seem like a big deal or revolutionary in any way.

IBM invented this a long time ago. Mainframes.

>They sell servers, but as a finished product. Not as a cobbled together mess of third party stuff where the vendor keeps shrugging if there is an integration >problem. They integrated it.

100%.

I’m actually extremely impressed. I want one. I haven’t worked in a data center in years, but I’d be tempted to do it again just to get my hands on one.

>Even if Oxide goes bust, you can still salvage things in a pinch.

Is this true? Can you set your own root of trust for the firmware signing key and build and deploy it yourself?

While working in telecom data centers circa 2016 I've seen many single rack computers from Dell, IBM, HP, Huawei... Not sure that's a new ideia, ex. the open source bits.

> Ironically this looks like the realization of Richard Stallman's dream where users can help each other if something doesn't work.

How is it ironic?

How is this different from AWS Outpost?

This is a turn-key solution, ready to use without eventually dealing with multiple devices with its own firmware and caveats revealing after where put to work together. The closest to that is that AWS managed rack that works with the web APIS you know already

Awesome. This would be especially useful in science but the lack of GPUs is a non-starter. :(

> Ironically this looks like the realization of Richard Stallman's dream where users can help each other if something doesn't work.

that's only true if you think that "users" means "people who operate cloud computers", which is about as far from understanding what Stallman is talking about as is possible. Someone who makes SaaS and runs it on an Oxide computer is no less of a rentier capitalist than someone who makes SaaS and runs it on AWS.

Anyone know specs or prices?

On site backup inventory

DC build out cost and effort

Power cooling requirements

Dark fiber bandwidth requirement

New headcount to support all this

No thanks , I have widgets to make and sell and a business to run.

I think they're mostly targeting customers who want an AWS- or GCP-like experience from a developer perspective (compute is abstracted and you can provision it with an API, etc.), but want to own their own compute infrastructure and have it on-prem. That market has mostly had to cobble together consumer-inspired HP/Cisco/whatever stuff historically (like, one of the early talks about the Oxide value proposition was complaining about why every server in the rack needs a CD drive, which was the norm from Dell), because the kinds of stripped-down, super-efficient hardware designs the hyperscalers were building weren't available to the general public, so this is that: hyperscaler-like technology for people who want to own it themselves.

I think the motivations for why people would want to own their own are probably a mix of financial (at a certain scale there's a tipping point and it gets cheaper), and regulatory/compliance/whatever, like if it's healthcare data, or defense, etc.

Essentially this same sentiment, applies to any number of things:

- Why would anyone buy the Framework laptop, they don't have nearly the support/pedigree that Dell, HP, etc. has?

- Why would anyone use iPhones in the enterprise/IT world, they don't have nearly the support/pedigree that Blackberry, Microsoft, etc. has?

- Why would anyone use Google Fiber, they don't have nearly the network or support that AT&T, Spectrum, etc. has?

- Why would anyone ever use Linux (in enterprise, let's say), compared to the support and adoption that Microsoft/Windows offers?

- ...

I'm purposely picking different examples with varying degrees of success or adoption. I am not claiming that Oxide will be an instant category-dominating success. I don't think Oxide expects to replace HP/Cisco/Dell/etc. overnight, and I don't think a business has to launch with that ambition from the start, to prove that it's worth launching.

But this take is so repetitive as to be bordering on cliche -- I don't know if you're self-aware enough to realize, you are literally just a living embodiment of the "Innovator's Dilemma" right now...

Bryan Cantrill is famously against vendor lock-in. He wrote a[n in]famous blog about the "FYO point" while at Sun. Oxide may be going for customers that also have the same aversion to vendor lock-in.

One thing that Bryan understands is that you can "lock" the customer in with great products and services, as well as continuing development, while also making the customer feel secure in having a way out should you turn into a company that treats locked-in customers as cash cows. The open source strategy (it is a strategy for Bryan and Oxide) is there precisely to do this: make the customer feel they can leave you, but then not.

For your deeply technical staff, having source code access is a big deal too, since it enables them to better understand the products they use.

How big is the market of sufficiently-vendor-lock-in-averse customers? I don't know -- that's not my remit. But there's the size of that market right now, and whether Oxide (and any other companies with similar visions) can grow that market by sheer willpower. I make no predictions.

What if Oxide can get the next Netflix to use their stuff instead of a public cloud?

You may not be aware of the pain that many large, non-software companies currently have on AWS. Gigantic monthly bills (hundreds of thousands per month) coming from subdivisions that aren't capable or motivated to reduce their AWS budget or usage. To the office of the CTO, Oxide's value proposition (buy instead of rent) could be very motivating.

"Hey subdivision A, could we buy a few Oxide racks and move your workload there from AWS? It looks like they would have all the storage and compute you need. Yes? Ok, in 36 months we'll pay your current IT department employees a bonus of 50% of whatever it has saved us vs your current AWS budget."

You’re writing like the status quo is a law of nature. At best it’s been that way for a decade or two

How many times has computing hardware changed in response to the economics of the parts and the economics of the businesses buying hardware?

There are downsides to new models, but money solves a lot of problems

So I don’t know about Oxide in particular, but it seems short sighted to bet on stagnation

Also Oxide is doing what Google did 20 years ago, and Facebook open sourced ~10 years ago, so it’s not exactly unproven

I guess there must be a largish market for this since AWS introduced Outpost to provide the "cloud" to onprem industries. I feel like this is competing with that market.

Since many of those use cases probably already run extensive on-prem infrastructure this could appeal to them. AWS outpost talks about industries like healthcare, telecom, media and entertainment, manufacturing, or highly regulated spaces like financial services. I've heard of media companies that process through things like IMAX cameras that have just tons of TB's of data sometimes just for 5 minutes worth of footage. That would simply be too cost prohibitive - in bandwidth alone - to try and move around in the cloud and you don't want to have to wait for things like AWS snowball or whatever.

While I think the space is "niche" those niche spaces are not small. Big companies with big budgets.

I think they are a competitor to the 'HP/Cisco/IBM/Dell... (blades or hyper-converged)' part of this. They basically saying 'we will do it better'.

Their marketing and story is supposed to convince you that you could save money running their things rather then Dell. And instead of paying for VMWare you get Open Source Software for most of it.

> Until this company has a certain/size and scale, no one serious will trust their black boxes at any type of scale.

I guess that a risk they are willing to take. Some costumers might wait for a few years until they see Oxide being big enough.

Other costumers might be sick of HP/Dell and might take a Risk on a smaller company.

Since they seem to have some costumers, some organizations are willing to take the risk to get away from Dell and friends.

So I think you are the target audience but you are not willing to risk it until they are larger and less likely to fail and they have a good story in regards to support. I assume they have a support story of some kind, no idea what it is 'Contact Sales' ....

In terms of 'trusting they will continue exists' all they can do is survive for a few years until they are pretty established, then more people will be willing buy their product. And hopefully in that time their existing costumers rave about how amazing the product is.

Lets hope they don't go bust because all potential costumers are just waiting. Then again, you can't anybody for not buying from a startup.

I think you actually said it without recognizing.

The current state of the art is a fucking train wreck. Choose any layer of abstraction and start picking at it and you’ll find mostly gaffer tape. Scaling up sucks. Rewriting old monoliths to micro services wasn’t a panacea except maybe for cloud vendor profits. You said the hardware market is well defined, which is another way of saying ossified. Particularly when you start comparing it to the expected pace of software. How’s Open19 going? How much liquid cooling do you have in customer racks?

In my opinion this is ultimately a software offering that happens to come on vertically integrated hardware. It offers a complete, highly polished API to a minuscule-scale DC. If they can find market fit and make a little bit of money, the next step is to start making deep improvements to things behind the abstraction. From where you sit, you are well aware that you could make huge improvements inside the DC demarc if you could do it without disrupting customers. But you’re probably limited by the terrible, terrible APIs you would be forced to use, that don’t offer the capabilities you need, from vendors who would be happy to chat but would provide timeframes in the 6-18 month range for even a modest improvement.

So this, IMO, is about defining that interface to a DC on a single hardware platform with vertical integration, and then scaling up from there. The current hyperscalers will adopt the suite of APIs and capabilities directly, make lower quality copies, or die.

Of course all assuming they’re successful enough to get the revenue machine going in the first place which seems likely to me, given the absolute dogshit state of the cloud world today, the trend towards multi cloud, and the business case for moving certain loads back on prem or to a bare metal colo++ offering.

> you obviously want your DCs as standardized as possible. Until this company has a certain/size and scale, no one serious will trust their black boxes at any type of scale.

This gets to the crux of my first thoughts when I read the marketing copy: can they deliver (reliability).

They do admit very clearly that what they're doing is hard and that at many points during development they were reluctant to be too ambitious (for obvious reasons), but at each stage they did just that: proceeded with the most ambitious option. That takes a huge amount of self belief that might be warranted or might be hubris. As you point out, untimely the success of Oxide won't only hinge on ability to deliver on that self belief, but more on their ability to convince prospective customers that they have competency to do what noone else can.

I fully support every part of their approach in theory but my wizened traveled self thinks it smells a little too good to be true.

Maybe you are not. This offering definitely sounds like something for on prem and not a large data center. Basically, if your core competence is hosting stuff you don't need the extra value they provide. But if your core competence is basically anything else and just need more than a single server under the IT guys' desk then this begins to look very exciting.

To be completely honest, this is for the idealists out there. Those of us who are itching to replace our vSphere with oVirt because 1) we have the time and skill to do it, 2) we believe in open source and 3) we believe we can make huge savings by using open source.

I expect the oxide supporters to have a hard few years ahead of them of finding bugs in high throughput environments. But at the end of the day it will be worth it just to have another competitor in a pretty boring playing field.

They literally had CTOs of F100 companies that want to buy this gear as part of VCs pitch. Because as you can imagine your question was the first question VC's asked.

> Beyond the tech, how would support services really work? We can have a technician from any of the large vendors on-site in less than 2 hours. In some of our DC clusters we actually have vendor support personnel 24x7 on-site with vendor paid spare parts inventory. How would they provide that level of service?

Forgive me for being naive, but that sounds like a great way to differentiate their offering.

E.g. the famous Maytag Repairman, who sits at his desk doing nothing because Maytag washers are so reliable that he has nothing to do.

> In a time in which the laundry appliances of major manufacturers had reached maturity, differing mostly in minor details, the campaign was designed to remind consumers of the perceived added value in Maytag products derived from the brand's reputation for dependability.

https://en.wikipedia.org/wiki/Maytag_Repairman#Ol'_Lonely

> however I don't understand who this is for.

I can't think of a lot of examples right now but I can already imagine one type of customers for such a product - universities wanting high performance computing. At my alma mater, the HPC cluster/server was in a different slightly distant location. Using something like AWS wouldn't be liked by almost any uni admin, and running a server on premises isn't a great idea in a place that gets the occasional (but rare) power or internet cut. Outsourcing some of these responsibilities may have been nice for our admin.

Every single one of the companies you listed started off with unverified hardware, some that failed in the field and continues to today. Why wouldn't you try something else, considering the status quo? This is a nothing-to-lose situation as long as you don't put all your chips into the bet. Every single datacenter has capacity and a need to diversify. It's not even a heroic feat of risk taking.

Having worked in a variety of related fields, my take on this is that hypervisors like VMware have largely eliminated the need for proprietary hardware with “support” contracts. Just buy 3x the capacity in white boxes and turn off anything that breaks.

The problem is that the industry hasn’t noticed this yet. The hyperscalers have, and are printing money as a consequence.

It's basically Sun Microsystems but RIR (rewrite in Rust). The incumbents will eventually release similar products and will eat their lunch.

I might not be some brilliant CTO, but this product obviously seems to fit _very well_ between two very large product offerings: fully cloud being one and fully on-prem being the other. I’ve often thought it would be cool if you could _buy_ a physical space in a server farm, like literally an area of square feet where the racks are all serviced and provided, but meets all of the needs a company who doesn’t want fully cloud has. This is actually way more clever than that.

A lot of people didn’t understand the need for the iPad mini or Mac mini or myriad other products without an obviously existing market. I think this is extremely keen of Oxide to fine the borders between these offerings and fit very snugly between them.

Re: who is this for.

My guess, someone who buys hyper-converged infrastructure today (e.g. Nutanix, VCE, etc) ... but that market is getting smaller and smaller by the day.

I wonder if they aim to target small operations and startups initially.

What about from an Integrator or VAR? Would you buy it then?

It’s an acquihire play. It might be possible to beat the hypers with $100 billion and some really good engineers. But building custom racks ain’t the angle to do it.

I really wonder what their mental image of a product market fit is. They're undeniably doing some cool stuff but myself and it seems like everybody else has to do some serious mental gymnastics to figure out who they sell this to and what needs it fulfills or niches it can fit into.

Thank you very much for the kind words! It has been important for us to do things the right way and to be a model for others -- so it's really meaningful for us to hear that that's appreciated; thank you!

Yeah, I don't honestly know much about the hardware side of things, nor the business/economic rationale for why one would prefer an Oxide rack over a more conventional setup, but reading through their posts, what they've done is just singularly so impressive that I really, really, really want them to succeed.

I've worked in startups most of my life, but it's tough feeling like you've got to take a bunch of shortcuts ("MVP"-it before you run out of runway, etc. etc.), so to see a startup just put a ton of real engineering prowess into a great product is a sight to behold.

Also, while there are obviously amazing products out there in the world, it's so often a "race to the bottom" that it feels like decently made products are just prohibitively expensive. I'm really into antique espresso machines, and there is a reason that seeing rebuilds of beautiful old lever espresso machines get so many oohs and ahhs online - they truly don't make them like they used to, so it's so cool to see these beasts brought back to life. For some reason I have a similar feeling looking at these Oxide racks - they just didn't cut corners to build an awesome machine. Again, not qualified to say whether that makes economic sense long term, but it's still a thing of beauty regardless.

> Shipping products is hard

And shipping full cabs is harder!

The industrial automation industry is highly risk adverse. In some systems, like safety, this mentality is justified without proper migration plans. In other areas, it's alarming, like choosing to run a system on hardware that has been out of support for 30+ years without adequate spare parts.

Operation/Shop floor technologies (OT) are treated like mechanical equipment, "When it fails, we'll swap it out for a new one." Well, this isn't a motor, it has programming, and it interfaces with I/O sensors and devices.

The main challenge is the lack of knowledge and skills in modern technologies among technical staff and decision makers in industrial organizations.

A an aside, in my early days I hopped on a 5-hour flight and drove 6 hours to replace a failed hard drive in a Windows NT machine used as an HMI. Then a year later, replaced it and all the other clients with a vSphere stack. The local resources, both internal and external, were too intimidated to touch it.

I'd be in favor of a "reset" in automation, it feels like fighting city hall.

I feel like a lot of people miss the point on PLC stuff.

“Stale PLC programming languages” might be stale and in need of a rework, but “rewrite in rust” entirely missed the value proposition. Memory safety is important, sure, but these systems aren’t usually ever allocating memory or taking actions that aren’t deterministic. PLCs usually operate in strict realtime environments. Like each operation contains a known number of clock cycles, and is timed to match physical “stuff” moving in the real world. To rewrite in rust, you’d need a flavor of rust where you can deterministically know the timing of ops and calibrate the timing of each branch of any control paths.

That said, there is a market for softer realtime PLCs and companies like Rockwell sell Windows IoT hardware- but they still contain a strict-realtime companion controller. Arduino is now selling a mountable PLC that can be programmed using their tools, but even they support traditional PLC programming languages.

It would be great if Oxide had something like Canonical's "Orange Box"/cloud-in-a-box for homelabs, evaluation, training (in the management bits) - and hobby work loads!

https://canonical.com/blog/jumpstart-training-with-the-orang...

https://arstechnica.com/information-technology/2014/06/hands...

I'd imagine they'll get to that eventually, these types of companies generally start at enterprise level because that's the most profitable and requires closing smaller numbers of deals. Once the product is proven and their support infrastructure is in place they can go for other market segments to try and maximize revenue

Even just a medium business offering would be great. I'd love to not have to use Dell or HP gear-- anything to get away from the cobbled-together stack of legacy IBM PC compatibility and third-party ODM/OEM stuff glue-and-taped together by the vendor.

Unfortunately they are not planning home lab things anytime soon, per a recent podcast episode [0].

If you want to play around with their Hubris OS: "You wanna buy an STM32H753 eval board. You can download Hubris, and then you’ve got – you’ve got an Oxide computer. You have it for 20 bucks.”

[0] https://changelog.com/friends/8

Same here!

I’ve not personally used it, but their stack of software is open source, and according to some commenters in the thread, super high quality.

Not 1U but perhaps a box design that isn’t noisy like a pizza box server.

Don’t know if oxide would want or be able to compete in the low cost market but a bigger a more expensive desktop/workstation as a mini homelab cloud could be a great option to get people trained on the oxide platform.

seconded. it would provide an on-ramp to get familiar with the software without forking over 500k

Manufacturing is done by Benchmark in Raleigh, NC. Its outsourced.

First rack shipped to costumer Jul 1, 2023: https://twitter.com/oxidecomputer/status/1674901883130114048

Here is a picture an incomplete rack: https://pbs.twimg.com/media/FfT7MHoUoAE90QZ?format=jpg&name=...

You can find other pictures on twitter and other places.

That comp number is apparently as-of 2021 and it was explained in a post by the CTO https://oxide.computer/blog/compensation-as-a-reflection-of-...

Certainly odd but not out of the ordinary for a small Bay Area start-up where the Founders have a ton of cash and the focus is mostly on what they personally want to do. Posts like these are b/c the Founders want to hire 1-3 people who fit exactly into alignment with them.

And they currently run a podcast weekly called "Oxide and Friends" where they talk about miscellaneous things in the software space.

The most recent few episodes have been about corporate open source and they've had excellent guests, like Kelsey Hightower. Definitely the best computer related podcasts out there. Bryan and Adam are great hosts and their humor is always a delight.

Added to my podcast player - Thanks!

In a similar vein, Embedded.fm is a great podcast for embedded SW. Though I bet Oxide's take on embedded rust is a lot different than the hosts of that show!

Not being familiar with SoftIron but I would imagine there can be more than one company working in a given niche? Why would it be surprising?

Thanks for pointing this out! I wasn't aware of SoftIron. I think its a big deal that there are two vendors on this path. And both seem to be doing it the right way. I think it makes the whole, stronger than the sum.

Yeah, I just skimmed through the reference docs for Hubris and was very impressed with what I saw. No "rocket science", just (apparently) solid technical decisions that are extremely well justified and documented.

https://oxidecomputer.github.io/hubris/reference/

> I'd like to add that their open source code is also EXTREMELY high quality.

* https://github.com/oxidecomputer

I agree. I stole some of their stuff from here https://github.com/oxidecomputer/third-party-api-clients/tre... when I needed a SendGrid integration. High quality code and proper use of Rust types.

> their open source code is also EXTREMELY high quality

I would expect nothing less from that crew. They did amazing things at Sun Microsystems, Inc. (RIP), and they continue to do even greater things now.

Tangent. In my admittedly limited experience, embedded code seems to have a tendency to be some of the worst code you can come across. The stuff is already low level and not the most easy to follow, but then embedded developers seem to despise names with more than two letters and a number. Without the datasheet at hand it is impossible to figure out what any of the code does because everything is just an abbreviation or acronym from some block or pin-out diagram.

The only big thing missing for now is their OS (or rather, their Illumos distribution), Helios[1].

Can't wait to read its source code, I was curious since reading this thread[2].

[1] https://github.com/oxidecomputer/helios (if this is not found for you, we're in the same boat)

[2] https://news.ycombinator.com/item?id=33337086

On a side note, I really liked their website. The ASCII animations are interesting -- wish there was a video game that had those.

Is that the same extremely high-quality open-source code that currently has a failing build?

https://github.com/oxidecomputer/hubris

And, that one underscore-delimited folder name in this repo just catches the eye, huh?

https://github.com/oxidecomputer/humility

> ... code is also EXTREMELY high quality. If you're an embedded developer go take a look at Hubris and Humility.

So, Humility is like MC/DC?

https://en.wikipedia.org/wiki/Modified_condition/decision_co...

Thought I had heard the cringiest random names for an oss project, and then...

They mentioned it really quickly in their Oxide an Friends podcast but, IIRC, prices start at $500k. Some of the audience asked if they were going to do a smaller configuration like half or quarter rack. And they said they were looking into it but weren't sure the of the business case.

Follow up question would be how much are equivalent solutions from established rack providers?

I would assume a million dollar. Ballpark.

https://0x1de.com redirects to https://oxide.computer so they did think of it.

You'll like https://admin.pci-ids.ucw.cz/read/PC/01de then.

You may be happy (or unhappy) to know that was definitely considered – it felt a little more trouble than it was worth, and dare I say a little too on the nose?

We did manage to get a neat PCI vendor ID: https://pcisig.com/membership/member-companies?combine=01de

"we needed a name — and once we hit on Oxide, we knew it was us: oxides form much of the earth’s crust, giving a connotation of foundation; silicon, the element that is the foundation of all of computing, is found in nature in its oxide; and (yes!) iron oxide is also known as Rust, a programming language we see playing a substantial role for us. Were there any doubt, that Oxide can also be pseudo-written in hexadecimal — as 0x1de — pretty much sealed the deal!"

from https://bcantrill.dtrace.org/2019/12/02/the-soul-of-a-new-co...

0x1de would be great branding for some kind of integrated hex editor (hence IDE)

Many of your questions have answers in TFM: https://docs.oxide.computer/guides/introduction

Thanks for stopping by! All Things Open is one of my favorite conferences, glad we were able to be there.

And yeah it's totally chill: this product is not for everyone. No slight taken!

I'll second that, they should make it clear what they're selling and what they have to offer. Every company should have this in their main/landing page... If I can't understand what you're selling/offering in 30sec, then I'm not interested :-)

If you take some time and read around their site you'll see that they're offering a ready to run (turnkey) server. They have everything packed together, they've integrated everything that is needed (CPU, disk, networking...) into a nice looking cabinet, with not too many wires and they're selling it as a complete package.

If you're in need of a server (cloud computer) and you don't want to but separate components and unpack them and connect them yourself, then this looks like a possible solution.

It appears to be an all in one, preconfigured/partially-or-completely assembled rack server for on-prem hosting, complete with pre-configured software.

Just my perception given the front page. I agree it's a bit vague.

They've reinvented the IBM Mainframe. Big rack-sized box with lots of redundant hardware that serves guest VMs. This is basically a zSeries in drag.

The key difference is the price structure -- IBM leases the hardware wherever they can get away with it, and uses a license manager to control how many of the machine's resources you can access (based on how much you can bear to pay). This, however, is like a mainframe you own.

The way I interpret it: an integrated stack of compute, storage and networking hardware and monitoring software that you can plug in to your data center (owned or colo) and can e.g. deploy Kubernetes to, and then use as a deployment target for your services.

The main USP: you own it, you're not renting it. You're not beholden to big cloud's pricing strategies and gotchas.

Secondary USP, why you buy this rather than DIY / rack computer vendor: it's a vertically integrated experience, it's preassembled, it's plug and play, compatibility is sorted out, there's no weird closed third party dependencies. Basically, the Apple experience rather than the PC experience.

It's a rack of servers you can buy for a lot of money and put in a datacenter. And once you plug it in and turn it on and do whatever setup is required, you're supposed to be able to spin up vms on it.

Here's a bit of history: https://bcantrill.dtrace.org/2019/12/02/the-soul-of-a-new-co...

The site is really hard to navigate. I eventually looked at the footer and found a link to the technical specifications https://oxide.computer/product/specifications They give an idea of the beast, especially the dimensions and weight

  Dimensions H x W x D
  2354mm (92.7") x 600mm (23.7”) x 1060mm (41.8")

  Weight
  Up to ~2,518 lbs (~1,145 kg)

It's a rack.

BTW, they are following some no pictures policy. I found only a few pictures of boards but no picture of the product as a whole.

The Oxide has everything you need to stand up your own private cloud, and everything works together down to the software. It's a great alternative to buying and managing servers, switches, storage, power management, KVMs, all of the software in between, and the tens of professional services contracts required to glue it all together.

It's the iPhone of hyperconverged infrastructure.

(Sorry; three sentences.)

Modern tech stack for managing data centers using tightly integrated hardware and software.

Designing a hardware-software combination to allow for the managing of compute (and networking and storage) to occur at the rack-level rather at the individual-device (server, switch, etc) level, so that large(r)-scale operators can manage cattle more easily (rather than herding cats/pets).

Also this. Looks technically great but how do you sell this to business? Whats the price range? Is this supposed to be an acceptable solution between on-prem and cloud as we start to realise the true costs of being hooked on cloud when your a large org and not a startup?

In a data center you have racks of computers performing all of the workloads. At this point these racks are fairly standardized in terms of sizing and ancillary features. These are built-out to solve the following:

    * Physical space - The servers themselves require a certain amount of room and depending on the workloads assigned will need different dimensions.  These are specified in rack "units" (U) as the height dimension.  The width is fixed and depths can vary but are within a standard limit.  A rack might have something like 44U of total vertical space and each server can take anywhere from 1-4U generally.  Some equipment may even go up to 6U or 8U (or more).

    * Power - All rack equipment will require power so there are generally looms or wiring schemes to run all cabling and outlets for all powered devices in the rack.  For the most part this can be run on or in the post rails and remains hidden other than the outlet receptacles and mounted power strips.  This might also include added battery and power conditioning systems which will eat into your total vertical U budget.  Total rack power consumption is a vital figure.

    * Cooling - Most rack equipment will require some minimum amount of airflow or temperature range to operate properly.  Servers have fans but there will also be a need for airflow within the rack itself and you might have to solve unexpected issues such as temperature gradients from the floor to the ceiling of the rack.  Net heat output from workloads is a vital figure.

    * Networking - Since most rack equipment will be networked there are standard ways of cabling and patching in networks built into many racks.  This will include things such as bays for switches, some of which may eat into the vertical U budget.  These devices typically aggregate all rack traffic into a single higher-throughput network backplane that interconnects multiple racks into the broader network topology.

    * Storage - Depending on the workloads involved storage may be a major consideration and can require significant space (vertical Us), power, and cooling.  You will also need to take into account the bus interconnects between storage devices and servers.  This may also be delegated out into a SAN topology similar to a network where you have dedicated switches to connect to external storage networks.

These are some of the major challenges with rack-mounted computing in a data center among many others. What's not really illustrated here is that since all of this has become so standardized we can now fully integrate these components directly rather than buying them piece-meal and installing them in a rack.

This is what Oxide has to offer. They have built essentially an entire rack that solves the physical space, power, cooling, networking, and storage issues by simply giving you a turn-key box you plant in your data center and hook power and interconnects to. In addition it is a fully integrated solution so they can capture a lot of efficiencies that would be hard or impossible in traditional design.

As someone with a lot of data center experience I am very excited to see this. It is built by people with the correct attitude toward compute, imo.

Weird there’s no mention of GPU at all when you’d think that’s what would prick up the ears of the hosting companies ….. stack a pile of GPUs in a rack, surely you can sell time on that.

The Oxide machines seem to be aimed at 2020.

Low risk of vendor lock-in.

Their pricing starts at $500K (but realistically will be $1-2M per order at a minimum). This is not intended for people who browse their website, click "buy" and fill in their credit card info. If you don't want to talk to a sales rep you were never their target customer, and I'm sure they aren't sweating the hypothetical lost sale.

Their entire point is to be different then that - if you want servers just go to HPE and spec out some servers. (Which really does beyond small scale, but at the rack level it's unheard of)

If you think you pay a fair price buying racks of servers from dell or hp by clicking buttons on their website, you are missing out. Purchase teams in my first company buying all sorts of hardware for our datacenters would spend their days negotiating and the end price was nowhere near the price listed online.

I feel the same. I'm also amazed at how much traction this is getting and I'm skimming through the comments to understand why this is so important. Pics would have helped.

This podcast on the backplane has the best set of photo's I've found so far:

https://oxide.computer/podcasts/oxide-and-friends/1281318

Just marketing fluff. It's a rack mounted server. Nothing "world's first" about it.

They're selling a rack that hosts an entire virtual cloud. Nobody has done that yet. There have been servers with hypervisors preinstalled, but nothing like this.

Great questions. We are careful about new service exposure -- object storage certainly makes sense (though having built one in the past, very mindful about trade-offs between cost/performance/availability/durability) but some higher level services may not (do not expect an Elastic Beanstalk equivalent!). There's a big gray area in the middle, where we will be customer- and demand-driven.

On accounting: yes! This is something that we have designed abstractions for from the start with respect to tenancy with our silo and project abstraction[0], and we are actively adding resource allocation and monitoring; stay tuned!

[0] https://docs.oxide.computer/guides/key-entities-and-concepts

> The irony is that Oxide caters to the HN crowd, but the HN crowd would not be the buyer of their offerings.

Rust, Oxide, and iron-y.

Depends. HN crowd will likely prefer this over existing ones in their day job.

You run VMs on their platform, so it doesn't lock you in more than any cloud host locks you into their provisioning interface.

The VMs are bhyve on illumos. They're about as hard to move off as moving a datacenter sized Proxmox cluster to anything else.

People raise those sorts of numbers for web apps all the time. It actually doesn’t sound that big for such a broad hardware and software company with long and high touch sales cycles ahead of them.

> People might just look at it and say "It's just a rack of CPUs and storage." And I can only imagine just how much you might be tempted to throttle them

Lol. Bro.

$10 per year, minus credit card fees and taxes. If you send a single email to customer support in 5 years the company already loses money. The company can't just sell you the HDD and put a warning in the FAQ that you are responsible for data loss. Because that will result in angry emails and bad reviews.

How many engineers and how many years does it take to build the infrastructure, write all the software, and deal with all the hardware to provide a photo service that "just works"? How many customers do they need before they break even? And obviously they can't raise VC because a business model that is predicated on making basically no money per customer can never have an exit. How would you bootstrap such a thing?

It feels strange that physical storage costs almost nothing on amazon but the storage attached to a cloud machine is expensive. But run the numbers and you'll see that running a cloud service isn't about hardware costs at all.

I guess I don't understand how that's a radical statement if colocation was a thing even before you could rent cloud resources.

This is interesting as I think Intel have stopped developing Tofino, after buying Barefoot networks back in 2019.

AMD recently purchased Pensando, I'm not sure if their network chips are similar to Tofino but they seem like they are P4 compatible DPUs so they might be a good choice when migrating off of the dead Tofino platform.

So if anything Oxide will be moving further away from Intel!

I would love to work on this kind of stuff, not sure how to get started to end up working somewhere like Oxide..

Because they can't. Their product currently has zero GPU compute.

> This is too startup-y and niche and too small to attract enough companies on the market.

From their press release:

> Oxide customers include the Idaho National Laboratory as well as a global financial services organization. Additional installments at Fortune 1000 enterprises will be completed in the coming months.

* https://www.bloomberg.com/press-releases/2023-10-26/oxide-un...

From an interview:

> And what was the reaction of lucky rack customer #1? “I think it’s fair to say that the customer has appreciated the transformationally different (and exponentially faster!) process of going from new rack to provisioned VMs.”

> The same day Cantrill appeared as a guest on the Software Engineering Daily podcast.[1] (“The crate, by the way? Its own engineering marvel! Because to ship a rack with the sleds, it’s been a huge amount of work from a huge number of folks…”) Cantrill wouldn’t identify the customer but said that “Fortunately when you solve a hard problem like this and you really broadcast that you intend to solve it… Customers present themselves and say, ‘Hey, we’ve been looking for — thank God someone is finally solving this problem’.”

* https://thenewstack.io/in-pursuit-of-a-superior-server-oxide...

Niches can be profitable. Not every company has to be "web scale".

> I know a 4bn/y revenue company that could greatly benefit from this but they will never even consider buying from a company like this.

What does "like this" mean? Small? Every company starts out that way.

I think its cool that someone is trying to do startup hardware things, as in recent years it seems like its mostly software. I don't a use for the product in my area of IT, but I wish them luck.

The value proposition might be worth the downside risk

"Microserver" designs have failed several times. AMD Epyc is basically twelve 7840s in an even smaller space and you can partition it any way you want.

Oxide is a bit of a darling child on HN - I think a lot of people (myself included) find it interesting that they are trying to re-engineer something somewhat commoditized and boring (servers) into something cool

It sort of reminds me of a Dell blade chassis that I pulled out of a DC last year - it was pretty impressive tech that I hadn't dealt with. 6 power supplies that I think could take 110v or 220v, but you needed 220v for full capacity. All of them were hot-swappable and as long as you had 3 powered it would keep running. The compute sleds were modular and it also had integrated networking as mentioned in this article. All obviously crafted to a very high standard; even the plastic fan cages felt premium (and I'm told it cost accordingly back in 2010 or so).

Also a former employer who was a bit below the hyperscaler level also had what they called "roll-in racks", though I believe they were just taking standard servers and networking gear and integrating them somewhere (presumably with cheap labor) and then bringing them into the DC as needed.

So I don't think it's a particularly new concept, but I agree it looks like it has some potential as a product.

It seems like they have managed to fit 32 1P servers in a 42U rack. That's pretty impressive, I have to admit. I don't think I've seen that level of density before.

Oxide racks curren use AMD CPUs and and after buying Pensando AMD now offer P4 DPUs. I would hazard a complete guess and say maybe a move from Tofino to Pensando is on the cards for future networking revisions.

If you are genuinely in the market for multiple racks of servers you (a) know how much a rack of hp/dell gear costs, which gets you within an order of magnitude of what this is going to cost, and (b) would not buy one of these without a sales call even if you could.

Weird criticism, for anything involving CapEx it will always require an RFQ and quote.

You're not buying a bag of peanuts and they will likely price it differently if you're buying 1x or 500x.

To be fair, if you’re in the market for a system like this, you’re probably not paying rack rate for AWS or Azure, either.

Sun certainly sold computers, yes.

I'm not sure what "weeks of overhead" you're referring to, specifically, as it could mean a few different things. Happy to elaborate if you will :)

There's two independent issues: capex vs. opex, and flexibility.

When opex is a primary benefit of the cloud, that's specifically for start-ups and other businesses that have little working capital. The actual primary benefit of the cloud for most cloud customers is flexibility - prior to AWS adoption, getting a server provisioned could be a multi-week if not multi-month affair negotiating between devs or operators and on-prem infrastructure workers to get the servers through capacity planning and provisioning. AWS made provisioning so simple, that you could start to set up auto-scaling, because you had extraordinarily high confidence that the capacity would be immediately available when your autoscaler tried to scale up.

But opex is not a benefit of the cloud for heavy/established businesses. Cloud opex is a financial expenditure every month that you can't get rid of and counts directly against your profits. Indeed, the desire for capex in the cloud is so high that businesses routinely purchase Reserved Instances and other forms of committed usage, which allows accountants to treat the cost of the RI as capex and then discount the expenditure through depreciation (to zero, since there will be nothing to sell when the RI expires) over the lifetime of the RI. It is normal and frequent for businesses to make capital expenditures to reduce their operational expenditures over time, thus increasing their monthly/quarterly profits.

Oxide's unique value proposition is to give customers, particularly those with high monthly cloud bills that they have difficulty reducing, the operational flexibility of cloud computing with the profit-improving benefits of capital expenditures.

That was the line we were sold by cloud vendors. There is a reason that significant portions of both native cloud and migrated workloads are shifting back to self-hosted; the cost savings never materialized.

My google-fu failed to find any articles for or against my statement that weren’t paid advertising or lightweight tech summaries. StackOverflow will have to do. https://stackoverflow.blog/2023/02/20/are-companies-shifting...

Private Cloud allows infrastructure team to run big server parks efficiently, while product teams can "buy" resources easily. It's essentially why Amazon made aws, why Google made Borg.

For a regional hospital for example, there might be a desire for in-house network and hardware - but perhaps the system for digital patient journals run on Kubernetes and managed databases.

Ya, in the past the messaging was more like "get the ergonomics of the cloud with the economics of owning your own hardware", I think it's confusing to just call it a "cloud computer"

Private cloud is a thing too: lots of folks run VMware, Hyper-V, or OpenStack in-house.

Sure but some businesses care about how much they spend and then about capex vs opex.

I'd rather have $1 in capex than $10 in opex

If they lease it to companies, would it be considered opex?

1. Not yet, we’ll see what the future holds

2. Eventually, right now we are just starting, so working closely with customers and in small quantities, but eventually that is the idea.

3. No. We are already doing very well with fans, the design means a very tiny power draw. The racks are quite quiet.

4. No.

5. Selling hardware and running a private cloud are two different businesses. The public clouds don’t also sell servers. We don’t plan on running a cloud, even if doing so would be a nice experience, we have to focus on what we’re doing.

I don't know enough about Azure Stack HCI, though I expect it's similar. For AWS Outposts you can't actually own the hardware. It's hardware leasing on your own premises. More so, it's a black box that you can't inspect, while Oxide's firmware is all open source. You can even flash your own modified firmware without going through Oxide.

iirc azure stack hci is only 1U and 2U configurations, not a full 32U rack.

I wrote this a while back, it's still true: https://news.ycombinator.com/item?id=30678324

Game development and desktop applications aren’t infrastructure.

That is an odd bug I have seen before with a browser (not sure when or when). It freaked me out back then though.

Open it in a private window and see if you can reproduce. By default, a private window uses no extensions though I prefer to have uBO enabled by default there, too.

Another peculiar detail I saw on landing page is screenshot of management interface. It must be old as it shows various VMs including FreeBSD 12.1 and Ubuntu 20.10. Ubuntu 20.10 is EOLed long ago and also wasn't a LTS release. Not something you wanted running in enterprise 3 years ago, nevermind recently.

I saw the same thing and wondered why nobody in the comments mentioned how painful this site was to read. Copy/paste the text and suddenly it's all back to normal. Saved the page to disk and loaded the local copy in the browser - also perfectly normal. Very strange. Maybe something to do with their CSS?

Can't repro in FF with JS off (not exactly NoScript). Could it be an extension doing something weird? Curious if you see normal text when you view source on the page.

Not sure about the tivoization. You could never send a pull request to the Tivo repo on Github, because it does not exist. And maybe the Oxide hardware allows you to change the certificates so you can roll your own?

I suppose we need to know if it's secure boot (tivoization), or verified boot (remote attestation).

The hypervisor OS is based on Illumos, which was forked from OpenSolaris, and it uses Bhyve from FreeBSD for virtualization.

I would imagine the system architecture is different enough that running Linux as the base OS would take some work, for example it doesn't have an AMIBIOS.

> Some say that you can end up using unsafe a lot and so it’s better to stick with C or even use Zig

Using unsafe in Rust isn't inherently bad. If you're doing embedded work, using unsafe is mandatory once you get to the level of interacting with the hardware.

That doesn't mean the code is literally unsafe, just that the interactions are happening in a way that the compiler can't guarantee. That's completely expected when you're poking at hardware registers.

You still get most of the benefits of Rust, the language. I've had good success with it.

No, there's just a lot more that can be said than they wanted to fit on this one page.

Specs of the switch are available here: https://oxide.computer/product/specifications