This feels like one of those Apple-like stories about inventing and discovering an amazing, brand new feature that delighted customers but not mentioning the motivating factor of the competing products that already had it. A more honest sentence might have been "After years of customers complaining that the other major cloud storage providers had strong consistency models, customers were relieved when we finally joined the party."
Imagine my surprise when porting some GCS code to S3 last year and realizing there is no way to get consistency guarantees without external lock service.
Where do you keep hearing this?
Having used both, AWS is trash in comparison. It's way to complicated to do anything simple. At work I wish we could migrate to GCP (or just something that's not AWS, really).
One of the things that shocks me about the system is the level of object durability. A few years ago I was taking an AWS certification course and learned that their durability number means that one can expect to loose data about once every 10,000 years. Since then anytime I talk about S3's durability I bring up that example and it always seems to convey the point for a layperson.
And it's "simplicity" is truly elegant. When I first started using S3 I thought of it as a dumb storage location but as I learned more I realized that it had some wild features that they all managed to hide properly so when you start it looks like a simple system but you can gradually get deeper and deeper until your S3 bucket is doing some pretty sophisticated stuff.
Last thing I'll say is, you know your API is good when "S3 compatable API" is a selling point of your competitors.
Counter-point: You know that you're the dominant player. See: .psd, .pdf, .xslx. Not particularly good file types, yet widely supported by competitor products.
Every file format accumulates cruft over thirty years, especially when you have hundreds of millions of users and you have to expand the product for use cases the original developers never imagined. But that doesn’t mean the success wasn’t justified.
A very popular file format pretty much defines the semantics and feature set for that category in everyone's mind, and if you build around those features, then you can probably expect good compatibility.
Nobody thinks about the actual on disk data layout, they think about standardization and semantics.
I rather like PDF, although it doesn't seem to be well suited for 500MB scans of old books and the like, they really seem to bog down on older mobile devices.
It’s by no means impossible to do that yourself, but it costs a lot more in time and upfront expense.
Of course, I lost that script when the server crashed, the one thing I didn't back up properly.
Over the years working on countless projects I’ve come to realize that the more “easy” something looks to an end user, the more work it took to make it that way. It takes a lot of work to create and polish something to the state where you’d call it graceful, elegant and beautiful.
There are exceptions for sure, but often times hidden under every delightful interface is an iceberg of complexity. When something “just works” you know there was a hell of a lot of effort that went into making it so.
What does that mean? If I have 1 million objects, I loose 100 per year?
If you had light bulbs that lasted 1,000 hrs on average, and you had 10k light bulbs, and turned them all on at once, then they would all last 1,000 hours on average. Some would die earlier and some later, but the top line number does not tell you anything about the distribution, only the average (mean). That’s what MTTF is; the mean time for a given part to where it has a greater likelihood to have failed by then vs not. It doesn’t tell you if the distribution of light bulbs burning out is 10 hrs or 500 hrs wide. it’s the latter, you’ll start seeing bulbs out within 750 hrs, but if the former it’d be 995 hrs before anything burned out.
If you have ten million objects, you should lose one every 10k years or so.
What do you see as special for S3? Isn’t it just another bucket?
This surprises me; has anyone done something similar and benefitted from it? It's the sort of thing where I feel like you'd maybe get a result 1% of the time if that, and then only years later when everyone has moved on from the problem they were facing at the time...
I'm going to write a script that keeps trying to call ecs:MakeFargateCacheImages.
Maybe such solutions are a reaction to previous more "complex" solutions, and they do indeed start simple, but inevitably get swallowed by the complexity monster with age.
If TFTP had gone the way of SNMP, we'd have 'tftp <src> <dest> --proto tcp --tls --retries 8 --log-type json' or some horrendous mess like that.
Most people don't even think about it. But authenticating trillions of operations per second is why AWS works. And the accounting and billing. Anyone that does authentication knows how hard it is. At AWS' scale it's well, the pinnacle of distributed systems.
It's only sad that the SDKs are often on the heavy side, I remember that the npm package used to be multiple megabytes as it bundled large parts of the core AWS SDK. Nowadays, I believe it's still around half a megabyte.
It's essentially just the 4 functions of C.R.U.D done to a file.
Most problems in tech are not that simple.
Note: not knocking the service. just pointing out not all things are so inherently basic (and valuable at the same time).
If you don't believe me, you might want to reconsider how skilled the average developer _really_ is.
A basic implementation is simple, but at S3 scale, that's a whole different ball game.
No problem. I’m sure ChatGPT could cook up a replacement in a weekend. Like Dropbox it’s just rsync with some scripts that glue it together. How hard could it possibly be?
I mean people serve entire websites right out of s3 buckets. Using it as a crude CDN of sorts.
It’s a modern marvel.
...is very difficult.
S3 is an engineering marvel, but it's an insanely complicated backend architecture just to store some files.
One server with a RAID array can survive, usually, 1 or maybe 2 drive failures. The remaining drives in the array will have to do more work when a failed drive is replaced and data is copied to the new array member. This sometimes leads to additional failures before replacement completes, because all the drives in the array are probably all the same model bought at the same time and thus have similar manufacturing quality and materials. This is part of why it's generally said that RAID != backup.
You can make a local backup to something like another server with its own storage, external drives, or tape storage. Capacity, recovery time, and cost varies a lot across the available options here. Now you're protected against the original server failing, but you're not protected against location-based impacts - power/network outages, weather damage/flooding, fire, etc.
You can make a remote backup. That can be in a location you own / control, or you can pay someone else to use their storage.
Each layer of redundancy adds cost and complexity.
AWS says they can guarantee 99.999999999% durability and 99.99% availability. You can absolutely design your own system that meets those thresholds, but that is far beyond what one server with a RAID array can do.
Standard RAID configurations can only handle 2 failures, but there are libraries and filesystems allowing arbitrarily high redundancy.
Much like twitter it's conceptually simple but it's a hard problem to solve at any scale beyond a toy.
Some details in https://www.allthingsdistributed.com/2023/07/building-and-op... / https://www.youtube.com/watch?v=sc3J4McebHE
The long and short of it is that getting anywhere near the redundancy, reliability, performance etc of S3, you're spending A Lot of money.
They also already have support for storage tiering, replication, encryption, ACLs, integration with other services (from web access to sending notifications of storage events to lambda, sqs, etc). Uou get all of this whether you're saving 1 eight bit file or trillions of gigabyte sized ones.
There are reasons why you may need to roll your own storage setup (regulatory, geographic, some other unique reason), but you'll never be more economical than S3, especially if the storage is mostly sitting idle.
It's more costly immediately. S3 storage prices are above what you would pay even for triply redundant media and you have to pay for data transfer at a very high rate to both send and receive data to the public internet.
It's far less complicated though. You just create a bucket and you're off to the races. Since the S3 API endpoints are all public there's not even a delay for spinning up the infrastructure.
Where S3 shines for me is two things. Automatic lifecycle management. Objects can be moved in between storage classes based on the age of the object and even automatically deleted after expiration. The second is S3 events which are also _durable_ and make S3 into an actual appliance instead of just a convenient key/value store.
I’ve paid pennies a year to store data in s3 for the better part of 5 years. Can’t even buy a hdd with what I’ve spent on s3.
It's always crazy to me that people will run a could be static site on Netlify/Vercel/etc.
Between the above philosophy, S3 Tables, and Express One Zone (SSD perf), it makes me really curious about what other storage modalities S3 moves towards supporting going forward.
But S3 Select is nearly way cheaper that the new iceberg support. So if your needs are only for reading one parquet snapshot, we no need to do updates, then this change is not welcome.
Great article though, and I was pleased to see this at the end:
> We’ve invested in a collaboration with DuckDB to accelerate Iceberg support in Duck,
I can't make head or tails of the beginning of this sentence:-
> Pricing for S3 Tables is all and all not bad.
Otherwise lovely article!
So they are saying the pricing is not bad considering everything it does
> + 1,000,000 GET requests x ($0.004 / 1,000 requests) = $9
I would hope that S3 could introduce a change in the operation of said fundamental building block (immutable object), rather than just slap an existing downstream abstraction. That's not what I call design for simplicity. As an external observer, I would think that's internal amazon moat management with some co-branding strategy.
Anyone willing to give a cliff notes about what's good about it?
I've been running various sites and apps for a decade, but have never touched S3 because the bandwidth costs are 1, sometimes 2 orders of magnitude more expensive than other static hosting solutions.
The biggest is, if I need to store some data. I know what the data is (so I don't need to worry about needing to in a moment notice traverse a file structure for example, I know my filenames), I can store that data, I don't need to figure out how much space I need ahead of time and it is there when I need it. Maybe it automatically moves to another storage tier to save me some money but I can reliably assume it will be there when I need it. Just that simplicity alone is worth a lot, I never need to think later that I need to expand some space, possibly introducing downtime depending on the setup, maybe dealing with partitions, etc.
Related to that is static hosting. I have ran a CDN and other static content out of S3 with cloud front in front of it. The storage cost was almost non existent due to how little actual data we were talking about and only paid for cloudfront costs when there were requests. If nothing was being used it was almost "free". Even when being used it was very cheap for my use cases.
Creating daily inventory reports in S3 is awesome.
But the thing that really is almost "magic" once you understand its quirks. Athena (and quick sight built on top of that and similar tools). The ability to store data in S3 like inventory reports that I already mentioned, access logs, cloud watch logs, or any structured data that you may not need to query often enough to warrant a full long running database. It may cost you a few dollars to run your Athena query and it is not going to be super quick, but if you know what you're looking for it is amazing.
Need to save a file somewhere? Dump it in S3. It's generally affordable (obviously dependent on scale and use), fast, easy, and super configurable.
Being able to expose something to the outside, or with a presigned URL is a huge advantage as well.
Off the top of my head, I think of application storage generally in this tier ordering (just off the top of my head based on the past few years of software development, no real deep thought here):
1. General application data that needs to be read, written, and related - RDBMS
2. Application data that needs to be read and written fast, no relations - Redis
3. Application data that is mostly stored and read - S3
Replace any of those with an equivalent storage layer.
I've used it lately in the following setup
- stuff on internet push their data into my bucket => this is mostly free (you only pay s3 operations)
- on object creation, an event is fired and a lambda is spawned => this is free
- the lambda code reads the object => this is mostly free (again, you only pay s3 operations)
- the lambda process the data, trim it, repackage it, compress it => you pay for the compute
- the lambda store the resulting data somewhere else and delete the s3 object => you pay the egress
For static hosting, it's fine, but as you say, it's not necessarily the cheapest, though you can bring the cost down by sticking a CDN (Cloudflare/CloudFront) in front of it. There are other use cases where it really shines though.
[1]: I say ongoing basis because you will need to figure out your security controls, etc. at the beginning so it's not totally no-thought.
I avoid getting tangled in authentication mess by simply naming my files using type4 GUIDs and dumping them in public buckets. The file name is effectively the authentication token and expiration policies are used to deal with the edges.
This has been useful for problems like emailing customers gigantic reports and transferring build artifacts between systems. Having a stable URL that "just works" everywhere easily pays for the S3 bill in terms of time & frustration saved.
Yeah, we can dump those to a filesystem. But then we have to ask which filesystem? What should the directory layout be? If there are millions or billions of objects, walking the whole tree gets expensive. Do we write a script to clean everything up? Run it via cron or some other job runner?
With S3, you just write your artifact to S3 with a TTL and it gets deleted automagically when it should. No cron jobs, no walking the whole tree. And you can set up other lifecycle options if you need it moved to other (cheaper) storage later on, backups, versioning, and whatever else.
For on-prem, you have Minio, Garage, or SeaweedFS. These are pretty nice to deploy the servers however you need for the level of reliability/durability you require.
Didn't Microsoft blame their customers for not running updates and using AV? Gosh I guess MS did a fine job. Customers were to blame.
Just like those s3 buckets. Customers ignored the warnings ...
There’s legitimate use cases for making a bucket public.
What are you advocating for?
A good model protects stupid people from themselves.
A password model that says "you MUST have a strong password" protects stupid people from their own stupidity. A model that says you can use bad passwords if you click through warnings is a shite model. Stupid people ignore warnings.
Seriously.
Didn't SQS launch publicly earlier than S3?
AWS typically considers the "GA" milestone as the "public launch" date, which is silly because the criteria for what is good enough for GA has changed over the years.
I agree, things before S3 weren't really that different. LAMP stacks everywhere, and developer skills were very portable between different deployments of these LAMP stacks. Single machines didn't scale as much then, but for most small-medium sites they really didn't need to.
https://aws.amazon.com/blogs/aws/introducing-queryable-objec...
The S3 API is not "simple."
Authentication being a good part of that.
Initially I thought this was about S3 standby mode.