undefined | Better HN

0 pointsicedchai8y ago0 comments

Do you have any evidence for these claims of low durability? (None of those issues, except for bugs in the service, would count against them, by the way.)

0 comments

skuhn8y ago

I think factors beyond the storage algorithm are pretty important to consider when thinking about storing data that's important to your business. To your specific point though:

1. Amazon claims 99.999999999% durability of objects over a year.

2. I store 1EB of data with an object size of 4MB for a year (so 250,000,000,000 objects).

3. I can expect to lose 250 objects in a year, or 1GB.

Now to my experience:

I have stored in excess of that amount of data in S3. I have lost considerably more data -- solely because of data losses internal to S3 -- that these numbers would suggest. It was a tolerable amount of data loss, I didn't curse Amazon's name or swear vengeance, but it was definitely not 1 gig.

The standard S3 SLA provides credits only based on uptime. There is no mention of durability whatsoever. That tells you that Amazon is not willing to put their money where there mouth is on their 99.999999999% durability claims. The reality is the number is a design target, not an operational guarantee.

lozenge8y ago

AFAIK, S3 only provides notification when reduced redundancy objects are lost, not regular objects. How did you detect your data loss?

skuhn8y ago

All of my objects used standard redundancy. My recollection is that regardless of object class, you will get a 405 error if you try to fetch an object that has been lost.

I didn't use SNS notifications at the time (which might only work for reduced redundancy).

So that left two options: find out when attempting to fetch the object, or run bookkeeping jobs against the object catalog to periodically spider the data and ferret out any objects that are lost.

The second option may be a tad nicer, but it is also more complex and more expensive and the end result is the same either way.

j / k navigate · click thread line to collapse

0 comments

skuhn8y ago

I think factors beyond the storage algorithm are pretty important to consider when thinking about storing data that's important to your business. To your specific point though:

1. Amazon claims 99.999999999% durability of objects over a year.

2. I store 1EB of data with an object size of 4MB for a year (so 250,000,000,000 objects).

3. I can expect to lose 250 objects in a year, or 1GB.

Now to my experience:

lozenge8y ago

AFAIK, S3 only provides notification when reduced redundancy objects are lost, not regular objects. How did you detect your data loss?

skuhn8y ago

All of my objects used standard redundancy. My recollection is that regardless of object class, you will get a 405 error if you try to fetch an object that has been lost.

I didn't use SNS notifications at the time (which might only work for reduced redundancy).

So that left two options: find out when attempting to fetch the object, or run bookkeeping jobs against the object catalog to periodically spider the data and ferret out any objects that are lost.

The second option may be a tad nicer, but it is also more complex and more expensive and the end result is the same either way.

j / k navigate · click thread line to collapse