undefined | Better HN

0 pointsebingdom4y ago0 comments

Thanks for the clarification. But now I'm confused about the limits:

> 3,500 PUT/COPY/POST/DELETE requests per second per prefix

> 5,500 GET/HEAD requests per second per prefix

Most of those APIs don't even take a delimiter. So for these limits, does the prefix get inferred based on whatever delimiter you've used for previous list requests? What if you've used multiple delimiters in the past?

Basically what I'm trying to determine is whether these limits actually mean something concrete (that I can use for capacity planning etc.), or whether their behavior depends on heuristics that S3 uses under the hood.

I'm fine with S3 optimizing things under the hood based on access my patterns, but not if it means I can't reason about these limits as an outsider.

0 comments

Macha4y ago

S3 does a lot of under the hood optimisation. e.g. Create a brand new bucket, leave it cold for a while, and start throwing 100 PUT requests a second at it. This is way less than the advertised 3500, but they'll have scaled the allocated resources down so much you'll get some TooManyRequests errors.

acdha4y ago

Those are what I would assume for performance when the system is stable. The concerns come from bursty behaviour — for example, if you put something new into production you might have a period of time while S3 is adjusting behind the scenes where you'll get transient errors from some operations before it stabilizes (these have almost always been resolved by retry in my experience). This is reportedly something your AWS TAM can help with if you know in advance that you're going to need to handle a ton of traffic and have an idea of what the prefix distribution will be like — apparently the S3 support team can optimize the partitioning for you in preparation.

ec1096854y ago

Delimiter isn’t used for writes, only list operations.

S3 simply looks at the common string prefixes in your object names and uses that to internally shard objects, so you can achieve a multiple of those request limits.

aaa122348

aaa484585

bbb484858

bbb474827

Would have same performance as:

aaa/122348

aaa/484585

bbb/484858

bbb/474827

j / k navigate · click thread line to collapse