Minimizing S3 API costs with distributed MMAP (opens in new tab)

(warpstream.com)

17 pointsrichieartoul2y ago3 comments

3 comments

refset2y ago

This is a really interesting design - kudos.

> Compacting the data may sound expensive, but in practice it's highly efficient since it only needs to happen once

Is there any handling for tombstone records?

ryanworl2y ago

[WarpStream co-founder]

WarpStream doesn't implement compacted topics today. It is on our roadmap, though. Compacted topics are typically not used in high-throughput workloads, so our plan is to delay compactions for longer than a disk-based system would to trade space amplification for write amplification.

refset2y ago

Thanks for clarifying.

> Compacted topics are typically not used in high-throughput workloads

TIL, but it makes sense. Compaction/retention policies certainly introduce a lot of extra tradeoffs dimensions.

j / k navigate · click thread line to collapse

3 comments

refset2y ago

This is a really interesting design - kudos.

> Compacting the data may sound expensive, but in practice it's highly efficient since it only needs to happen once

Is there any handling for tombstone records?

ryanworl2y ago

[WarpStream co-founder]

refset2y ago

Thanks for clarifying.

> Compacted topics are typically not used in high-throughput workloads

TIL, but it makes sense. Compaction/retention policies certainly introduce a lot of extra tradeoffs dimensions.

j / k navigate · click thread line to collapse