undefined | Better HN

0 pointsRhodesianHunter7y ago0 comments

If your job is easily parallelizable then you can run multiple lambdas in parallel. For the above use case they probably should have kicked off one lambda per prefix or similar.

0 comments

djhworld7y ago

That's exactly what we were doing. 1 Lambda to download and aggregate all files under a prefix.

The problem was the task just couldn't complete in < 5 minutes.

manigandham7y ago

You have to fan out further then, process each file separately and aggregate the aggregates, using SQS or something else to queue up the processing.

Azure's Durable Functions have an advantage here in making extreme fan-out situations easy.

djhworld7y ago

We considered it, but at the time we just felt implementing map/reduce over Lambda would just introduce a more complex architecture for such a simple problem.

Maybe the recently introduced SQS->Lambda support might make it a bit cleaner, but in the end we opted for EC2.

j / k navigate · click thread line to collapse

0 comments

djhworld7y ago

That's exactly what we were doing. 1 Lambda to download and aggregate all files under a prefix.

The problem was the task just couldn't complete in < 5 minutes.

manigandham7y ago

You have to fan out further then, process each file separately and aggregate the aggregates, using SQS or something else to queue up the processing.

Azure's Durable Functions have an advantage here in making extreme fan-out situations easy.

djhworld7y ago

We considered it, but at the time we just felt implementing map/reduce over Lambda would just introduce a more complex architecture for such a simple problem.

Maybe the recently introduced SQS->Lambda support might make it a bit cleaner, but in the end we opted for EC2.

j / k navigate · click thread line to collapse