There are very cool things you can do today without too much complexity with the primitives that modern database are capable of. Should one rebuild it for their startups, no, but if you were to extract the very core of the durable execution engine of windmill for instance, it would actually be surprisingly reasonable given that postgresql does the heavy lifting. I strongly believe the benefits of our platforms mostly come from the overall virtue to be standardized, opinionated and working out-of-the-box in a way that make everything fit together rather than the overall engineering complexity of it.
As for the article, I think the main point being driven here is:
"building a system with queues requires much more than just the queue itself"
I would imagine almost anyone who has built a production grade queue-based message processing system would agree.
For me, I would say that for the majority of software being developed, the investment to build all of that yourself just doesn't make sense. Obviously there will be exceptions, but Inngest gives you incredible power at a very simple layer of abstraction.
We went from ESB's because the were opaque. And this seems like it's very opaque.
All that work that you put into queues... its ugly but it's transparent. You can rationalize it, you're not abstracting things away into magic.
If your early to mid-stage company needs queues to depth then what is in them is vital to your business.
You better know what is going on with mission critical data at every step of its journey. Or the data isnt that important and you should write something lighter than this because it can be lossy...
> They offer reliability through guaranteed delivery, persistence, and dead letter queues, so developers know they aren't sending workloads into a black hole.
I disagree with this reason to use queues. If this is the only reason for using SQS or RabbitMQ or similar, perhaps the application is over-engineered.
If you want reliability, and that alone, use a transaction-based system.
There are many tools in the toolbox for backend infrastructure: nosqls (memcache/redis/keydb), dlms (zk), kafka, rabbitmq, ejabberd, 0mq, nng. Some scale better than others, and some are more atomic or durable than others. OLTP and infrastructure orchestration will have different needs. Sometimes, cross-cutting concerns can be added by gating the sender, receiver, or both with "controller"-like middleware proxies or modifications.
"Use good judgement" is the prime directive.
One catch is that you're going to have to version your code if your workflows/orchestrations run for days, or if there's no windows without running workflows. And there's no built-in support for this, so expect to duplicate your entire workflow for new versions, so the old one can run to the end with the old code.
