Too many people end up with their own half rolled pubsub via things like grpc, and they'd be far better off using this, particularly in the early stages of development.
It's an infinitely more friendly version of Kafka, pub/sub, etc that is extremely lightweight. Yet every environment trends towards Kafka because it was "chosen" by the big corps.
Edit: spelling
(for more detail check out this and the other YouTube videos on the Synadia channel https://youtu.be/WH55czo1BNk)
It's nothing to do with these apps being superior in any way - often it is the opposite - like reasonable faults and glitches can add more hours to bill the customers for "fixing".
RabbitMQ is a "classical" message broker. It routes messages between queues. Messages are treated like little letters that fly everywhere. They're filed in different places, and consumers come by and pick them up.
Core NATS isn't really a message broker, but more of a network transport. There are no queues as such, but rather topologies of routes where messages are matched from producers and consumers through a "subject". You don't "create a queue"; you announce interest in a subject (which is a kind of path that can contain wildcards, e.g. "ORDERS.us.nike"), and NATS routes stuff according to the interests. So there's nothing on disk; and if a consumer isn't there to receive a message, the message is gone. Thus you can send messages back and forth, both point-to-point or one-to-many. NATS itself isn't reliable, but you can build reliable systems on NATS.
A common example of the lightweight, ephemeral nature of NATS is the request-reply pattern. You send out a message and you tag it with a unique reply address, the "inbox subject". The subject is just a random string (it may be called "INBOX.8pi87kjwi"). The recipient replies by sending its reply to that inbox. The inbox isn't something that exists; it's just a subject temporarily being routed on. So the sender sends a message and waits for the reply. NATS encourages you to use these ephemeral subjects as much as possible, and there can be millions of them. You can do RPC between apps, and that's a popular use of NATS.
JetStream is a subsystem built on core NATS, and is what you get when the designer of NATS thinks he can outsmart the designers of Kafka. JetStream is basically a database. Each stream is a persistent sequential array of messages, similar to Kafka topics or a RabbitMQ queue. A stream can be replicated as well as mirrored; one stream can route into another, so you can have networks of streams feeding into bigger rivers. Unlike core NATS, but similar to RabbitMQ, streams and their consumers have to be created and destroyed, as they are persistent, replicated objects that survive restarts.
Similar to Kafka, streams are just indexed arrays; you can use it for ephemeral events, or you can store long histories of stuff. Consumers can go back in time and "seek" through the stream. Streams are indexed by subject, so you can mix lots of types of data in a single stream (as opposed to multiple streams) and simply filter by subject; NATS is very efficient at using the index to filter. Like RabbitMQ but unlike Kafka, streams don't need to be consumed in order; you can nack (!) messages, or set an ack timeout, causing redelivery if acks aren't sent in time. In other words, JetStream can work like Kafka (where you always read by position) or like RabbitMQ (where messages are skipped once acked, but retried once nacked). JetStream has deduplication and idempotency, which allows you to build "exactly once" delivery, which is awesome.
Similar to how someone built a database on top Kafka (KSQL), the NATS team has built a key-value store on JetStream, as well as a blob store. They work the same way, through message ID deduplication. A stream is basically a bunch of database rows, with the message ID acting as primary key. So the stream acts as a primitive to build new, novel things on top of.
I think it's fair to say that RabbitMQ gives you opinionated tools to do certain things, whereas NATS and JetStream are a hybrid "multi model" system that can be used for more purposes. For example, you can embed NATS in your app and use it as a really lightweight RPC mechanism. You can use JetStream as a classic "work queue" where each worker gets a single copy of each message and has to ack/nack so the queue moves forward. You can use JetStream as a log of all actions taken in a system, with retention going back years. (NATS/JS is actually awesome for logging.) And so on.
We use NATS for different use cases at my company. In one use case, clients connect to an API to follow live, low-latency change events. For each such client connection, we register a NATS subject; then tons of processes will see this subject (and its settings, such as filters) and will all start send changes to that one subject. There's no single "controller"; it's all based on point-to-point and one-to-many communication.
(Full disclosure: I'm not familiar with newer RabbitMQ versions or the streaming stuff they've added, so it's possible that RabbitMQ has caught up here in some ways.)
I must say that rmq in k8s, is possible but hard to admin. It’s not a “toy”. But has great documentation. Will take several iterations to key in the right configuration for the use case.
NATS and Kafka can handle higher volumes on the same resources but IMO the use cases are different or you have to write lots of code app side to implement what rmq does with these tools.
NATS, especially in its most elementary form, is braindead simple, stateless, and ootb functional. If I only want a message broker/pubsub, I pick that. If I know on day 1 that I need queues or persistence, I would probably pick Rabbit over NATS’ offering (Jetstream).
"Key value", "pub-sub", "microservices can live anywhere" And?
Why do I care? What does "microservices can live anywhere" even mean and what does it have to do with NATS?
I find RabbitMQ sometimes inscrutable but I think even their website is better
The new Khepri metadata store in RabbitMQ 4.0 allows for even better scalability.
Just wondering what the biggest selling points are for somebody making a decision?
We needed features like job priorities, and complex routing logic and on top of that our software needs to send millions of messages a minute. What we miss is a nice UI to see the state of jobs, and monitor for problems more easily like: https://taskforce.sh/
> RabbitMQ is a reliable and mature messaging and streaming broker, which is easy to deploy on cloud environments, on-premises, and on your local machine.
What does that mean? "Messaging and streaming broker"? I understand the need for worker queues to process videos, images, emails and such but I can't easily tell if that's what this is.
Also, what are the benefits of this over just processing incomplete records straight out of my database? i.e. using MySQL as a queue.
Mainly throughput and latency. I haven’t used MySQL recently so some of this may apply more to Postgres.
Postgres has LISTEN/NOTIFY which helps with latency. I don’t think MySQL has LISTEN/NOTIFY, which means you’d have to resort to polling.
You have to use the `SELECT … FOR UPDATE SKIP LOCKED LIMIT 1` features to grab a message from the queue table, so multiple consumers don’t pull the same message.
The biggest issue, if you’re trying to achieve decent throughput, is dealing with bloat (empty pages still sitting on the disk that haven’t been cleaned up yet). You can run vacuum but an online vacuum will only mark the pages as available for reuse (doesn’t free up the disk space). And if you run a full vacuum (which will free the disk space) it locks the entire database while it runs the vacuum. This can compound if you’re using indexes.
One way of dealing with this is setting up partitioning by message timestamp, so that as old messages roll out, you can just drop those partitions and not have to deal with vacuum.
It can work if your queue needs are low throughput or can tolerate higher latency, but there are some things to manage, and realistically setting up a Redis instance is probably less complex than trying to work around the database-specific quirks, unless you’re already very familiar with your database’s inner workings.
If you are looking at RabbitMQ with "Maybe this is too much". Beanstalkd likely has features you need with almost none of the setup. Just don't web expose it ;)
I only used it on one project years ago and it was a pleasure, dead easy to get up and running and rock solid.
You have to read a lot of docs or you WILL hold RabbitMQ wrong.
The ideal RMQ cluster has 3 servers and is dedicated to just a few apps.
RMQ immediately slows down (due to mnesia causing delays) and processes start dropping messages despite having system resources to grow.
The biggest issue back then was finding a quality client implementation for the language you were using. Not sure what the status of that is these days.
Its unfortunate your team ran into performance issues, as Erlang can be inefficient in some situations. Were you using static routes on the DNS load balanced parallel consumers, or relying on the disk caching mechanisms?
I haven't touched it in years so I can't expand, but when I did, I had to write so many wrappers and add extra logic to use it properly.
Does RabbitMQ have equivalent features?
Reminds me, I'll have to check if they have a working donation link someplace. =3
BEAM languages (including Elixir and Gleam) share the benefits Erlang enjoys by also being part of the ecosystem.
I loved it and the only issues we had were due to our fuckups.
Perhaps people are still thinking in single point of ingress design paradigms. Admittedly RabbitMQ can be a pain to administer for Juniors, but then again so are the other options. =3
I can’t speak to the new version, but it comes with support for even more messaging patterns out of the box.
I would love to be able to use a distributed log like Kafka/Redpanda since it's HA out of the box, but it simply does not fit that use-case.
Earlier this year I tried setting it up as a websockets message broker in a Spring Boot app but failed miserably. I ended up using Spring’s simple broker.
Edit: FWIW, NATS also supports MQTT: https://docs.nats.io/running-a-nats-service/configuration/mq...
They didn't lose my respect for the removal of the argument, however, they lost my respect for whatabouting the conversation calling SemVer a "no true scotsman" fallacy, then trying to claim that removing a redundant argument is not a breaking change, and other reality-warping nonsense, before blocking myself and other complainants from their issues - and even deleting some of their own comments to mop up some of their own terrible reasoning.
I'm sure there is no love lost on their side, either. Personal rant over.
Well, at least AMQP 1.0 is now supported so I expect that for most things you are able to use any client now.
https://docs.celeryq.dev/en/stable/getting-started/backends-...
https://github.com/rabbitmq/rabbitmq-server/blob/main/releas...
