{"id":50441216,"title":"Show HN: QueenMQ – slow LLM calls blocked our Kafka partitions, so we built it","points":3,"user":"aliceviola","time":1794322356,"time_ago":"4 hours ago","type":"link","content":"

Hi HN, I'm Alice.

Over the last year, at Smartness, the company where I work, we built a new message broker, open source under Apache 2.0. During this year it has replaced Kafka almost completely for us. I would like to show it to you and try to explain what is different about it.

First, long story short: in 2025 we launched a new product in our suite. Like all the others it used Kafka for messaging, and like many other companies we were using Kafka as a message queue, wrongly and knowingly.

In this product head-of-line blocking hurt a lot: an LLM call takes seconds to minutes, and it blocked every other chat in the same partition. We first built a hack: we saved the Kafka messages in Postgres, ordered by the entity we needed (a chat_id, for example), processed them, and sent them back to Kafka. Then we decided to build a real broker on that idea. It took a year, from C++ plus Postgres to, today, a single Rust binary with data replicated on disk by Raft.

I designed this system, so I am biased. Still, I believe QueenMQ holds a new paradigm that turns a message broker into something more powerful, while staying simple.

What QueenMQ is, compared to what you already know:

Like Kafka, it has partitions, but in QueenMQ they are dynamic (not preallocated) and you can use a lot of them (tested up to 10 million). Like Kafka it is a log, but there is one Raft leader per tenant, so all of a tenant's writes go through one node, which allows transactions without 2PC. Head-of-line blocking can only happen inside one partition, which is one business entity, like a customer or a chat. If you need a plain queue, you can use a different partition for every message.

Like RabbitMQ, it has ACK, NACK, retries and a DLQ: this makes it a message queue. It has no routing on push and no exchanges, though.

Like a database: thanks to its etcd-like design, it runs transactions that include ACKs, pushes to other queues, KV writes and timers, and it has atomic locks. The KV store is replicated by Raft and keeps the application's state. It is linearizable.

It also has strong streaming features: the SDKs can receive one single log with all the messages of all the queues, with one clock and one order. On top of it there are the classic streaming features, like windows, and some less common ones.

Clients talk to it over HTTP (the clients are thin, the intelligence is in the broker), and it is Kafka-compatible (although that way you cannot use all of its strengths).

The idea underneath is the change from seeing a message broker as a pure "handler" of messages to one where you can offload a lot of application logic that you would otherwise write yourself. In our company it let us remove outbox tables, cron jobs and many other things.

On three beefy machines (24 vCPU, NVMe, 88 GB of RAM), QueenMQ sustains 3M msg/s of 256-byte messages, up to 10M partitions, with p99 latency around 20 ms and every write fsynced on two nodes. Of course, with a single leader per tenant, it does not scale by adding more machines.

We tested it hard: 226 tests of our own, written with the Jepsen library, on up to 5 nodes.

It is changing the way we design the systems and products of our company.

I believe in this project and I believe in open source, and I am really glad that Smartness thinks the same way. It is hard to find the time and the money to build something like this without a company behind you.

I would love to get questions, the harsh ones too (like how many times this thing crashed production), and feedback.

  docker run -p 6632:6632 ghcr.io/queen-mq/queen:2.8.0

","url":"https://github.com/queen-mq/queen","domain":"github.com","comments":[],"comments_count":0}