Skip to content
queen-mqPublic

About

The ACID message broker. Every customer, order and conversation gets its own queue, and every step commits as one: the ack, the next message, the state, the timer. No Redis lock, no cron job, no retry table.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

70 stars

Watchers

2 watching

Forks

Latest commit

 

History

2,101 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Queen, message queue

The ACID message broker.

Imagine a Kafka with millions of named partitions, created on the fly, where you ack or nack each message, where one transaction covers the ack, the next messages and your state, with its own KV store and stream processing.
That is QueenMQ.

Proof: 226 Jepsen tests passed · 3M msg/s on three nodes and ten million queues, with every write synced to disk on two.

Open source (Apache-2.0), Rust, one binary. In production at Smartness.

docker run -p 6632:6632 ghcr.io/queen-mq/queen:2.8.1

QueenMQ in five minutes

The short version, in JavaScript. The full guide has nine steps in JavaScript, Python, Go, Rust and PHP. Every snippet, here and on the site, is cut from a program that runs them all and checks each one.

1. Run a node

docker run -p 6632:6632 ghcr.io/queen-mq/queen:2.8.1
npm install queen-mq

The dashboard is at http://localhost:6632, served by the same process.

2. Connect, and configure a queue

Optional: the first push creates a queue with the defaults.

import { Queen } from 'queen-mq'

const queen = new Queen({ url: 'http://localhost:6632' })

await queen.queue('orders')
  .namespace('shop')              // groups queues in the dashboard
  .config({
    leaseTime: 30,                // seconds a worker holds a message before it is redelivered
    retryLimit: 3,                // failures before the message goes to the dead-letter queue
    retentionEnabled: true,       // retention runs only when this is on
    retentionSeconds: 86400,      // then messages older than a day are removed
  })
  .create()

3. Push to a partition

Partitions are for ordering. Not for scaling. In Kafka you choose a number of partitions and hash keys onto them, so a slow customer holds up every customer that shares its partition. Here a partition is one entity's own queue: a customer, an order, a conversation. The push that names it creates it, order holds inside it, and one queue has held ten million.

await queen.queue('orders')
  .partition('customer-42')       // one ordered partition per customer, created by this push
  .push([{
    transactionId: 'order-1001',  // the dedup key: pushing the same id again is a no-op
    data: { orderId: 1001, customer: 42, total: 59.9 },
  }])

4. Consume

Without a consumer group, each message goes to one worker. Consumer groups, leases and every pop option are in the full guide.

await queen.queue('orders')
  .each()                         // the handler gets one message at a time
  .limit(1)                       // stop after one message; a worker leaves this out
  .consume(async (msg) => {       // acked when the handler returns, retried if it throws
    console.log(msg.data)
  })

5. One step, one commit

The ack, the order's new state, the next event and tomorrow's reminder commit as one log entry, or none of them does. That one call replaces the outbox table, the idempotency table and the cron job.

await queen.queue('orders')
  .group('fulfillment')
  .subscriptionMode('all')
  .autoAck(false)
  .each()
  .limit(1)
  .consume(async (msg) => {
    const { orderId, customer } = msg.data
    await queen.transaction()
      .ack(msg, 'completed', { consumerGroup: 'fulfillment' })  // the input is done
      .kv.put('orders', `order-${orderId}`, { status: 'paid' }, { ttl: '7d' })
      .queue('shipments').partition(msg.partition)              // the next event
      .push([{ transactionId: `ship-${orderId}`, data: { orderId } }])
      .timer('reminders').key(`review-${orderId}`).delay('24h') // a message tomorrow
      .payload({ orderId, customer }).schedule()
      .commit()
  })

6. Read it again

Consuming does not delete: the queue is a log, kept for its retention. A new group can start from a point in time and read the same orders again.

await queen.queue('orders')
  .group('audit')                           // a new group, with its own cursor
  .subscriptionFrom('2026-10-22T00:00:00Z') // everything pushed from this time on
  .each()
  .limit(1)
  .consume(async (msg) => {
    console.log('again:', msg.data)
  })

Or read a partition by offset, with no group, lease or ack:

const read = await queen.queue('orders').fetch(['customer-42']) // no group, lease or ack
for (const record of read.get('customer-42').records) {
  console.log(record.offset, record.payload)
}

The full guide goes on with locks, stream windows and request/reply over in-memory queues, in every language.

What else it does

  • Streams with windows. Tumbling, sliding and session windows over any queue, in JavaScript, Python, Go and Rust. The ack, the state and the output commit together. Streams
  • Conflation. A conflating consumer group gets only the newest message of each partition, for values where only the latest matters. Last value
  • Ephemeral queues. In memory, outside the log, for request/reply: fast, and lost on a crash by design. Ephemeral queues
  • KV, locks and timers. State per entity, one worker at a time, and messages scheduled for later, all inside your transactions. KV, locks and timers
  • Retries and dead letters. A failed message comes back after a delay, a few times, then goes to the dead-letter queue and its partition moves on. Replay it once fixed. Retries
  • Kafka clients. Unchanged Kafka producers and consumers work against QueenMQ's Kafka port. Kafka
  • PostgreSQL and S3, run by the broker. A table into a queue, a queue into a table, and queues into an S3 bucket as JSONL or Parquet, with no other process to run. PostgreSQL · S3
  • Tenants and standby. Many tenants on one cluster behind the built-in proxy, and a standby cluster that takes over with one call. Tenants · Standby

Numbers

Three GCP nodes, 24 vCPU each, local NVMe. A write is confirmed only after two nodes have written it to disk (fsync). Kafka, Pulsar and Redpanda ran on the same machines.

  • 3M msg/s in and out of one queue, from 10 to 10M partitions, p99 15-25 ms. Kafka and Redpanda could not keep up at 50,000 partitions, Pulsar at 100,000.
  • 1M msg/s of 1 KB messages (about 1 GB/s), p99 16-22 ms.
  • 5M msg/s on one queue at most.
  • 100 GB of lag drained in 32 seconds (3.4 GB/s).
  • 28 s for a node away five minutes to rejoin a cluster holding 521 GB a node: it gets only what it missed, a 1.1 GB delta, and a restart reads hash sidecars instead of its logs (20 s where it took 171 to 201 s).
  • 4 ms p99 for a 10-message transaction, at 6,000 transactions per second.
QueenMQ 2.4 end to end p99 as one queue grows from 10 to 10 million partitions on three GCP nodes with local NVMe: 17 to 29 ms at 3M msg/s of 256-byte messages, 17 to 25 ms at 1M msg/s of 1 KB messages

QueenMQ 2.4 on three GCP c4-standard-24-lssd nodes with local NVMe, 2026-10-15: end to end p99 at 3M msg/s of 256-byte messages and at 1M msg/s of 1 KB messages, the worst load process in seconds 30-60 of 70 (at 3M and 1,000 partitions, the median of five runs). 2.3.0's 3M line read 47 to 78 ms on the same axes.

Full details: queenmq.com/benchmarks

Proven

  • In production at Smartness.
  • 226 Jepsen tests on 2.7, all passed. No acknowledged message lost, no lease held twice.
  • Written in Rust. Apache-2.0.

Documentation

Using a coding agent? Add the MCP server:

claude mcp add --transport http queen https://queenmq.com/mcp

License

Apache-2.0, see LICENSE.md.


Nothing happens halfway.

QueenMQ is built at Smartness.

About

The ACID message broker. Every customer, order and conversation gets its own queue, and every step commits as one: the ack, the next message, the state, the timer. No Redis lock, no cron job, no retry table.

Topics

Resources

Code of conduct

Contributing

Security policy

Stars

70 stars

Watchers

2 watching

Forks

Releases

Packages

Contributors

Languages