Managed RabbitMQ

RabbitMQ 4, one node or three across zones with quorum queues.

What it is

RabbitMQ 4 — the open-source message broker, unmodified, from the official image — on machines of its own in your region, reachable only inside your network. One node for development, or three nodes in one cluster across two zones where quorum queues keep serving and keep every confirmed message when a node goes. Find it under RabbitMQ in the console.

Anything that speaks AMQP 0-9-1 connects with an amqps:// URL: Celery, Rails/Sneakers, Spring AMQP, MassTransit, NestJS microservices, pika, amqplib, php-amqplib, bunny.

Concepts

Term Meaning
Broker One RabbitMQ cluster of 1 or 3 nodes, identified by the name you give it.
Node One machine running RabbitMQ on a data volume of its own. Three nodes sit in two zones (de1a, de1b, de1a) and never share a physical host.
Quorum queue A replicated queue (Raft): every message is on a majority of the nodes before the broker confirms it. Queues declared without a type are quorum queues on Deutsche Cloud. One node is the queue's leader; the console shows which.
Classic queue Declared with x-queue-type: classic, or exclusive / non-durable (Celery's event and reply queues): lives on one node, not replicated.
Private address The broker's address in its network: AMQPS on 5671. With three nodes it is a private balancer that sends connections to every serving node (any node serves any queue); the management UI is on https://<address>/. With one node the management UI is on 15671.
TLS Always. No plaintext port. The certificate is the platform CA's, for the broker address and every node address. Traffic between the nodes (Erlang distribution, quorum-queue replication) is TLS too, with certificates checked both ways.
Users admin (tag administrator) is made with the broker. Add more with the API, CLI or console: each password is filed in Secrets as RABBITMQ_<BROKER>_<USER>_PASSWORD. Permissions are RabbitMQ's: three regexes (configure, write, read) on vhost /.
Configuration backup An encrypted copy of the broker's definitions (users with their password hashes, queues, exchanges, bindings, policies) in object storage: one a day (kept 7 days) plus the ones you take. Messages are not in it.
Activity The event stream on the broker page: every provisioning step, nodes going down and coming back, memory and disk alarms, users applied, configuration backups, what the nodes reported.

The data volumes are LUKS2 with a key per volume, held by the control plane under the platform key or one of your KMS keys (kmsKeyId), released to the node only while the volume is attached to it. Deleting the broker deletes the keys.

How to

Create a broker

RabbitMQ → Create broker, or:

POST /rabbitmq/api/brokers
{ "identifier": "orders", "nodeType": "performance-2", "nodes": 3, "volumeSizeGib": 40 }

or dcloud rabbitmq brokers create --identifier orders --node-type performance-2 --nodes 3, or Terraform dcloud_rabbitmq_broker. Without vpcId the broker joins your default network, where deployed apps run. It is creating until every node runs in one cluster and the admin user is in place — about three to five minutes; the Activity tab shows each step.

nodes: 1 (the API default) is one machine: nothing is replicated, a reboot or a lost machine stops the broker and loses what was queued. Use three nodes for anything you would miss.

Connect

From an app or a machine in the same network, with the password from Secrets:

import pika, ssl
ctx = ssl.create_default_context(cafile="dcloud-ca.pem")
params = pika.URLParameters("amqps://admin:<password>@10.0.0.9:5671/%2F")
params.ssl_options = pika.SSLOptions(ctx, "10.0.0.9")

dcloud-ca.pem is GET /rabbitmq/api/ca-bundle (app containers already trust it). The broker page shows the URL with the password left out, and the nodes' own addresses: the balancer sits in one zone, so a client that takes several hosts (Spring addresses, MassTransit cluster nodes, Celery broker_url failover list) can list them too.

Celery

Works unchanged; the celery queue becomes a quorum queue. Turn on publisher confirms so a task Celery reports as sent is on a majority:

app.conf.broker_url = "amqps://admin:<password>@10.0.0.9:5671//"
app.conf.broker_use_ssl = {"ca_certs": "/etc/ssl/certs/ca-certificates.crt", "cert_reqs": ssl.CERT_REQUIRED}
app.conf.broker_transport_options = {"confirm_publish": True}

Celery's event (-E) and reply queues are non-durable classic queues, which RabbitMQ 4.3 refuses by default; Deutsche Cloud brokers allow them.

Not losing messages

RabbitMQ keeps a message you published safe only when it confirmed it: use publisher confirms (confirm_delivery() in pika, ConfirmChannel in amqplib, confirm_publish in Celery) and treat an unconfirmed publish as not sent. Consumers ack after the work is done (auto_ack=False). A message can then be delivered twice after a failover (at-least-once): make consumers idempotent, e.g. by message_id.

Failover

With three nodes, when a node's machine stops, the quorum queues it led elect a new leader among the other two, and the balancer's health check (every 3 s: "does this node accept AMQPS, and is it connected to a majority of the cluster?") stops sending connections to it. Clients connected to it notice by the AMQP heartbeat and reconnect to the same address. Nothing in the control plane is on this path. The node, back, rejoins and catches up by itself.

Measured on dev (3 × performance-2, hard power-off of the node leading the queue, a producer with confirms and a consumer through the balancer): 0 confirmed messages lost in every run (three runs, up to 16,206 messages), no duplicate deliveries. A client connected to a surviving node paused publishing for 6.5 s (the new leader's election); a client connected to the node that died noticed by its 5 s AMQP heartbeat and reconnected through the balancer, about 18 s in all. The node, powered on again, had its queue member back within one to four minutes.

When a load balancer machine stops (a three-node broker's endpoint runs on Deutsche Cloud's pair of load balancer machines in your network, one address each, behind the endpoint's name): new connections by name go to the other machine at once. A connection open through the stopped machine gets a reset once the other one takes its address over, and a client with default settings is back after about 28 s (measured with pika and amqplib). Connections through the other machine pause about 13 s while it takes the address. Connect by the name, not an address, and use publisher confirms: without them a publish into a stalled connection reports success and the message is lost.

RabbitMQ 4 keeps its metadata (users, queues, bindings) in Khepri, a Raft store, like the quorum queues themselves: a node cut off from the other two cannot commit anything, so there is no split brain; it keeps accepting connections, but the health check takes it out of the balancer within about 25 s and its publishes are refused (nacked), never confirmed.

Two zones, three nodes: de1 has two zones, so two nodes sit in Falkenstein (de1a) and one in Nuremberg (de1b). Losing Nuremberg, or any one machine, leaves a majority. Losing all of Falkenstein leaves one node, a minority: the broker stops taking messages until Falkenstein is back. Measured: with both de1a machines powered off, every publish was nacked (none falsely confirmed); 43 s after they were powered on again publishes were confirmed, and no confirmed message was lost. A third zone fixes this when the region gets one.

Users

POST /rabbitmq/api/brokers/{id}/users
{ "name": "worker", "tags": "", "configure": "^$", "write": "^orders$", "read": "^orders\\." }

A node applies it within 30 seconds (applied: true). PATCH …/users/{userName} changes tags or permissions; POST …/users/{userName}/rotate-password files a new Secrets version and the node applies it within 30 seconds — the old password stops working then (RabbitMQ keeps one per user), so redeploy the apps that use it. Users you make in the management UI are yours: Deutsche Cloud does not touch them, and their passwords are not in Secrets. The platform's own users (dcloud on loopback only, ews-events-* for the event bus) are re-made if removed.

Patch releases and OS updates

The nodes run RabbitMQ by its minor (4.3); POST /rabbitmq/api/brokers/{id}/upgrade (or Maintenance → Restart onto the newest patch) restarts them onto the newest 4.3.x, one node at a time:

  1. A node is taken only while the broker is whole: every node running without alarms or partitions, every quorum queue with all its members online.
  2. It drains first (rabbitmq-upgrade drain): its quorum queue leaders move to the other nodes and its client connections close, so clients reconnect to a serving node.
  3. It pulls the image, restarts, and catches up.
  4. The next node waits until every quorum queue is complete again. A node that does not come back pauses the run; resume (.../maintenance/resume) or cancel (.../maintenance/cancel) it.

POST .../patch does the same with the operating system's packages now, rebooting a node when its kernel asks. Security updates also install nightly by themselves; reboots then happen in the platform window, one node at a time, only while every node runs without alarms. With one node either is downtime. The health check looks at the 20 deepest queues the report carries.

App bindings

Bind a deployed app to the broker as one of its users: PUT /deploy/api/apps/{appId}/rabbitmq { "brokerId": "rmq-…", "userName": "worker" }, or the app's Configuration → Kafka and RabbitMQ panel. At every rollout the app gets AMQP_URL (amqps://worker:<password>@<broker address>:5671/%2F, through the balancer with 3 nodes), AMQP_HOSTS (every node's own address:5671, for clients that take a list), AMQP_USERNAME, AMQP_PASSWORD and AMQP_CA_FILE (the platform CA in the container). The user is named on purpose — there is no silent admin. It needs deploy:ManageBrokerBinding and secrets:GetSecretValue on the user's password; a bound user cannot be deleted, and after a rotation Activity names the apps that need a rollout.

Metrics and alarms

Every node reports every 30 seconds: running or not, memory used against its watermark, disk free against its limit, file descriptors, network partitions; and the broker's connections, channels, queues, consumers, ready/unacked messages, publish and deliver rates, and its 20 deepest queues with their type, leader node and members online. A memory or disk alarm blocks publishers (RabbitMQ's own flow control) and appears in Activity at once with the numbers. The limits are sized from the machine: memory watermark 60 % of (RAM − 512 MB) (so a node needs at least 2 GiB; smaller types are refused), disk-free limit = RAM, at most 10 % of the volume, at least 1 GiB.

Health. Beside its status a broker has a health, worked out every minute: healthy; degraded with healthReasons (node_down, resource_alarm — a memory or disk alarm on any node blocks publishers on every node —, reports_stale); or no_quorum: fewer than a majority of the nodes run (1 of 3), so quorum queues refuse publishes (nacks) until enough are back. Each change is a line in Activity and an event on the bus (source ews.rabbitmq, detail-type Broker Health Change).

Metrics in Monitoring, namespace EWS/RabbitMQ, dimension BrokerId, one point a minute: RunningNodes, ResourceAlarms (nodes in a memory or disk alarm), DiskUsedPercent (the fullest node; per node with dimension Node too), MessagesReady, MessagesUnacked, Connections, Consumers, Queues, PublishRate, DeliverRate (messages/s). No per-queue series yet.

Default alarms, made when the broker becomes available, as rabbitmq-<name>-…: no-quorum (RunningNodes below a majority for a minute; Monitoring judges a minute 2 minutes after it ends, so it fires 3–4 minutes after the loss, while health says no_quorum within about 40 s), resource-alarm (any node in alarm, 1 minute), disk (a volume over 85 % for 2 minutes). They email alarmEmails from the create (the console fills in yours; the API defaults to none) and say each change in Activity. They are yours to change or delete; deleting the broker deletes those still linked to it.

Configuration backups and restore

Config backups → Back up now, or POST /rabbitmq/api/brokers/{id}/exports. A node exports the definitions (the platform's own users left out), encrypts them with a key of the backup's own and uploads them. Restore = a new broker from a configuration backup: Config backups → Restore into a new broker, or

POST /rabbitmq/api/brokers
{ "identifier": "orders-2", "nodeType": "performance-2", "nodes": 3, "exportId": "rmd-…" }

Once its nodes run, one imports the definitions; the users you made with the API come back with their passwords, filed in Secrets again under the new broker's name. Messages are not restored.

Publish to it from the event bus

An event-bus rule can target a broker's exchange (type: "rabbitmq", broker, exchange, optional routingKey, default the event's source such as ews.ec2). The target's role needs rabbitmq:PublishMessage on the broker. The control plane publishes as a RabbitMQ user of the target's own (ews-events-…, allowed to write to that one exchange and nothing else), over TLS to the nodes' public addresses on port 15671, which the broker's firewall opens to the control plane only. A message the exchange routes to no queue is a failed delivery (retried, then the dead-letter queue), never a silent success. Every message carries message_id <event id>:<target id> and the x-ews-* headers.

Delete a broker

Danger zone → Delete, or DELETE /rabbitmq/api/brokers/{id}: the machines, volumes (and their keys), balancer, firewall and user passwords go; queued messages go with them. Daily configuration backups are deleted, manual ones stay until you delete them.

IAM

Action What it allows
rabbitmq:DescribeRabbitmqBrokers List and read brokers, nodes, queues, metrics, Activity
rabbitmq:CreateRabbitmqBroker Create a broker (on a configuration backup too, to restore it)
rabbitmq:RebootRabbitmqNode Restart one node
rabbitmq:DeleteRabbitmqBroker Delete a broker
rabbitmq:DescribeRabbitmqUsers List users (never passwords: those are secrets:GetSecretValue)
rabbitmq:CreateRabbitmqUser Create users
rabbitmq:UpdateRabbitmqUser Change tags and permissions
rabbitmq:RotateRabbitmqUserPassword New password
rabbitmq:DeleteRabbitmqUser Delete users
rabbitmq:PublishMessage Publish through the platform (an event-bus target's role)
rabbitmq:*Export* List, take and delete configuration backups

Resources: rn:ews:de1:<org>:rabbitmq:broker/<id>, …:rabbitmq:user/<broker id>/<name>, …:rabbitmq:export/<id>.

Pricing

Every node bills per hour as rabbitmq.<type>.hour (the machine, its public IPv4 and the managed service); the balancer is included. Volumes bill as storage.

Limits

  • One vhost (/); more are a next step. Users made in the management UI are not in Secrets.
  • nodes and the node type are fixed at creation; patch releases of 4.3 arrive with a node's restart. Rolling upgrades across minors are a next step.
  • Germany (de1) only for now.
Did this page help you?