Managed Apache Kafka
Kafka 4 in KRaft mode, one node or three across zones, SCRAM users, managed topics.
What it is
Apache Kafka 4 — unmodified, from the official apache/kafka image, in KRaft mode (no ZooKeeper) — on machines of its own in your region, reachable only inside your network. One node for development, or three nodes across two zones where every partition lives on three nodes and a write acknowledged with acks=all is on at least two of them. Find it under Apache Kafka in the console.
Anything that speaks the Kafka protocol connects: the Java client, Kafka Streams, Kafka Connect, Debezium, Flink, confluent-kafka (Python, Go, .NET), librdkafka, franz-go, @confluentinc/kafka-javascript, Spring Kafka.
Concepts
| Term | Meaning |
|---|---|
| Cluster | One Kafka cluster of 1 or 3 nodes, identified by the name you give it. |
| Node | One machine running Kafka as controller and broker (KRaft combined mode, the right shape for 1–3 nodes) on a data volume of its own. Three nodes sit in two zones (de1a, de1b, de1a), never on one physical host; each node's zone is its Kafka broker.rack, so a partition's replicas spread over both zones. |
| Controller quorum | The three nodes elect an active controller (Raft). It needs a majority: two of three. The console shows which node it is. |
| Bootstrap servers | Every node's private address, port 9093. There is no balancer: a Kafka client talks to each broker by its own address, so give it all three. |
| Security | SASL_SSL with SCRAM-SHA-512, always. No plaintext port, no anonymous access. The certificate is the platform CA's (GET /kafka/api/ca-bundle; app containers already trust it). Traffic between the nodes and to the controllers is TLS with client certificates, and only the cluster's own nodes reach those ports. |
| Replication | Three nodes: replication factor 3 and min.insync.replicas 2 by default, unclean leader election off. One node: replication factor 1 — nothing is replicated. |
| Users | admin is made with the cluster: everything on every topic, consumer group and transactional id, and it may create topics — but not change ACLs, users, quotas or broker settings (those belong to the API). Add users with ACL grants; each password is filed in Secrets as KAFKA_<CLUSTER>_<USER>_PASSWORD. |
| Grants | A user's permissions: {resourceType: topic | group | transactional_id, name, patternType: literal | prefixed, access: read | write | readwrite | all}. name: "*" is every one. A consumer needs read on its topics and on its group. |
| Managed topic | A topic declared through the API, CLI, console or Terraform. The cluster keeps it as declared: a setting changed over the Kafka protocol is set back and Activity says so. Topics created over the protocol (by admin, or a client) are listed too, read-only, and can be adopted. Internal topics (__consumer_offsets) stay Kafka's. |
| Activity | The event stream on the cluster page: every provisioning step, nodes down and back, the active controller moving, partitions under-replicated or without a leader, users and topics applied, upgrades, disk filling up. |
The data volumes are LUKS2 with a key per volume, held by the control plane under the platform key or one of your KMS keys (kmsKeyId), released to the node only while the volume is attached to it. Deleting the cluster deletes the keys.
Messages are not backed up. Three nodes keep three copies across two zones; that protects against a machine or a zone, not against deleting a topic. What is backed up is the cluster's definitions — topics and users, see Configuration backups. Retention is by time (7 days by default): set retention.bytes on a topic if the volume must not fill.
How to
Create a cluster
Apache Kafka → Create cluster, or:
POST /kafka/api/clusters
{ "identifier": "events", "nodeType": "performance-4", "nodes": 3, "volumeSizeGib": 100 }or dcloud kafka clusters create --identifier events --node-type performance-4 --nodes 3, or Terraform dcloud_kafka_cluster. Without vpcId the cluster joins your default network, where deployed apps run. A node needs at least 2 GB of memory (the JVM takes a quarter of the machine, the rest is page cache). It is creating until every node is in the quorum and admin is in place — about three to five minutes; the Activity tab shows each step. engineVersion is 4.4.0 (default) or 4.3.2.
Connect
bootstrap.servers=10.0.0.10:9093,10.0.0.11:9093,10.0.0.12:9093
security.protocol=SASL_SSL
sasl.mechanism=SCRAM-SHA-512
sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin" password="<from Secrets>";
ssl.truststore.type=PEM
ssl.truststore.location=/path/to/dcloud-ca.pemfrom confluent_kafka import Producer
p = Producer({
"bootstrap.servers": "10.0.0.10:9093,10.0.0.11:9093,10.0.0.12:9093",
"security.protocol": "SASL_SSL", "sasl.mechanism": "SCRAM-SHA-512",
"sasl.username": "app", "sasl.password": password,
"ssl.ca.location": "dcloud-ca.pem",
"enable.idempotence": True, "acks": "all",
})The cluster page shows the bootstrap list. An app bound to the cluster gets it as KAFKA_BOOTSTRAP_SERVERS with the user's credentials (see App bindings).
Client versions
Clients built on librdkafka need 2.6.1 or newer. Older librdkafka sends a malformed SCRAM final message: it repeats the client nonce, a bug from v0.0.99 to v2.6.0 (librdkafka #4895). The Kafka 4 brokers refuse it, and the client reports SASL authentication error: Authentication failed, as if the password were wrong. This affects confluent-kafka (Python, Go, .NET), kcat, rdkafka (Rust), node-rdkafka and @confluentinc/kafka-javascript, which use the librdkafka they bundle or the one installed on the system. Lowering the SCRAM iteration count does not help.
Measured on dev (2026-11-15, Kafka 4.4.0, SCRAM-SHA-512) with a user made through the API:
| Client | librdkafka | Logs in |
|---|---|---|
confluent-kafka 2.3.0, 2.4.0, 2.5.3, 2.6.0 |
same as package | no |
confluent-kafka 2.6.1, 2.8.0, 2.10.0 |
same as package | yes |
kcat from Ubuntu 22.04 (kafkacat) |
1.8.0 | no |
kcat from Debian 12 |
2.0.2 | no |
kcat from Ubuntu 24.04 |
2.3.0 | no |
kcat from Debian 13 |
2.8.0 | yes |
kcat from Alpine 3.22 |
2.10.0 | yes |
kcat from Ubuntu 26.04 |
2.13.0 | yes |
Java client (kafka-console-producer 3.9.1) |
— | yes |
kafkajs 2.2.4 |
— | yes |
On Ubuntu 24.04 or Debian 12, don't use the distribution's kcat. Use pip install "confluent-kafka>=2.6.1", Confluent's librdkafka packages, or a debian:13 / alpine container. The Java client and kafkajs implement SCRAM themselves and are not affected.
Topics
POST /kafka/api/clusters/{id}/topics
{ "name": "orders", "partitions": 12, "configs": { "retention.ms": "604800000", "retention.bytes": "5368709120" } }The collecting node creates it within a minute (applied: true). PATCH adds partitions (never fewer) or replaces the settings: a key left out returns to the broker default. Settings: retention.ms, retention.bytes (per partition), cleanup.policy (delete, compact, compact,delete), max.message.bytes, min.insync.replicas. To manage a topic someone created over the protocol: POST .../topics/{name}/adopt keeps its current settings and makes it managed.
Users
POST /kafka/api/clusters/{id}/users
{ "name": "billing-consumer", "acls": [
{ "resourceType": "topic", "name": "orders", "access": "read" },
{ "resourceType": "group", "name": "billing-", "patternType": "prefixed", "access": "read" } ] }The password is in Secrets as KAFKA_EVENTS_BILLING_CONSUMER_PASSWORD; the cluster applies the user within a minute. PATCH replaces the grants (the new ones are added before the old ones are removed, so a client never loses access in between). rotate-password files a new Secrets version and applies it: new connections need it, open ones keep working until they reconnect. Redeploy bound apps.
App bindings
Bind a deployed app to the cluster as one of its users (there is no silent admin: create a user with just the grants the app needs):
PUT /deploy/api/apps/{appId}/kafka
{ "clusterId": "kfk-…", "userName": "billing-consumer", "envPrefix": "" }or the app's Configuration → Kafka and RabbitMQ panel. At every rollout the app gets KAFKA_BOOTSTRAP_SERVERS (every node, port 9093), KAFKA_SECURITY_PROTOCOL=SASL_SSL, KAFKA_SASL_MECHANISM=SCRAM-SHA-512, KAFKA_SASL_USERNAME, KAFKA_SASL_PASSWORD (the user's current password from Secrets) and KAFKA_SSL_CA_LOCATION (the platform CA, mounted in every app container). An envPrefix (EVENTS_) lets one app bind several clusters. Binding hands the app the password, so it needs deploy:ManageBrokerBinding and secrets:GetSecretValue on that password; the app's machines must be in the cluster's network. A bound user cannot be deleted; after a rotation Activity names the apps that need a rollout.
Watch it
The cluster page shows, from a node's report every 30 s and a cluster view every minute: each node (running, zone, active controller, partition leaders, disk), the quorum, partitions under-replicated / without a leader / below min.insync.replicas, every topic, and every consumer group with its lag and the topics it lags on. The page says which node collected it and when.
Health. Beside its status (creating, available, upgrading, …) a cluster has a health, worked out every minute:
health |
Meaning |
|---|---|
healthy |
Every node runs, every partition is in sync and has a leader. |
degraded |
It serves, and healthReasons say what to look at: node_down, under_replicated, under_min_isr (acks=all writes to those partitions fail), offline_partitions (on topics with fewer replicas), producers_throttled, reports_stale. |
no_quorum |
Fewer than a majority of the nodes run (1 of 3), or there is no active controller. Writes with acks=all cannot be made durable and time out; nothing acknowledged is lost. Back to degraded/healthy as soon as a majority runs again. |
A change is a line in Activity and an event on the bus: source ews.kafka, detail-type Cluster Health Change, with health, previousHealth, reasons, runningNodes.
Metrics in Monitoring, namespace EWS/Kafka, dimension ClusterId, one point a minute: RunningNodes, QuorumNodes, UnderReplicatedPartitions, OfflinePartitions, UnderMinIsrPartitions, Partitions, Topics, ConsumerGroups, MaxConsumerLag, ProducerThrottled (0–3), TieredStorageBytes (with tiered storage on: what the bucket holds, from its last listing), DiskUsedPercent (the fullest node; per node with dimension Node too), and ConsumerLag per consumer group (dimension ConsumerGroup, the 10 groups that lag most, two fewer per replication into the cluster), and per replication into the cluster ReplicationLag and ReplicationSourceReachable (dimension ReplicationId, see Replication below). While the nodes cannot take a cluster view (no quorum), the view's series are not written rather than written as 0. They count toward the organization's custom-series limit (at most 25 per cluster).
Default alarms, made when the cluster becomes available, in Monitoring as kafka-<name>-…: no-quorum (RunningNodes below a majority for a minute; Monitoring judges a minute 2 minutes after it ends, so it fires 3–4 minutes after the loss, while health says no_quorum within about 40 s), offline-partitions (any, 1 minute), under-min-isr (any, 2 minutes), under-replicated (any for 10 minutes: a rolling upgrade under-replicates for a minute or two per node), disk (a volume over 85 % for 2 minutes). They email the addresses in alarmEmails at create (the console fills in yours; the API defaults to none — their state is in the console and on the bus either way) and say each change in the cluster's Activity. They are yours: change or delete them in Monitoring; a deleted one is not made again. Deleting the cluster deletes the ones still linked to it.
Configuration backups and restore
Every day the platform takes a configuration backup of the cluster's definitions and keeps it 7 days; POST /kafka/api/clusters/{id}/snapshots (or Config backups → Back up now, dcloud kafka snapshots create --id …) takes one you keep until you delete it (20 per cluster), also after the cluster is gone. A configuration backup holds:
- every topic with its partitions, replication factor and the settings the API manages (
retention.ms,retention.bytes,cleanup.policy,max.message.bytes,min.insync.replicas) — managed topics as declared, topics made over the protocol as the last cluster view saw them (a view older than 10 minutes marks the backuppartial); adminand every API user with their grants and passwords.
It is encrypted with its own key and stored in object storage. Messages and consumer offsets are not in it, nor the event bus's users (their targets make them again).
Restore = a new cluster: POST /kafka/api/clusters { …, "snapshotId": "kfs-…" } (or Restore into a new cluster on the Config backups page), with at least the backup's node count. The users come back with the same passwords, filed in Secrets under the new cluster's names; the topics come back as managed topics (also those made over the protocol — the restore says which). Apps keep their credentials and need the new bootstrap address. Restoring needs kafka:CreateKafkaCluster on the backup.
Tiered storage: long retention in object storage
Keep a topic's messages for months or years without volumes to match: with tiered storage a topic keeps only its newest part on the brokers and the older segments in a bucket of your object storage. Consumers read old offsets exactly as before; Kafka fetches them from the bucket (through a cache on the node's volume). It is Apache Kafka's own remote log storage (KIP-405) with Aiven's open-source plugin tiered-storage-for-apache-kafka 1.2.0 (Apache-2.0).
It is long retention, not a backup. The remote segments belong to this cluster's topics: deleting a topic or the cluster deletes them, and a restored cluster does not get them back.
- Turn it on for the cluster: at create (
"tieredStorage": true, the console's Tiered storage → On) or later:POST /kafka/api/clusters/{id}/tiered-storage {"enabled": true}(Turn on tiered storage on the cluster page,dcloud kafka clusters enable-tiered-storage <id> --enabled, Terraformtiered_storage = truein place). Later means a rolling restart like an upgrade: one node at a time, the next only when every partition is back in sync;tiered.stateisenablinguntil the last node runs it, thenon. With one node the cluster is down while it restarts. It cannot be turned off for a whole cluster. A cluster created before tiered storage existed (2026-11-15) cannot turn it on in place (its nodes' software predates it): create a new one withtieredStorage: true, from a configuration backup to keep topics and users. - Then per topic:
remote.storage.enable=true, and how much stays on the brokers:local.retention.ms(default 24 hours, or the whole retention when that is shorter) orlocal.retention.bytes.retention.ms/retention.bytesstay the total;-1keeps forever. Not on compacted topics (Kafka keeps those local only). - Turning a topic's tiered storage off deletes every message it holds in object storage (Kafka cannot keep them readable without it): the API refuses unless you send
"deleteRemoteData": true, which needskafka:DeleteKafkaTopicRemoteData. - Where: one bucket per cluster, named after it (
kfk-…), in your object storage. You can see and read it in the S3 console; only the cluster writes it, with a key that opens that bucket alone. Every segment is encrypted on the node before upload, to the cluster's own RSA key pair (the private key is kept by the platform, sealed under its master key), so the store only ever holds ciphertext. - Pause a topic's uploads:
remote.log.copy.disable=true(set it back tofalseto resume). Kafka only pauses a topic whose local retention equals its total, so the API setslocal.retention.ms(andlocal.retention.bytes) to the totals while paused; set a local retention again when you resume. What is already remote stays readable and is still deleted by retention; new segments stay on the brokers meanwhile, so the local retention no longer frees space — watch the volume. - Only rolled segments move (a segment rolls at
segment.bytes, 1 GiB, orsegment.ms, 7 days): a quiet topic may show nothing remote for a while. - What it costs: what the bucket holds, €0.02 per GB-month (
kafka.tiered.gb_month), measured by listing it; the cluster page andGET …/clusters/{id}show the total (tiered.remoteBytes) and each topic's share (remoteBytesin the topic list), as of the last listing (every 15 minutes). - Delete: deleting the cluster empties and deletes the bucket after the machines are gone.
Replication between clusters (MirrorMaker 2)
A replication keeps a copy of a cluster's topics in another cluster, to fail over to when the first is lost — message-level disaster recovery. It is Apache Kafka's own MirrorMaker 2 (three connectors on Kafka Connect) run by the platform:
- What is copied: every topic matching
topics(exact names or prefixes ending in*, likepayments.*; default*, which includes topics made later), with the same names, partition counts and settings (notmin.insync.replicasor tiered-storage settings: the target's own apply), from the oldest message the source still holds. And the committed offsets of the consumer groups matchinggroups, translated to the copy's offsets and written onto the target's groups every 5 seconds. Users and ACLs are not copied (they are API rows; create the users your consumers need on the target). - Where it runs: a Kafka Connect worker on every node of the target cluster, started while the cluster is the target of a replication. No extra machines and no extra charge; it uses up to an eighth of each target node's memory (256 MB – 1 GB) and some CPU. It reads the source over its client port as a read-only SCRAM user the platform makes on the source (
ews-mirror-…: read and describe on the selected topics, describe on the groups, never a write). The password never leaves the platform and the nodes. - Create:
POST /kafka/api/replications {"name": "orders-dr", "sourceClusterId": "kfk-…", "targetClusterId": "kfk-…", "topics": ["orders", "payments.*"], "groups": ["*"]}(console: the target cluster's Replication page;dcloud kafka replications create; Terraformdcloud_kafka_replication). It needskafka:CreateKafkaReplicationon the target andkafka:ReplicateKafkaClusteron the source, so a cluster can be closed to replication whatever else is allowed. Both clusters must be available and in the same network; the target must have been created after 2026-11-15 (older nodes cannot run the worker: recreate the target from a configuration backup).creatinguntil every connector runs, thenrunning. - Refused: a cycle (A → B while B → A, or through other clusters: with names kept it would copy forever), two replications writing matching topics into one target (two sources would merge into one topic), a target that already has a managed topic the copy would write into, and more than 3 replications into one cluster. While a replication runs, creating or adopting a matching topic on the target is refused.
- Watch it:
lag(messages the copy is behind, over all copied topics, and per topic), the connectors and their tasks, the groups synced onto the copy, on the API and the console, refreshed every minute. Connect records how far it has copied every 5 seconds, so a healthy replication shows up to 5 seconds of messages (on dev: 400–1,200 at 200 msg/s; a burst of 3,000 msg/s was caught up within the minute). In Monitoring (EWS/Kafka, dimensionsClusterId= the target,ReplicationId):ReplicationLagandReplicationSourceReachable(1/0). When the source cannot be read,ReplicationSourceReachableis 0 and noReplicationLagis written (a zero would look healthy); Activity says so with the error. An alarm onReplicationSourceReachable < 1for 2 periods tells you when to consider failing over. Kafka Connect itself reports its tasks running while the source is gone, which is why the platform checks the source itself (the target's collector reads the source's end offsets as the mirror user). - Fail over:
POST /kafka/api/replications/{id}/failover(kafka:FailoverKafkaReplication; console Fail over). It works with the source down. Copying stops for good within a minute (failed_over; the connectors are removed, so a source that comes back cannot overwrite anything), andfailover.groupslists each group on the copy with its lag there. Then point consumers — same group id — and producers at the target's bootstrap servers. A consumer starts at its translated offset. Delivery is at-least-once: offsets are translated from sync points (one per 10 messages per partition), so a moved group re-reads a few messages per partition, and a group that committed in the last seconds before the source died may re-read up to the last 5-second sync. Messages the source acknowledged but had not yet copied are not on the copy (the lag at that moment). The copied topics become ordinary topics on the target (adopt them to manage them). There is no failback: replicate the other way into a fresh cluster. Measured on dev (source powered off while producing 200 msg/s and consuming): Activity said the source could not be read 36 s later, failover took 47 s, the moved group got its first message 3.3 s after it started on the copy; of 11,111 messages the source acknowledged none was missing, 435 were read twice (about 2 s of traffic). In a 6-minute outage without failover the copy kept every one of 54,098 acknowledged messages and caught up when the source returned. - Delete:
DELETE /kafka/api/replications/{id}stops copying; the copied topics stay; the mirror user on the source goes. It always works, whichever side is down. Deleting the target deletes its replications; deleting a source is refused while a replication reads from it (fail over or delete it first).
Near a full disk: the producer throttle
Kafka stops a broker whose data volume fills, and consumers lose it too. So when a node's volume passes 90 % the platform throttles producers: producer_byte_rate on every user (Kafka delays the answers, it does not refuse writes), per producer and per broker:
| Fullest volume | Each producer may write to each broker |
|---|---|
| ≥ 90 % | 1 MiB/s |
| ≥ 95 % | 64 KiB/s |
| ≥ 97 % | 1 KiB/s — effectively stopped: producers time out |
| every one < 85 % | no limit again (no restart, producers recover by themselves) |
A level is set within about half a minute of a node reporting the crossing (each node reports every 30 s), said in Activity, shown on the cluster page and as ProducerThrottled, and the health says producers_throttled. The limit is per producer user: many users together still write more (on dev, 10 users filled a 10 GiB volume from 91 % to 99 % in 3 minutes at level 1; level 3 then held it at 99.8 %), so keep retention.bytes on busy topics rather than rely on the throttle. A throttled producer keeps unsent records in memory: bound it (queue.buffering.max.kbytes in librdkafka, whose default is 1 GB; buffer.memory in Java) or a slowed producer can run its machine out of memory. Consumers are not throttled. Retention is never lowered for you: delete topics, lower retention.ms / retention.bytes, or move to a cluster with larger volumes.
Upgrade and patch
POST /kafka/api/clusters/{id}/upgrade { "version": "4.4.0" } rolls the cluster to a newer release (the same minor or the next; never down): one node at a time, the active controller last. Each node stops with a controlled shutdown (its partition leaders move first), restarts on the new image, and must be back in every in-sync replica set before the next node is taken. A node that does not come back pauses the rollout; .../maintenance/resume retries it, .../maintenance/cancel stops. After the last node the feature versions (metadata.version) are finalised — from then on the upgrade cannot be rolled back.
Operating system: security updates install nightly by themselves; POST .../patch patches now, the same one-node-at-a-time way, rebooting a node when its kernel asks.
With one node, an upgrade or patch is downtime.
Send events to Kafka
An event-bus rule can target a topic: { "type": "kafka", "cluster": "events", "topic": "orders", "key": "id" } (key: the event id — default, spreads evenly —, source or detail-type to keep those in order in one partition, or none). The target's role needs kafka:PublishMessage on the topic. Each target produces as its own user, allowed to write to that one topic, idempotently with acks=all; a delivery succeeds only when every in-sync replica has the event. Headers carry x-dcloud-event-id, x-dcloud-event-source, x-dcloud-event-detail-type, x-dcloud-delivery-id. A missing topic is a failed delivery (retried, then the dead-letter queue).
Delete a cluster
DELETE /kafka/api/clusters/{id} removes the machines, the volumes (and their keys), every topic and message, the users' secrets, the daily configuration backups and the default alarms still linked to it. Manual configuration backups stay.
Limits and failure behaviour
- Two zones. The region has two zones: nodes go de1a, de1b, de1a. Losing de1b leaves two nodes (a quorum): the cluster keeps working. Losing all of de1a stops the cluster — one node is not a quorum, and partitions with
min.insync.replicas2 refuseacks=allwrites — until de1a is back. No acknowledged write is lost either way. - One node is not replicated: a reboot stops the cluster, a lost machine loses the data.
- When a node dies, producers pause about 12 s (Kafka fences the dead broker after its 9 s session timeout, then partitions get new leaders). A consumer whose group coordinator was on that node can stall up to its own
session.timeout.ms(45 s by default) on top: lower it (e.g. 10 s) or use the new consumer protocol (group.protocol=consumer) where your client supports it. - A full volume stops a Kafka broker. Activity warns at 80 %; from 90 % producers are throttled (above); set
retention.bytes. - Backups are of definitions (topics, users), not messages; tiered storage is long retention, not a backup; a replication (MirrorMaker 2) is a live copy in another cluster of the same network, not cross-network or cross-organization. No customer Kafka Connect or Schema Registry yet; no mTLS client authentication.
- 1 or 3 nodes, fixed at creation.
Measured on dev
Filled in from the dev proof in claude-plans/managed-brokers.md (Kafka section).
Tiered storage (2026-11-15, 3 × performance-2, 20 GiB volumes):
- Turning it on for a running cluster: 4 minutes for three nodes (41–96 s each, the active controller last), with a producer writing 200 records/s throughout — every record was read back.
- 610 MB produced at ~10 MB/s with 64 MiB segments and
local.retention.msof a minute: within 5 minutes the brokers kept only the newest segments; the bucket held 624 MB, encrypted. - After a broker restart, reading from offset 0 returned all 600,000 records (99 s) and all 120,000 of a second topic (75 s), none missing, none twice.
- Deleting the cluster emptied and deleted its bucket within a minute.