Fail over a cluster
Hand leadership to a standby that has caught up, without changing the endpoint.
A planned switchover hands leadership from the current leader to one of the cluster's two
standbys. Patroni waits for that standby to catch up before promoting it, so no committed
transaction is lost. Use it to move the leader back to de1a after a failover put it in de1b, or
to rehearse what your application does when the leader changes. Readers are never candidates.
Replace ewdb-0123456789ab with your database's id.
Prerequisites
- A cluster with a running leader.
- Permission for
rds:FailoverDBInstance. The console shows Switch over only to someone who has it. - In the console, also
rds:DescribeDBInstancesto open the database's page (without it the page is not found).
Switch over
- Open the database, then choose the Nodes tab.
- Choose Switch over. There is no confirmation; the request is sent at once, and the panel reloads the cluster's live state when it is done.
dcloud rds instances failover ewdb-0123456789abPOST /rds/api/instances/ewdb-0123456789ab/failoverThe answer names the node that led before: { "success": true, "data": { "from": "<member>" } }.
Not available in Terraform. A failover is an action, not configuration.
The request is refused when:
| Answer | Message |
|---|---|
400 |
A single-node database has nothing to fail over to |
409 |
The cluster has no leader to hand over |
502 |
Patroni's own refusal, or No node answered when no node could be reached |
Result
A standby becomes the leader; the endpoint does not change, and clients reconnect to it. Writes
fail briefly while the leader changes. On a cluster spread over zones with its leader in de1b,
both standbys are in de1a, so the switchover brings the leader back there. Nodes shows the new primary, and GET /rds/api/instances/{id}/cluster returns the cluster
membership and replication state, live from Patroni. Within about two minutes Activity records
the change (phase failover): The leader moved from <old> to <new> (timeline <n>). Clients reconnect to the same endpoint., followed by what that means for acknowledged commits.
Automatic recovery
You do not fail over by hand when the leader stops answering: Patroni promotes a standby on its own. Nor when a machine is lost:
- Cluster: a member whose machine no longer exists at the provider is recreated automatically
with the same name, address and data volume, and rejoins as a replica. The platform checks every
two minutes, and only while the cluster has a running leader, so if the leader was the machine
lost, a standby is promoted first. The cluster stays available on its other members meanwhile
(phase
repairon Activity). - Single node: rebuilt the same way, on its data volume and at its endpoint address, with its
data and settings as they were. It is unavailable until the new machine is up. If the data volume
itself is gone, nothing is rebuilt (an empty database at the old address would be worse than the
outage): Activity says
The data volume … no longer exists, so … cannot be rebuilt with its data. Restore the database from a backup.
A single node whose endpoint address was reserved by an earlier release keeps it now too, but one lost before the change came back at a new address. Its hostname follows; clients that connect by IP need the new one.
Measured downtime for each kind of failure is in How Managed PostgreSQL behaves when things fail.
Related
- Stop, start or reboot a database: a cluster reboot also hands over once.
POST /rds/api/instances/{id}/failover·dcloud rds instances failover