# Fail over a cluster

Hand leadership to a standby that has caught up, without changing the endpoint.

A planned switchover hands leadership from the current leader to one of the cluster's two
standbys. Patroni waits for that standby to catch up before promoting it, so no committed
transaction is lost. Use it to move the leader back to `de1a` after a failover put it in `de1b`, or
to rehearse what your application does when the leader changes. Readers are never candidates.
Replace _ewdb-0123456789ab_ with your database's id.

## Prerequisites

- A cluster with a running leader.
- Permission for `rds:FailoverDBInstance`. The console shows **Switch over** only to someone who
  has it.
- In the console, also `rds:DescribeDBInstances` to open the database's page (without it the page is
  not found).

## Switch over

**Console**
1. Open the database, then choose the **Nodes** tab.
2. Choose **Switch over**. There is no confirmation; the request is sent at once, and the panel
   reloads the cluster's live state when it is done.

**CLI**
```sh
dcloud rds instances failover ewdb-0123456789ab
```

**API**
```http
POST /rds/api/instances/ewdb-0123456789ab/failover
```

The answer names the node that led before: `{ "success": true, "data": { "from": "<member>" } }`.

**Terraform:** not available. A failover is an action, not configuration.

The request is refused when:

| Answer | Message                                                                    |
| ------ | -------------------------------------------------------------------------- |
| `400`  | `A single-node database has nothing to fail over to`                       |
| `409`  | `The cluster has no leader to hand over`                                   |
| `502`  | Patroni's own refusal, or `No node answered` when no node could be reached |

## Result

A standby becomes the leader; the endpoint does not change, and clients reconnect to it. Writes
fail briefly while the leader changes. On a cluster spread over zones with its leader in `de1b`,
both standbys are in `de1a`, so the switchover brings the leader back there. **Nodes** shows the new `primary`, and `rds.instances.cluster` returns the cluster
membership and replication state, live from Patroni. Within about two minutes **Activity** records
the change (phase `failover`): `The leader moved from <old> to <new> (timeline <n>). Clients
reconnect to the same endpoint.`, followed by what that means for acknowledged commits.

## Automatic recovery

You do not fail over by hand when the leader stops answering: Patroni promotes a standby on its
own. Nor when a machine is lost:

- **Cluster:** a member whose machine no longer exists at the provider is recreated automatically
  with the same name, address and data volume, and rejoins as a replica. The platform checks every
  two minutes, and only while the cluster has a running leader, so if the leader was the machine
  lost, a standby is promoted first. The cluster stays available on its other members meanwhile
  (phase `repair` on **Activity**).
- **Single node:** rebuilt the same way, on its data volume and at its endpoint address, with its
  data and settings as they were. It is unavailable until the new machine is up. If the data volume
  itself is gone, nothing is rebuilt (an empty database at the old address would be worse than the
  outage): **Activity** says `The data volume … no longer exists, so … cannot be rebuilt with its
data. Restore the database from a backup.`

A single node whose endpoint address was reserved by an earlier release keeps it now too, but one
lost before the change came back at a new address. Its hostname follows; clients that connect by
IP need the new one.

Measured downtime for each kind of failure is in
[How Managed PostgreSQL behaves when things fail](failure-behaviour.md).

## Related

- [Stop, start or reboot a database](stop-start-or-reboot.md): a cluster reboot also hands over
  once.
- `rds.instances.failover` · `dcloud rds instances failover`
