Auto Scaling
Keep machines at a size across zones; scale on CPU, alarms or a schedule.
What it is
An auto scaling group keeps a number of identical machines running, spread evenly across the availability zones you choose, registered with your load balancers, and replaced when they fail. It grows and shrinks on a metric, on any monitoring alarm, or on a schedule. Find it under Auto Scaling in the console, or at /autoscaling/api.
The machines are ordinary Compute instances: you see them under Compute → Instances, reach them over SSH, and pay for them as instances. What the group adds is the decision of how many there are and where.
Concepts
| Term | Meaning |
|---|---|
| Launch template | What the group's machines are made from: server type, image, network, SSH key, security groups, user data, tags. Id lt-…. |
| Version | Every change to a template is a new numbered version; a version never changes. A group launches $Default, $Latest, or a fixed number. |
| Group | Id asg-…. minSize ≤ desiredCapacity ≤ maxSize, a list of zones, optionally load balancers and subnets. At most 20 machines. |
| Desired capacity | How many machines the group runs now. You set it, or a policy or schedule does. |
| Health check | ec2: the machine exists and is powered on. lb: also passes the health check of every load balancer the group names, after the grace period. |
| Scaling policy | Target tracking (keep a metric near a value) or steps on a monitoring alarm. |
| Scheduled action | A change of min, max or desired at a time (cron, UTC) or once. |
| Instance refresh | Replace every machine that is not on the group's current template version. |
| Activity | Everything the group decided, with the reason: launches, terminations, health, scaling, schedules. |
How to
Create a launch template
POST /autoscaling/api/launch-templates
{
"name": "web",
"regionId": "de1",
"spec": {
"serverType": "performance-2",
"imageId": 114690387,
"securityGroupIds": ["3f0c…"],
"userData": "#!/bin/bash\napt-get install -y nginx"
}
}spec takes the fields of an instance launch except where it goes (region, zone, subnet) and its name. A change is a new version: POST /autoscaling/api/launch-templates/{id}/versions with a new spec, and "makeDefault": true to make it what $Default means.
Create a group
POST /autoscaling/api/groups
{
"name": "web",
"regionId": "de1",
"launchTemplateId": "lt-1a2b3c4d",
"minSize": 2,
"maxSize": 6,
"availabilityZoneIds": ["de1a", "de1b"],
"loadBalancerIds": ["9c2e…"],
"healthCheckType": "lb",
"healthCheckGracePeriodSeconds": 120
}Creating a group also needs ec2:RunInstances: the group launches machines on your authority. Within a minute it launches up to its desired capacity (default: minSize), each machine into the zone that has the fewest, and registers each one with every load balancer in loadBalancerIds once it is running. A zone that refuses a launch (out of stock for the server type, say) is skipped and the machine goes to another zone; the refusal is in the activity.
Watch it
Auto Scaling → the group shows the machines with their zone, state, health and template version, the policies and schedules, and the activity:
GET /autoscaling/api/groups/{id}Replace failed machines
Nothing to do. A machine that disappears, is powered off, or is terminated through the instance API is replaced within about a minute. With "healthCheckType": "lb", a machine that fails its load balancer's health check for 60 seconds after the grace period is deregistered, terminated and replaced. To take a machine out yourself:
POST /autoscaling/api/groups/{id}/instances/{instanceId}/terminate
{ "decrementDesiredCapacity": false }false (the default) replaces it; true shrinks the group by one.
Scale on CPU
PUT /autoscaling/api/groups/{id}/policies
{ "type": "target_tracking", "name": "cpu-50", "metric": "cpu_percent", "targetValue": 50 }The group's CPU is the average over its machines that have been in service longer than the warmup (defaultInstanceWarmupSeconds, default 300), over the last five minutes, as a percentage of each machine: 100 means every vCPU busy, whatever the server type. The new capacity is ceil(measured machines × CPU / target): two machines at 90 % against 50 % become four. A machine still booting or warming up is not measured, so it is not counted twice: the group does not scale out again for load the last scale-out already answered. Scaling out happens at once; scaling in waits until no machine is warming, CPU is 10 % under target and the group's cooldown (defaultCooldownSeconds, default 300) has passed.
Scale on any alarm
Create an alarm under Monitoring → Alarms on any metric, including your own custom metrics, then:
PUT /autoscaling/api/groups/{id}/policies
{
"type": "step",
"name": "queue-backlog",
"alarmId": "5d1e…",
"adjustmentType": "change_in_capacity",
"stepAdjustments": [
{ "lowerBound": 0, "upperBound": 100, "adjustment": 1 },
{ "lowerBound": 100, "upperBound": null, "adjustment": 3 }
]
}While the alarm is in ALARM, the step whose [lowerBound, upperBound) holds the breach (the metric minus the alarm's threshold) applies, at most once per cooldown. adjustmentType is change_in_capacity, exact_capacity or percent_change_in_capacity. For an alarm on a custom metric the breach counts as 0, so the step containing 0 applies.
Scale on a schedule
PUT /autoscaling/api/groups/{id}/scheduled-actions
{ "name": "office-hours", "recurrence": "0 7 * * 1-5", "minSize": 4, "desiredCapacity": 4 }recurrence is five-field cron in UTC. Without one, give startTime and the action runs once.
Roll out a new template version
POST /autoscaling/api/groups/{id}/instance-refreshes
{ "instanceWarmupSeconds": 120 }The group launches one extra machine on the current version; once it has served for the warmup and is healthy, one old machine is deregistered and terminated, and so on until none is left. The group never drops below its desired capacity, and costs one extra machine while it runs. POST …/instance-refreshes/cancel stops it where it is.
Keep scaling when a type is sold out
The provider sells from live stock per zone. A group always leaves a zone that is sold out of its type for its other zones (capacityFallback.otherZones, on by default; turn it off to keep the spread strict). To keep launching when the type is sold out everywhere, list fallback types, in order, and/or allow equivalent types:
{ "instanceTypes": ["performance-2", "performance-4"], "capacityFallback": { "otherZones": true, "equivalentTypes": true } }On POST or PATCH /autoscaling/api/groups/{id}. Each machine is launched as the first of the template's type, the listed types, then the equivalents (same CPU architecture, same CPU class or better, at least the vCPUs and memory, at most 1.25× the price plus 5 cents) that its zone sells, keeping the zone before the type. Listed types must have the template's architecture (no Arm types are sold at the moment). A substitution is in the group's activity ("performance-2 is sold out in de1a: fell back: performance-4 instead of performance-2") and raises an Insufficient Capacity event; the machine is billed as the type bought. When nothing allowed is in stock, the group logs it and tries again every minute.
Keep busy machines (CI runners, workers)
A machine that is doing work nobody should cut off (a CI runner in the middle of a job) protects itself from scale-in while busy and lifts the protection when idle:
dcloud autoscaling groups set-instance-protection asg-1a2b3c4d i-5e6f7a8b --protected-from-scale-in
dcloud autoscaling groups set-instance-protection asg-1a2b3c4d i-5e6f7a8b --no-protected-from-scale-inOn the machine itself no key is needed: its instance identity lets it protect itself and nothing else (see Compute, "Give a machine its own identity"):
ews-instance protect on # job started
ews-instance protect off # job finished
ews-instance lifecycle # {"success": true, "data": {"leaving": true, …}} once it is picked to go: stop taking jobs
ews-instance complete # deregistered and done: terminate me nowA GitHub Actions runner does this from its job-started and job-completed hooks (ACTIONS_RUNNER_HOOK_JOB_STARTED, ACTIONS_RUNNER_HOOK_JOB_COMPLETED), and a small loop polls lifecycle to stop the runner service and complete once it is idle. Recipe: GitHub Actions runner pool sets up a whole pool this way, with a schedule and a queue-depth scaler.
Scale-in only picks unprotected machines, so the idle ones go first. When every machine above the desired capacity is protected, the group stays above it until one is free; GET …/groups/{id} says so in scaleInBlockedByProtection.
A termination lifecycle hook covers the rest (a machine picked before it could protect itself, a replaced unhealthy machine, a hand-made terminate): a leaving machine waits in terminating_wait, out of its load balancers, until it completes the action or the heartbeat timeout passes:
dcloud autoscaling groups put-lifecycle-hook asg-1a2b3c4d --name drain --heartbeat-timeout-seconds 1800
# on the leaving machine, or from anywhere with the permission, when the job is done:
dcloud autoscaling groups complete-lifecycle-action asg-1a2b3c4d i-5e6f7a8b
# still busy? hold it one more timeout (48 hours at most in all):
dcloud autoscaling groups record-lifecycle-heartbeat asg-1a2b3c4d i-5e6f7a8bOne hook per group (termination only). defaultResult (CONTINUE or ABANDON) is what the timeout counts as; for a termination hook both terminate the machine. Deleting the hook lets every waiting machine go.
A machine that scale-in picked a moment ago, and that asks for protection while the hook still holds it (ews-instance protect on: a job started on it before it noticed it was leaving), is kept: back in_service, protected, and the group picks another machine on its next pass. The answer has "scaleInCancelled": true, the group's activity says so, and lifecycle reports it in service at once. This is unlike AWS, which refuses. Only a scale-in wait whose action is not completed and whose timeout has not passed is undone; a machine leaving because it is unhealthy, replaced by an instance refresh, terminated by hand or by a group deletion still gets 409.
Pay only for hours you use
Cloud machines are billed per started hour, counted from each machine's creation. Every scale-in takes, among otherwise equal machines, the one whose paid hour ends first. With "deferScaleInUntilPaidHour": true (on create or PATCH) the paid hour comes before the zone spread, and a machine picked for scale-in keeps serving until the last 5 minutes of its hour (never more than 55 minutes): the hour is paid anyway. A group on your own hosts has no paid hour (the provider bills no increment), so a picked machine leaves at once. The machine shows scaleInDeferredUntil, and the group's activity says when it goes. If the group scales out again in the meantime, the machine simply stays.
Pause the group
Suspending a step pauses only that part of the group: its machines keep running, and the step picks up again when you resume it. In the console, click a step under Suspended on the group's page. Through the API, PATCH /autoscaling/api/groups/{id} with "suspendedProcesses": ["ReplaceUnhealthy"] pauses that step; an empty list resumes everything.
| Console label | API name |
|---|---|
| Launching new machines | Launch |
| Removing machines | Terminate |
| Health checks | HealthCheck |
| Replacing unhealthy machines | ReplaceUnhealthy |
| Scaling on alarms | AlarmNotification |
| Scheduled scaling | ScheduledActions |
Delete a group
DELETE /autoscaling/api/groups/{id}: its machines are deregistered and terminated, then the group goes.
Run outside Germany (Helsinki, fi1)
Groups are in Germany (de1) unless your organization decides otherwise. After an administrator turns on Allow EU locations outside Germany (Settings → Data location), create the launch template and the group with "regionId": "fi1": its machines are launched in Helsinki, Finland (zone fi1a), and its network, subnets and load balancer must be in fi1 too. Without the setting the create answers 403.
Reference
API
| Operation | Permission |
|---|---|
GET /autoscaling/api/launch-templates, …/{id} |
autoscaling:DescribeLaunchTemplates |
POST /autoscaling/api/launch-templates |
autoscaling:CreateLaunchTemplate |
POST …/launch-templates/{id}/versions, PATCH …/launch-templates/{id} |
autoscaling:CreateLaunchTemplateVersion |
DELETE …/launch-templates/{id} |
autoscaling:DeleteLaunchTemplate |
GET /autoscaling/api/groups, …/{id} |
autoscaling:DescribeAutoScalingGroups |
POST /autoscaling/api/groups |
autoscaling:CreateAutoScalingGroup and ec2:RunInstances |
PATCH …/groups/{id} |
autoscaling:UpdateAutoScalingGroup |
PUT …/instances/{instanceId}/protection |
autoscaling:SetInstanceProtection (UpdateAutoScalingGroup still works for now) |
PUT …/groups/{id}/lifecycle-hooks, DELETE …/lifecycle-hooks/{name} |
autoscaling:PutLifecycleHook / autoscaling:DeleteLifecycleHook |
POST …/instances/{instanceId}/complete-lifecycle-action |
autoscaling:CompleteLifecycleAction |
POST …/instances/{instanceId}/lifecycle-heartbeat |
autoscaling:RecordLifecycleActionHeartbeat |
PUT …/groups/{id}/desired-capacity |
autoscaling:SetDesiredCapacity |
POST …/groups/{id}/instances/{instanceId}/terminate |
autoscaling:TerminateInstanceInAutoScalingGroup |
DELETE …/groups/{id} |
autoscaling:DeleteAutoScalingGroup |
PUT …/groups/{id}/policies, DELETE …/policies/{policyId} |
autoscaling:PutScalingPolicy / autoscaling:DeletePolicy |
PUT …/groups/{id}/scheduled-actions, DELETE …/{actionId} |
autoscaling:PutScheduledUpdateGroupAction / autoscaling:DeleteScheduledAction |
POST …/groups/{id}/instance-refreshes, …/cancel |
autoscaling:StartInstanceRefresh / autoscaling:CancelInstanceRefresh |
Every write is in the audit log. The CLI has the same operations (dcloud autoscaling groups create …), and each is an MCP tool.
How a group decides
Once a minute, and right after every change, the group compares what it has with what it should have and takes one step: it never waits for a machine to boot. In order: machines that vanished or failed are replaced; scheduled actions and policies set the desired capacity; an instance refresh retires one old machine if a new one is ready; machines are launched (up to 20 per pass, five at a time; a group still short gets its next pass a few seconds later) or taken out. Scale-in takes from the zone with the most machines, then an old template version, then a machine still booting, then the one whose paid hour ends first; a machine protected from scale-in (PUT …/instances/{id}/protection) is never chosen. A leaving machine goes through the termination lifecycle hook when the group has one. A restart of the platform in the middle loses nothing: a machine that was created but not yet recorded is found again by its label.
What is not supported
- Launch lifecycle hooks, more than one hook per group, hook notifications to a queue (the machine or a script polls instead), warm pools, predictive scaling.
- More than 100 machines in one group.
- Groups across regions. A group lives in one region and spreads over its zones.
- Target tracking on custom metrics (use a step policy on an alarm).
Troubleshooting
The group stays below its desired capacity. Read the activity: a launch refused for stock (409 … out of stock) names the zone, and a broken template (a deleted image, a missing security group) names what is missing. Fix the template with a new version.
Machines are replaced over and over. With healthCheckType: lb, a machine that does not answer the load balancer's health check within the grace period is replaced. Raise healthCheckGracePeriodSeconds to cover your boot, or check the health check path.