Recipe: GitHub Actions runner pool
Self-hosted GitHub runners on an auto scaling group: ephemeral, busy-aware, on a schedule.
What it is
A pool of self-hosted GitHub Actions runners on Deutsche Cloud machines: an auto scaling group whose machines install GitHub's official runner at boot, register one ephemeral runner after another with your labels, protect themselves from scale-in while a job runs, and leave cleanly when the group shrinks. Sizes follow a schedule (60 machines on weekday office hours, 15 in the evening, 5 at night, all yours to change), and an optional scaler adds machines for queued jobs.
No GitHub credential ever reaches a machine. The GitHub App key (or token) lives in one Deutsche Cloud function, the pool's broker. A machine proves who it is with a five-minute instance identity token and gets back a just-in-time configuration for exactly one runner: named after the machine, with the pool's labels and runner group fixed by the broker. The machines have no instance role at all, so a job running as root on one finds nothing that could read the credential.
Your workflows do not change. A job with runs-on: hetzner-16 (or [self-hosted, hetzner-16]) runs on any runner carrying that label, on Deutsche Cloud or on the machines you run today.
The recipe is a handful of files the console serves:
| File | What it is |
|---|---|
setup.sh |
Creates the secret, broker, launch template, group, schedule and (optionally) the scaler with plain API calls. |
runner-user-data.sh |
The launch template's user data: installs the runner and the services below on Ubuntu 24.04. |
token-broker.mjs |
The broker, a Deutsche Cloud function (Node.js 22): instance identity token in, one runner configuration out. |
github-auth.mjs |
GitHub App or token credentials, shared by the broker and the scaler. |
scaler.mjs |
The queue-depth scaler, a Deutsche Cloud function (Node.js 22). |
teardown.sh |
Rollback (desired capacity 0), or removal of everything setup.sh created. |
prove.sh |
Builds a small pool on a throwaway repository, runs the tests below against it and removes it. |
pool-burst.yml |
A test workflow: ten jobs at once on the pool's label. pool-long.yml: one long job. |
mkdir runner-pool && cd runner-pool
for f in setup.sh runner-user-data.sh token-broker.mjs github-auth.mjs scaler.mjs teardown.sh prove.sh \
pool-burst.yml pool-long.yml; do
curl -fsSLO "https://console.platform-development.deutschecloud.app/recipes/github-runner-pool/$f"
done
chmod +x setup.sh teardown.sh prove.shThey need bash, curl and jq, and an access key allowed to create what is listed under Reference → What setup creates (the AdministratorAccess policy is enough).
Concepts
| Term | Meaning |
|---|---|
| Registration credential | A GitHub App installed on your organization or repository (recommended), or a token, allowed to manage self-hosted runners. Stored once, as a Deutsche Cloud secret bound to the broker function only. |
| Broker | The function <POOL_NAME>-broker. Its URL is public, but it answers only an instance identity token signed by Deutsche Cloud for its URL, from a machine of your organization and of this pool's group, and hands that machine one runner. |
| Ephemeral runner | A runner for exactly one job (a just-in-time configuration). GitHub removes it after the job; the machine then asks the broker for the next one and starts with an empty work directory. The machine stays, so its paid hour is used. |
| Busy | A job is running. The runner's job-started hook protects the machine from scale-in; its job-completed hook lifts the protection. |
| Drain | When the group picks a machine to leave, the machine stops registering, lets a running job finish, deregisters its runner and tells the group it may go (the group's termination lifecycle hook drain). A job that starts on the idle runner just as the group picks the machine keeps it: its protect call cancels that scale-in and another machine leaves. |
| Floor | The group's minSize. Each scheduled step sets the floor and the capacity; the scaler moves the capacity between the floor and maxSize. |
| Paid hour | Machines are billed per started hour. The group is created with deferScaleInUntilPaidHour: a machine picked for scale-in keeps taking jobs until the last 10 minutes of its hour, and the one whose hour ends first goes first. |
How to
Create the GitHub credential
A GitHub App (recommended). Its credentials belong to no person, are scoped to where you install it, and every call uses an installation token that lasts an hour. In GitHub: your organization → Settings → Developer settings → GitHub Apps → New GitHub App. No webhook. Permissions:
| Runners for | RUNNER_URL |
Permission |
|---|---|---|
| every repository of an organization (default) | https://github.com/ORG |
Organization → Self-hosted runners: Read and write |
| one repository | https://github.com/ORG/REPO |
Repository → Administration: Read and write |
| the scaler (optional) | the repositories it watches | Repository → Actions: Read |
Create it, note its App ID, generate a private key (a .pem file is downloaded), and Install App on the organization (or the one repository). setup.sh stores the key as a Deutsche Cloud secret and finds the installation itself.
Or a token. A fine-grained personal access token with the same permissions (GitHub → Settings → Developer settings). Simpler, but tied to the person who made it. Give the scaler a separate read-only token (SCALER_TOKEN_SECRET_ID, a secret you create under Secrets) when you can; otherwise it uses the registration credential.
Set up the pool
export EWS_API_URL=https://console.platform-development.deutschecloud.app
export EWS_ACCESS_KEY=EWS1…:… # an administrator's key, used only here
export RUNNER_URL=https://github.com/ORG
export RUNNER_LABELS=hetzner-16
export GITHUB_APP_ID=123456 GITHUB_APP_PRIVATE_KEY_FILE=./acme-runners.2026-11-14.private-key.pem
# or, with a token: read -rs RUNNER_ADMIN_TOKEN && export RUNNER_ADMIN_TOKEN (not echoed)
./setup.shsecret 3f0c… (GITHUB_APP_PRIVATE_KEY)
broker https://github-runners-broker-k3x9.fn.<root>
template lt-1a2b3c4d (performance-16)
group asg-5e6f7a8b (max 70, start 0, hook drain 3600s, paid-hour scale-in)
schedule day: 60 at '30 6 * * 1-5' (UTC)
schedule evening: 15 at '0 17 * * 1-5' (UTC)
schedule night: 5 at '0 23 * * *' (UTC)
done: github-runners.state.jsonThe group starts at 0 machines and follows the schedule from its next step. Keep github-runners.state.json: teardown.sh reads it. Every setting is an environment variable; see Reference → Settings.
To start runners now instead of at the next step:
curl -X PUT "$EWS_API_URL/autoscaling/api/groups/asg-5e6f7a8b/desired-capacity" \
-H "Authorization: EWS-ACCESS-KEY $EWS_ACCESS_KEY" -H 'Content-Type: application/json' \
-d '{"desiredCapacity": 5}'A machine runs its first job about 90 seconds after it is launched (Ubuntu 24.04, Docker and the runner installed at boot). It appears in GitHub under Settings → Actions → Runners as its instance id (i-…), with your labels. The broker's log (Functions → github-runners-broker → Logs) has one line per runner it handed out: which machine, which provider, which name.
Change the schedule
Each step is a scheduled action of the group (five-field cron, UTC) that sets the floor and the capacity. Set the sizes and times when you run setup.sh:
DAY_SIZE=60 DAY_CRON='30 6 * * 1-5' EVENING_SIZE=15 EVENING_CRON='0 17 * * 1-5' \
NIGHT_SIZE=5 NIGHT_CRON='0 23 * * *' ./setup.shor later on the group: Auto Scaling → the group → Scheduled actions, or
curl -X PUT "$EWS_API_URL/autoscaling/api/groups/asg-5e6f7a8b/scheduled-actions" \
-H "Authorization: EWS-ACCESS-KEY $EWS_ACCESS_KEY" -H 'Content-Type: application/json' \
-d '{"name": "day", "recurrence": "45 5 * * 1-5", "minSize": 60, "desiredCapacity": 60}'A step with the name of an existing one replaces it. Cron runs in UTC: 07:30 in Germany is 30 6 in winter and 30 5 in summer. Start the morning step 15 minutes before the first jobs: sixty machines launch within a minute and run jobs about 90 seconds later.
Scale on queued jobs (optional)
With SCALER=1, setup.sh also creates a function and a schedule that calls it once a minute:
SCALER=1 SCALER_REPOSITORIES=ORG/web,ORG/api SCALER_TOKEN_SECRET_ID=7d1e… ./setup.shEach call counts the machines that are busy (protected) and the jobs GitHub has queued for the pool's labels in those repositories, and sets the capacity to busy + queued + SCALER_SPARE, never below the floor and never above maxSize. When the queue empties it sets the capacity back to the floor; the idle machines then finish their paid hour and drain. Its result is in the function's logs (Functions → github-runners-scaler → Logs) as one JSON line per call:
{"groupId":"asg-5e6f7a8b","minSize":15,"maxSize":70,"desired":15,"busy":15,"queued":7,"want":22,"apiCalls":9,"changed":true}Each call makes 2 × repositories + queued and running workflow runs GitHub API requests (apiCalls in the log). A personal token allows 5,000 an hour, about 80 a minute.
Roll back
Scale the pool to zero and keep everything else:
./teardown.shThis pauses the scaler, removes the scheduled steps and sets the floor and the capacity to 0. Running jobs finish (their machines are protected); idle machines drain and go. GitHub sends hetzner-16 jobs to the runners you still have elsewhere. To bring the pool back, run the PUT …/scheduled-actions calls above again, or delete it and run setup.sh again.
In a hurry, one call does the same for the group alone:
curl -X PATCH "$EWS_API_URL/autoscaling/api/groups/asg-5e6f7a8b" \
-H "Authorization: EWS-ACCESS-KEY $EWS_ACCESS_KEY" -H 'Content-Type: application/json' \
-d '{"minSize": 0, "desiredCapacity": 0}'(The scheduled steps then raise it again at their next time.)
Remove the pool
./teardown.sh --deleteRolls back, waits until every machine has drained, then deletes the group, the launch template, the broker, the scaler with its role and policy, and the secret if setup.sh created it.
Rotate the GitHub credential
Change the secret's value (Secrets → the secret → Rotate, or POST /secrets/api/secrets/{id}/rotate with { "value": "…" }): a new App private key (generate it in the App's settings, rotate, then delete the old key in GitHub) or a new token. The broker's instances pick it up when they next start (within its idle timeout, 5 minutes); machines are not touched.
Upgrading a pool made before 2026-11-14
Earlier versions of this recipe gave each machine an instance role that could read the credential. Run ./teardown.sh --delete with the old files, then setup.sh with the new ones: the new pool has no runner role.
Try it on a throwaway repository first
prove.sh builds a small pool (performance-2, at most 4 machines, label pool-proof, scaler on) for a repository you create for the purpose, and checks the three things that matter, then removes the pool:
# EWS_API_URL and EWS_ACCESS_KEY set as above
export GITHUB_REPO=you/runner-pool-test # a new repository with one commit
read -rs GITHUB_TOKEN && export GITHUB_TOKEN # fine-grained, that repository only: Administration,
# Actions, Contents and Workflows, read and write
./prove.sh- It commits
pool-burst.ymlandpool-long.ymlto.github/workflows/and starts pool-burst: ten jobs at once. All ten must succeed, on more than one machine; the scaler grows the group for them. - It pauses the scaler, starts pool-long (one 10-minute job) and sets the capacity to 0. The machine running the job must stay in service until the job has succeeded; the idle machines drain and go.
- The group must reach 0 machines. Then
teardown.sh --deleteremoves the pool.
Each step prints the group's machines (* marks a protected one). It takes about 20 minutes and costs a few cents.
Reference
Settings
Environment variables of setup.sh:
| Variable | Default | Meaning |
|---|---|---|
RUNNER_URL |
required | https://github.com/ORG or https://github.com/ORG/REPO |
GITHUB_APP_ID |
— | a GitHub App's id: the broker and scaler use the App |
GITHUB_APP_PRIVATE_KEY_FILE |
with GITHUB_APP_ID |
the App's private key (.pem), stored as a new secret |
GITHUB_APP_INSTALLATION_ID |
looked up | the App's installation, if you want to name it |
RUNNER_ADMIN_TOKEN |
without an App | a GitHub token, stored as a new secret |
TOKEN_SECRET_ID |
— | an existing secret holding the App key or the token instead |
RUNNER_LABELS |
hetzner-16 |
comma-separated; the runner adds self-hosted, Linux, X64 itself |
RUNNER_GROUP |
Default | an organization runner group |
POOL_NAME |
github-runners |
names the group, template, roles, function; the state file is <POOL_NAME>.state.json |
SERVER_TYPE |
performance-16 |
16 vCPU, 32 GB |
IMAGE_ID / MACHINE_IMAGE_ID |
Ubuntu 24.04 | a catalog image, or a machine image of yours (ewi-…) with warm caches |
INSTALL_DOCKER |
1 |
install Docker and let jobs use it |
ZONES |
de1a,de1b |
Falkenstein and Nuremberg; machines are spread evenly |
MAX_SIZE |
70 |
at most 100 |
START_SIZE |
0 |
capacity at creation |
HOOK_TIMEOUT |
3600 |
the longest a leaving machine waits for its running job, in seconds (30–7200) |
DAY_*, EVENING_*, NIGHT_* |
60 / 15 / 5, see above | _SIZE and _CRON of each step; SCHEDULES=0 creates none |
SCALER |
0 |
1 creates the queue-depth scaler |
SCALER_REPOSITORIES |
— | ORG/REPO,… the scaler watches |
SCALER_TOKEN_SECRET_ID |
the registration credential | a secret with a read-only (Actions: Read) token |
SCALER_SPARE |
0 |
idle machines to keep above the demand |
SSH_KEY_PAIR_ID |
none | an SSH key pair, to log in to machines |
TAGS_JSON |
{} |
extra tags on every machine, e.g. {"team":"ci"}; pool=<POOL_NAME> is always set |
GITHUB_API |
https://api.github.com |
GitHub Enterprise Server: https://HOST/api/v3 |
What setup creates
| Resource | Permission it needs |
|---|---|
Secret <POOL_NAME>_GITHUB_APP_PRIVATE_KEY (or …_GITHUB_TOKEN) |
secrets:CreateSecret (with a KMS key) |
Function <POOL_NAME>-broker, public URL, the secret bound, no role |
functions:CreateFunction, functions:MakeFunctionPublic, secrets:GetSecretValue on the secret (to bind it), functions:UpdateFunctionConfiguration |
Launch template <POOL_NAME>, no instance role |
autoscaling:CreateLaunchTemplate |
Group <POOL_NAME> + lifecycle hook drain |
autoscaling:CreateAutoScalingGroup, ec2:RunInstances, autoscaling:PutLifecycleHook |
Scheduled actions day, evening, night |
autoscaling:PutScheduledUpdateGroupAction |
Scaler: policy + role <POOL_NAME>-scaler, function and schedule <POOL_NAME>-scaler |
the role may describe the group, set its capacity, invoke the function and read CI runs; creating them needs functions:CreateFunction, scheduler:CreateSchedule, iam:PassRole |
On each machine
| Path / unit | What it is |
|---|---|
/opt/ews-runner/runner |
the runner, owned by user runner |
ews-runner.service |
the loop: ask the broker, run.sh --jitconfig …, wipe _work, again |
ews-runner-drain.service |
polls ews-instance lifecycle every 10 s; on leaving: no new registration, wait for the job, stop, deregister, ews-instance complete |
/opt/ews-runner/ews-runner-token |
root only: ews-instance token <broker URL> → POST to the broker (register or deregister) |
/etc/ews-runner/pool.env |
the broker's URL; nothing secret |
/opt/ews-runner/job-started.sh |
ACTIONS_RUNNER_HOOK_JOB_STARTED: ews-instance protect on |
/opt/ews-runner/job-completed.sh |
ACTIONS_RUNNER_HOOK_JOB_COMPLETED: ews-instance protect off |
/etc/sudoers.d/ews-runner |
lets user runner run exactly ews-instance protect on and … protect off as root |
/var/log/ews-runner-bootstrap.log |
the bootstrap's result; journalctl -u ews-runner has every registration and job |
The runner is pinned to 2.338.2 with its SHA-256 and runs with --disableupdate. GitHub stops sending jobs to runner versions it no longer supports; set RUNNER_VERSION and its checksums at the top of runner-user-data.sh to a current release and roll the group onto the new template version (Auto Scaling → the group → Instance refresh).
What a job can reach
Jobs run as user runner and cannot read the instance identity (/etc/ews/instance-identity, root only). With INSTALL_DOCKER=1 the runner user is in the docker group, which is root on that machine, so plan for a job that is root. What root on a pool machine can get:
- No GitHub credential. The App key or token is a secret bound to the broker function only. The machine has no instance role and no access key.
- An identity token (
ews-instance token), worth five minutes, for one audience. - From the broker, nothing more than the machine already has. On GitHub a runner's name is unique: the broker names every runner after its machine, so a machine has at most one, and while that runner runs a job (the job asking is one) the broker refuses a second. The labels and the runner group are the broker's, not the machine's. It may deregister its own runner, which only ends its own job.
- Its own scale-in protection and lifecycle (
ews-instance protect,complete): it can keep itself or let itself go, nothing about other machines.
Each token a machine gets is in the audit log (ec2:GetInstanceIdentityToken, with the audience), and each runner the broker hands out is in the broker's log. Run untrusted pull requests from forks on GitHub's hosted runners all the same: a job that is root can still leave things behind for the next job on the same machine, which is why _work is wiped and Docker is shared (use INSTALL_DOCKER=0 or a fresh machine per job where that matters).
On Deutsche Cloud CI (PROVIDER=ews) the broker hands out a one-time registration token instead, with labels and repository pinned by the token, and refuses while the machine's runner is online. The token does not bind the runner's name, so a root job that first stops its own runner could still register one more runner with the pool's labels; GitHub's just-in-time configuration does not have that gap.
Cost
Prices at the time of writing, per machine, including its public IPv4 (see /pricing for today's):
| Server type | vCPU / RAM | Per hour | Monthly cap per machine |
|---|---|---|---|
performance-16 |
16 / 32 GB | €0.2091 | €130.49 |
performance-8 |
8 / 16 GB | €0.1122 | — |
dedicated-8 |
8 dedicated / 32 GB | €0.2227 | — |
The default schedule with performance-16:
| Step | Machines | Per hour | Hours a week | Per week |
|---|---|---|---|---|
| day, Mon–Fri 06:30–17:00 UTC | 60 | €12.55 | 52.5 | €658.66 |
| evening, Mon–Fri 17:00–23:00 | 15 | €3.14 | 30 | €94.10 |
| night and weekend | 5 | €1.05 | 85.5 | €89.39 |
| Total | 168 | ≈ €842 a week, ≈ €3,660 a month |
The scaler adds what queued jobs need above the schedule, at the same hourly price; its own calls are a function at well under €1 a month. Each scale-in step costs nothing extra: a machine leaves at the end of an hour already paid.
What is not supported
- Windows and macOS runners; Linux x64 and arm64 (
caxserver types) only. - Scaling on GitHub's
workflow_jobwebhook; the scaler polls once a minute. - Jobs waiting on
needs:are counted by the scaler only once GitHub lists them as queued.
Troubleshooting
| Symptom | Cause | Fix |
|---|---|---|
| Machines are in service but no runner shows in GitHub | the App lacks the permission or is not installed on RUNNER_URL, or the token expired |
the broker's log names the GitHub error; journalctl -u ews-runner says the broker gave no runner config |
broker: HTTP 403 not a machine of this pool |
the machine is not in the group the broker serves (launched by hand from the template) | launch through the group |
broker: HTTP 409 … is running a job |
the machine's runner is busy (a job asked for a second runner) | expected; the loop asks again after the job |
ews-runner-bootstrap: no ews-instance helper |
the machine was launched without an instance identity (a machine image taken before 2026-11-13) | launch from the catalog image or a newer machine image |
| The group stays above its capacity | machines are busy; scaleInBlockedByProtection in the group says how many |
nothing: they leave when their job ends |
A machine stays in terminating_wait |
it is finishing a job that started just before it was picked | it goes when the job ends, or after HOOK_TIMEOUT |
A job's log says could not protect this machine from scale-in |
its machine was leaving for another reason than scale-in (terminated by hand, unhealthy) | nothing: the drain waits for the job before the machine goes (up to HOOK_TIMEOUT) |
| Idle machines stay up to 50 minutes after the capacity drops | paid-hour scale-in keeps them until the last 10 minutes of their hour | expected; they keep taking jobs meanwhile |
| Jobs stay queued although machines are idle | the job names a label the pool lacks | compare the job's runs-on with RUNNER_LABELS |
The scaler's log says GitHub …: HTTP 403 |
the scaler's token lacks Actions: Read on a repository, or hit the rate limit | fix the token, or watch fewer repositories |