Recipe: GitHub Actions runner pool

Self-hosted GitHub runners on an auto scaling group: ephemeral, busy-aware, on a schedule.

What it is

A pool of self-hosted GitHub Actions runners on Deutsche Cloud machines: an auto scaling group whose machines install GitHub's official runner at boot, register one ephemeral runner after another with your labels, protect themselves from scale-in while a job runs, and leave cleanly when the group shrinks. Sizes follow a schedule (60 machines on weekday office hours, 15 in the evening, 5 at night, all yours to change), and an optional scaler adds machines for queued jobs.

No GitHub credential ever reaches a machine. The GitHub App key (or token) lives in one Deutsche Cloud function, the pool's broker. A machine proves who it is with a five-minute instance identity token and gets back a just-in-time configuration for exactly one runner: named after the machine, with the pool's labels and runner group fixed by the broker. The machines have no instance role at all, so a job running as root on one finds nothing that could read the credential.

Your workflows do not change. A job with runs-on: hetzner-16 (or [self-hosted, hetzner-16]) runs on any runner carrying that label, on Deutsche Cloud or on the machines you run today.

The recipe is a handful of files the console serves:

File What it is
setup.sh Creates the secret, broker, launch template, group, schedule and (optionally) the scaler with plain API calls.
runner-user-data.sh The launch template's user data: installs the runner and the services below on Ubuntu 24.04.
token-broker.mjs The broker, a Deutsche Cloud function (Node.js 22): instance identity token in, one runner configuration out.
github-auth.mjs GitHub App or token credentials, shared by the broker and the scaler.
scaler.mjs The queue-depth scaler, a Deutsche Cloud function (Node.js 22).
teardown.sh Rollback (desired capacity 0), or removal of everything setup.sh created.
prove.sh Builds a small pool on a throwaway repository, runs the tests below against it and removes it.
pool-burst.yml A test workflow: ten jobs at once on the pool's label. pool-long.yml: one long job.
mkdir runner-pool && cd runner-pool
for f in setup.sh runner-user-data.sh token-broker.mjs github-auth.mjs scaler.mjs teardown.sh prove.sh \
  pool-burst.yml pool-long.yml; do
  curl -fsSLO "https://console.platform-development.deutschecloud.app/recipes/github-runner-pool/$f"
done
chmod +x setup.sh teardown.sh prove.sh

They need bash, curl and jq, and an access key allowed to create what is listed under Reference → What setup creates (the AdministratorAccess policy is enough).

Concepts

Term Meaning
Registration credential A GitHub App installed on your organization or repository (recommended), or a token, allowed to manage self-hosted runners. Stored once, as a Deutsche Cloud secret bound to the broker function only.
Broker The function <POOL_NAME>-broker. Its URL is public, but it answers only an instance identity token signed by Deutsche Cloud for its URL, from a machine of your organization and of this pool's group, and hands that machine one runner.
Ephemeral runner A runner for exactly one job (a just-in-time configuration). GitHub removes it after the job; the machine then asks the broker for the next one and starts with an empty work directory. The machine stays, so its paid hour is used.
Busy A job is running. The runner's job-started hook protects the machine from scale-in; its job-completed hook lifts the protection.
Drain When the group picks a machine to leave, the machine stops registering, lets a running job finish, deregisters its runner and tells the group it may go (the group's termination lifecycle hook drain). A job that starts on the idle runner just as the group picks the machine keeps it: its protect call cancels that scale-in and another machine leaves.
Floor The group's minSize. Each scheduled step sets the floor and the capacity; the scaler moves the capacity between the floor and maxSize.
Paid hour Machines are billed per started hour. The group is created with deferScaleInUntilPaidHour: a machine picked for scale-in keeps taking jobs until the last 10 minutes of its hour, and the one whose hour ends first goes first.

How to

Create the GitHub credential

A GitHub App (recommended). Its credentials belong to no person, are scoped to where you install it, and every call uses an installation token that lasts an hour. In GitHub: your organization → Settings → Developer settings → GitHub Apps → New GitHub App. No webhook. Permissions:

Runners for RUNNER_URL Permission
every repository of an organization (default) https://github.com/ORG Organization → Self-hosted runners: Read and write
one repository https://github.com/ORG/REPO Repository → Administration: Read and write
the scaler (optional) the repositories it watches Repository → Actions: Read

Create it, note its App ID, generate a private key (a .pem file is downloaded), and Install App on the organization (or the one repository). setup.sh stores the key as a Deutsche Cloud secret and finds the installation itself.

Or a token. A fine-grained personal access token with the same permissions (GitHub → Settings → Developer settings). Simpler, but tied to the person who made it. Give the scaler a separate read-only token (SCALER_TOKEN_SECRET_ID, a secret you create under Secrets) when you can; otherwise it uses the registration credential.

Set up the pool

export EWS_API_URL=https://console.platform-development.deutschecloud.app
export EWS_ACCESS_KEY=EWS1…:…                       # an administrator's key, used only here
export RUNNER_URL=https://github.com/ORG
export RUNNER_LABELS=hetzner-16
export GITHUB_APP_ID=123456 GITHUB_APP_PRIVATE_KEY_FILE=./acme-runners.2026-11-14.private-key.pem
# or, with a token:  read -rs RUNNER_ADMIN_TOKEN && export RUNNER_ADMIN_TOKEN   (not echoed)
./setup.sh
secret        3f0c… (GITHUB_APP_PRIVATE_KEY)
broker        https://github-runners-broker-k3x9.fn.<root>
template      lt-1a2b3c4d (performance-16)
group         asg-5e6f7a8b (max 70, start 0, hook drain 3600s, paid-hour scale-in)
schedule      day: 60 at '30 6 * * 1-5' (UTC)
schedule      evening: 15 at '0 17 * * 1-5' (UTC)
schedule      night: 5 at '0 23 * * *' (UTC)
done: github-runners.state.json

The group starts at 0 machines and follows the schedule from its next step. Keep github-runners.state.json: teardown.sh reads it. Every setting is an environment variable; see Reference → Settings.

To start runners now instead of at the next step:

curl -X PUT "$EWS_API_URL/autoscaling/api/groups/asg-5e6f7a8b/desired-capacity" \
  -H "Authorization: EWS-ACCESS-KEY $EWS_ACCESS_KEY" -H 'Content-Type: application/json' \
  -d '{"desiredCapacity": 5}'

A machine runs its first job about 90 seconds after it is launched (Ubuntu 24.04, Docker and the runner installed at boot). It appears in GitHub under Settings → Actions → Runners as its instance id (i-…), with your labels. The broker's log (Functions → github-runners-broker → Logs) has one line per runner it handed out: which machine, which provider, which name.

Change the schedule

Each step is a scheduled action of the group (five-field cron, UTC) that sets the floor and the capacity. Set the sizes and times when you run setup.sh:

DAY_SIZE=60 DAY_CRON='30 6 * * 1-5' EVENING_SIZE=15 EVENING_CRON='0 17 * * 1-5' \
NIGHT_SIZE=5 NIGHT_CRON='0 23 * * *' ./setup.sh

or later on the group: Auto Scaling → the group → Scheduled actions, or

curl -X PUT "$EWS_API_URL/autoscaling/api/groups/asg-5e6f7a8b/scheduled-actions" \
  -H "Authorization: EWS-ACCESS-KEY $EWS_ACCESS_KEY" -H 'Content-Type: application/json' \
  -d '{"name": "day", "recurrence": "45 5 * * 1-5", "minSize": 60, "desiredCapacity": 60}'

A step with the name of an existing one replaces it. Cron runs in UTC: 07:30 in Germany is 30 6 in winter and 30 5 in summer. Start the morning step 15 minutes before the first jobs: sixty machines launch within a minute and run jobs about 90 seconds later.

Scale on queued jobs (optional)

With SCALER=1, setup.sh also creates a function and a schedule that calls it once a minute:

SCALER=1 SCALER_REPOSITORIES=ORG/web,ORG/api SCALER_TOKEN_SECRET_ID=7d1e… ./setup.sh

Each call counts the machines that are busy (protected) and the jobs GitHub has queued for the pool's labels in those repositories, and sets the capacity to busy + queued + SCALER_SPARE, never below the floor and never above maxSize. When the queue empties it sets the capacity back to the floor; the idle machines then finish their paid hour and drain. Its result is in the function's logs (Functions → github-runners-scaler → Logs) as one JSON line per call:

{"groupId":"asg-5e6f7a8b","minSize":15,"maxSize":70,"desired":15,"busy":15,"queued":7,"want":22,"apiCalls":9,"changed":true}

Each call makes 2 × repositories + queued and running workflow runs GitHub API requests (apiCalls in the log). A personal token allows 5,000 an hour, about 80 a minute.

Roll back

Scale the pool to zero and keep everything else:

./teardown.sh

This pauses the scaler, removes the scheduled steps and sets the floor and the capacity to 0. Running jobs finish (their machines are protected); idle machines drain and go. GitHub sends hetzner-16 jobs to the runners you still have elsewhere. To bring the pool back, run the PUT …/scheduled-actions calls above again, or delete it and run setup.sh again.

In a hurry, one call does the same for the group alone:

curl -X PATCH "$EWS_API_URL/autoscaling/api/groups/asg-5e6f7a8b" \
  -H "Authorization: EWS-ACCESS-KEY $EWS_ACCESS_KEY" -H 'Content-Type: application/json' \
  -d '{"minSize": 0, "desiredCapacity": 0}'

(The scheduled steps then raise it again at their next time.)

Remove the pool

./teardown.sh --delete

Rolls back, waits until every machine has drained, then deletes the group, the launch template, the broker, the scaler with its role and policy, and the secret if setup.sh created it.

Rotate the GitHub credential

Change the secret's value (Secrets → the secret → Rotate, or POST /secrets/api/secrets/{id}/rotate with { "value": "…" }): a new App private key (generate it in the App's settings, rotate, then delete the old key in GitHub) or a new token. The broker's instances pick it up when they next start (within its idle timeout, 5 minutes); machines are not touched.

Upgrading a pool made before 2026-11-14

Earlier versions of this recipe gave each machine an instance role that could read the credential. Run ./teardown.sh --delete with the old files, then setup.sh with the new ones: the new pool has no runner role.

Try it on a throwaway repository first

prove.sh builds a small pool (performance-2, at most 4 machines, label pool-proof, scaler on) for a repository you create for the purpose, and checks the three things that matter, then removes the pool:

# EWS_API_URL and EWS_ACCESS_KEY set as above
export GITHUB_REPO=you/runner-pool-test     # a new repository with one commit
read -rs GITHUB_TOKEN && export GITHUB_TOKEN # fine-grained, that repository only: Administration,
                                             # Actions, Contents and Workflows, read and write
./prove.sh
  1. It commits pool-burst.yml and pool-long.yml to .github/workflows/ and starts pool-burst: ten jobs at once. All ten must succeed, on more than one machine; the scaler grows the group for them.
  2. It pauses the scaler, starts pool-long (one 10-minute job) and sets the capacity to 0. The machine running the job must stay in service until the job has succeeded; the idle machines drain and go.
  3. The group must reach 0 machines. Then teardown.sh --delete removes the pool.

Each step prints the group's machines (* marks a protected one). It takes about 20 minutes and costs a few cents.

Reference

Settings

Environment variables of setup.sh:

Variable Default Meaning
RUNNER_URL required https://github.com/ORG or https://github.com/ORG/REPO
GITHUB_APP_ID — a GitHub App's id: the broker and scaler use the App
GITHUB_APP_PRIVATE_KEY_FILE with GITHUB_APP_ID the App's private key (.pem), stored as a new secret
GITHUB_APP_INSTALLATION_ID looked up the App's installation, if you want to name it
RUNNER_ADMIN_TOKEN without an App a GitHub token, stored as a new secret
TOKEN_SECRET_ID — an existing secret holding the App key or the token instead
RUNNER_LABELS hetzner-16 comma-separated; the runner adds self-hosted, Linux, X64 itself
RUNNER_GROUP Default an organization runner group
POOL_NAME github-runners names the group, template, roles, function; the state file is <POOL_NAME>.state.json
SERVER_TYPE performance-16 16 vCPU, 32 GB
IMAGE_ID / MACHINE_IMAGE_ID Ubuntu 24.04 a catalog image, or a machine image of yours (ewi-…) with warm caches
INSTALL_DOCKER 1 install Docker and let jobs use it
ZONES de1a,de1b Falkenstein and Nuremberg; machines are spread evenly
MAX_SIZE 70 at most 100
START_SIZE 0 capacity at creation
HOOK_TIMEOUT 3600 the longest a leaving machine waits for its running job, in seconds (30–7200)
DAY_*, EVENING_*, NIGHT_* 60 / 15 / 5, see above _SIZE and _CRON of each step; SCHEDULES=0 creates none
SCALER 0 1 creates the queue-depth scaler
SCALER_REPOSITORIES — ORG/REPO,… the scaler watches
SCALER_TOKEN_SECRET_ID the registration credential a secret with a read-only (Actions: Read) token
SCALER_SPARE 0 idle machines to keep above the demand
SSH_KEY_PAIR_ID none an SSH key pair, to log in to machines
TAGS_JSON {} extra tags on every machine, e.g. {"team":"ci"}; pool=<POOL_NAME> is always set
GITHUB_API https://api.github.com GitHub Enterprise Server: https://HOST/api/v3

What setup creates

Resource Permission it needs
Secret <POOL_NAME>_GITHUB_APP_PRIVATE_KEY (or …_GITHUB_TOKEN) secrets:CreateSecret (with a KMS key)
Function <POOL_NAME>-broker, public URL, the secret bound, no role functions:CreateFunction, functions:MakeFunctionPublic, secrets:GetSecretValue on the secret (to bind it), functions:UpdateFunctionConfiguration
Launch template <POOL_NAME>, no instance role autoscaling:CreateLaunchTemplate
Group <POOL_NAME> + lifecycle hook drain autoscaling:CreateAutoScalingGroup, ec2:RunInstances, autoscaling:PutLifecycleHook
Scheduled actions day, evening, night autoscaling:PutScheduledUpdateGroupAction
Scaler: policy + role <POOL_NAME>-scaler, function and schedule <POOL_NAME>-scaler the role may describe the group, set its capacity, invoke the function and read CI runs; creating them needs functions:CreateFunction, scheduler:CreateSchedule, iam:PassRole

On each machine

Path / unit What it is
/opt/ews-runner/runner the runner, owned by user runner
ews-runner.service the loop: ask the broker, run.sh --jitconfig …, wipe _work, again
ews-runner-drain.service polls ews-instance lifecycle every 10 s; on leaving: no new registration, wait for the job, stop, deregister, ews-instance complete
/opt/ews-runner/ews-runner-token root only: ews-instance token <broker URL> → POST to the broker (register or deregister)
/etc/ews-runner/pool.env the broker's URL; nothing secret
/opt/ews-runner/job-started.sh ACTIONS_RUNNER_HOOK_JOB_STARTED: ews-instance protect on
/opt/ews-runner/job-completed.sh ACTIONS_RUNNER_HOOK_JOB_COMPLETED: ews-instance protect off
/etc/sudoers.d/ews-runner lets user runner run exactly ews-instance protect on and … protect off as root
/var/log/ews-runner-bootstrap.log the bootstrap's result; journalctl -u ews-runner has every registration and job

The runner is pinned to 2.338.2 with its SHA-256 and runs with --disableupdate. GitHub stops sending jobs to runner versions it no longer supports; set RUNNER_VERSION and its checksums at the top of runner-user-data.sh to a current release and roll the group onto the new template version (Auto Scaling → the group → Instance refresh).

What a job can reach

Jobs run as user runner and cannot read the instance identity (/etc/ews/instance-identity, root only). With INSTALL_DOCKER=1 the runner user is in the docker group, which is root on that machine, so plan for a job that is root. What root on a pool machine can get:

  • No GitHub credential. The App key or token is a secret bound to the broker function only. The machine has no instance role and no access key.
  • An identity token (ews-instance token), worth five minutes, for one audience.
  • From the broker, nothing more than the machine already has. On GitHub a runner's name is unique: the broker names every runner after its machine, so a machine has at most one, and while that runner runs a job (the job asking is one) the broker refuses a second. The labels and the runner group are the broker's, not the machine's. It may deregister its own runner, which only ends its own job.
  • Its own scale-in protection and lifecycle (ews-instance protect, complete): it can keep itself or let itself go, nothing about other machines.

Each token a machine gets is in the audit log (ec2:GetInstanceIdentityToken, with the audience), and each runner the broker hands out is in the broker's log. Run untrusted pull requests from forks on GitHub's hosted runners all the same: a job that is root can still leave things behind for the next job on the same machine, which is why _work is wiped and Docker is shared (use INSTALL_DOCKER=0 or a fresh machine per job where that matters).

On Deutsche Cloud CI (PROVIDER=ews) the broker hands out a one-time registration token instead, with labels and repository pinned by the token, and refuses while the machine's runner is online. The token does not bind the runner's name, so a root job that first stops its own runner could still register one more runner with the pool's labels; GitHub's just-in-time configuration does not have that gap.

Cost

Prices at the time of writing, per machine, including its public IPv4 (see /pricing for today's):

Server type vCPU / RAM Per hour Monthly cap per machine
performance-16 16 / 32 GB €0.2091 €130.49
performance-8 8 / 16 GB €0.1122 —
dedicated-8 8 dedicated / 32 GB €0.2227 —

The default schedule with performance-16:

Step Machines Per hour Hours a week Per week
day, Mon–Fri 06:30–17:00 UTC 60 €12.55 52.5 €658.66
evening, Mon–Fri 17:00–23:00 15 €3.14 30 €94.10
night and weekend 5 €1.05 85.5 €89.39
Total 168 ≈ €842 a week, ≈ €3,660 a month

The scaler adds what queued jobs need above the schedule, at the same hourly price; its own calls are a function at well under €1 a month. Each scale-in step costs nothing extra: a machine leaves at the end of an hour already paid.

What is not supported

  • Windows and macOS runners; Linux x64 and arm64 (cax server types) only.
  • Scaling on GitHub's workflow_job webhook; the scaler polls once a minute.
  • Jobs waiting on needs: are counted by the scaler only once GitHub lists them as queued.

Troubleshooting

Symptom Cause Fix
Machines are in service but no runner shows in GitHub the App lacks the permission or is not installed on RUNNER_URL, or the token expired the broker's log names the GitHub error; journalctl -u ews-runner says the broker gave no runner config
broker: HTTP 403 not a machine of this pool the machine is not in the group the broker serves (launched by hand from the template) launch through the group
broker: HTTP 409 … is running a job the machine's runner is busy (a job asked for a second runner) expected; the loop asks again after the job
ews-runner-bootstrap: no ews-instance helper the machine was launched without an instance identity (a machine image taken before 2026-11-13) launch from the catalog image or a newer machine image
The group stays above its capacity machines are busy; scaleInBlockedByProtection in the group says how many nothing: they leave when their job ends
A machine stays in terminating_wait it is finishing a job that started just before it was picked it goes when the job ends, or after HOOK_TIMEOUT
A job's log says could not protect this machine from scale-in its machine was leaving for another reason than scale-in (terminated by hand, unhealthy) nothing: the drain waits for the job before the machine goes (up to HOOK_TIMEOUT)
Idle machines stay up to 50 minutes after the capacity drops paid-hour scale-in keeps them until the last 10 minutes of their hour expected; they keep taking jobs meanwhile
Jobs stay queued although machines are idle the job names a label the pool lacks compare the job's runs-on with RUNNER_LABELS
The scaler's log says GitHub …: HTTP 403 the scaler's token lacks Actions: Read on a repository, or hit the rate limit fix the token, or watch fewer repositories
Did this page help you?