Custom capacity scaling

Build a provider-specific scaler against the Capacity deployment protocol at /capacity/v1. Helmr supplies desired-capacity and Worker lifecycle status. Your scaler owns provider policy and mutations, including creating and terminating hosts.

This is a deployment infrastructure protocol for a trusted capacity owner, including an operator performing full-stop maintenance. It is not the tenant Developer REST API. Its credential grants every Capacity operation and Worker group in the deployment.

Configure authentication

Generate one canonical 32-byte base64url token:

CAPACITY_TOKEN="$(openssl rand -base64 32 | tr -d '\n=' | tr '/+' '_-')"

Store it in your secret manager. Provide the same value as CAPACITY_TOKEN to Control Plane and as a secret to the scaler. Do not place it in source, command-line arguments, logs, or ordinary configuration files. When CAPACITY_TOKEN is unset, Control Plane rejects every Capacity request.

Send the token in every request:

Authorization: Bearer <capacity-token>
Accept: application/json

Use HTTPS outside localhost and 127.0.0.1.

Operations

The Control Plane base URL is shown as $CONTROL_PLANE_URL below.

Worker group, pool, and instance IDs are canonical UUIDv7 values. region_id, names, and provider resource_id values are opaque canonical strings.

Resolve and plan

Resolve the group and pools, then observe provider and Worker inventory in the same reconciliation cycle. Record observed_at before starting those reads. The group response includes an opaque supply_fingerprint; return it as expected_supply_fingerprint in the plan request. Re-resolve after changing the primary pool, pool lifecycle or Worker inventory.

Ask Helmr how many additional Workers each provider pool can use:

printf 'Authorization: Bearer %s\n' "$CAPACITY_TOKEN" |
  curl --fail-with-body --get \
  --header @- \
  --header 'Accept: application/json' \
  --data-urlencode "region_id=$REGION_ID" \
  --data-urlencode "name=$GROUP_NAME" \
  "$CONTROL_PLANE_URL/capacity/v1/worker-groups/resolve"

printf 'Authorization: Bearer %s\n' "$CAPACITY_TOKEN" |
  curl --fail-with-body \
  --request POST \
  --header @- \
  --header 'Accept: application/json' \
  --header 'Content-Type: application/json' \
  --data @capacity-plan.json \
  "$CONTROL_PLANE_URL/capacity/v1/worker-groups/$GROUP_ID/plan"

The request file has this shape (use the current observation time and fingerprint):

{
  "expected_supply_fingerprint": "<fingerprint from group resolution>",
  "observed_at": "2026-10-22T00:00:00Z",
  "pools": [{
    "pool_id": "019...",
    "max_additional_workers": 10,
    "active_workers": 1,
    "registering_workers": 0
  }]
}

Include every active pool, including pools with zero provider headroom. max_additional_workers is the number of additional VM-capable hosts the provider can supply beyond the active Workers already represented in this observation. Account for desired, in-flight, unregistered and unresolved provider instances; do not derive it by subtracting registered Workers from the provider maximum. Report the active and registering counts from the same Worker inventory.

An observation must be no more than 30 seconds old and must still match the current supply fingerprint. Old, future-dated or mismatched observations return execution_capacity_unavailable; reobserve instead of retrying the stale body. Requests cover at most 64 pools and at most 1,000 additional Workers per pool. A partial pool set or mismatched inventory counts can still produce an advisory plan, but cannot refresh capacity evidence for new execution admission.

Helmr reserves bounded execution claims against current capacity and this fresh provider evidence. When evidence expires or topology changes, new work can use verified existing capacity but cannot rely on the old provider headroom. Keep observations current even when the provider is at its maximum. A rejected request has no accepted Turn waiting for capacity; the caller must submit new work after resolving the reported condition.

The plan response has this shape:

{
  "worker_group_id": "019...",
  "worker_group_name": "default",
  "region_id": "aws-us-east-1",
  "group_status": "active",
  "pools": [
    {
      "pool_id": "019...",
      "pool_name": "default",
      "recommended_additional_workers": 2,
      "compatible_queued_items": 2,
      "registering_workers": 0,
      "active_workers": 1,
      "complete": true,
      "saturated": false,
      "scale_in_blocked": false
    }
  ],
  "unmatched_demand": [],
  "complete": true,
  "computed_at": "2026-08-29T00:00:00Z"
}

Treat the recommendation as input to your provider policy, not as an imperative desired count. The scaler decides mutation rate, limits, and provider calls.

Scale out can use positive recommended_additional_workers subject to provider limits. Scale in needs a complete, internally consistent observation. Do not start a drain unless all of these are true:

• the group plan has complete: true and unmatched_demand is empty;
• the selected pool has complete: true, saturated: false, scale_in_blocked: false, and compatible_queued_items: 0;
• the plan’s active/registering counts match Worker inventory observed in the same reconciliation cycle;
• inventory did not reach the requested limit, no Worker is registering or already draining, and no provider mutation is in flight; and
• provider inventory is complete and stable under its scale-in protection.

The drain fence and the advisory queued-demand check for idle_scale_in are final safety checks. They do not make a partial plan or incomplete inventory safe for scale-in.

Bound optional work per Worker

Read or replace a pool’s policy at GET /capacity/v1/worker-groups/{group}/pools/{pool}/resource-policy and PUT /capacity/v1/worker-groups/{group}/pools/{pool}/resource-policy. The replacement requires every field and the revision returned by the read:

{
  "expected_revision": 1,
  "optional": {
    "cpu_millis": 0,
    "memory_bytes": 0,
    "disk_bytes": 0,
    "vm_slots": 0,
    "max_tasks": 0,
    "io_bytes_per_second": 0
  },
  "local_headroom_slots": 0
}

All limits default to zero. Optional seed generation, prepared Computers and cache share these per-Worker ceilings. A paused source retained after a durable Computer checkpoint uses these optional ceilings too; it reserves its full CPU, memory and lifecycle disk until physical teardown completes. Retention is an optimization, not a promise that the source will remain available. local_headroom_slots keeps full VM shapes available to real work by restricting optional admission. It does not request additional hosts or guarantee fleet-wide ready capacity. Set a provider minimum size when physical standing capacity is required. Replacing policy requires the current revision; a stale revision returns HTTP 409. A successful update records desired policy, not proof that every Worker has finished reclaiming resources.

Workers charge seed generation, prepared Computers and cache to the same physical reservation ledger as real allocations. Cache limits include filesystem overhead. Policy reduction cancels optional fills and stops seed attempts through Control Plane; open readers, VM processes and files remain charged until their owners release them. reclaiming_workers can therefore remain nonzero after a policy update. An optional transfer rate of zero disables optional uploads and downloads. The rate is shared across concurrent requests, multipart transfers and retries.

A prepared Computer is a private, paused VM for an exact qualified image and runtime shape. Preparation does not run authored initialization or receive Session credentials or a Computer write key. Its source access expires and must be renewed; policy changes, stale observations and source invalidation remove unused preparation. Every leaf reserves a full lifecycle slot, including local RAM ciphertext retained through restoration and subsequent checkpoint capture.

A fresh Computer allocation can atomically take ownership of a matching prepared VM and its reservation. Fresh execution authority and bootstrap identity are installed before execution. Personal checkpoint restoration keeps its exact saved state and does not use a prepared image as a substitute. A miss follows the ordinary startup path. Pending real work suppresses refill and reclaims conflicting optional resources; failed cleanup retains the charge and prevents reuse until physical release is verified. Prepared counts are observations, not execution authority or a guarantee of startup latency.

A Worker crash can leave an unclaimed prepared VM’s attachment arena behind. Startup quarantines retained attachment state and requires operator reconciliation, including when no customer Computer was running. Before relocating the exact arena, stop the Worker and verify that its processes and VM cgroups are gone and its backing devices are exclusively available and idle. Preserve the arena for inspection, then restart the Worker. Do not delete attachment markers or treat a stale host observation as proof of physical release. Enabling prepared VMs therefore requires an operator recovery procedure; automatic recovery of these arenas is not provided.

Checkpoint capture uses a separate host queue before sealing dispatch. Waiting for that queue leaves normal save processing available. Publication yields the capture permit so other Computers can capture while durable upload proceeds; retries preserve the same cut. A zero optional policy does not disable required checkpoint publication.

All optional limits default to zero, which disables new artifact-cache fills as well as prepared VMs. Repeated starts can therefore fetch artifacts again. To fund only the local artifact cache, set nonzero disk_bytes, max_tasks and io_bytes_per_second in the optional policy while leaving optional CPU, memory and VM slots at zero. The network rate covers optional fetches and uploads; copying an already verified local file uses the cache’s disk and task allowance. Cache fills occur during image preparation and seed construction; ordinary Computer starts do not populate the cache. A seed construction task and its cache fill each consume a task allowance, so caching alongside seed construction needs max_tasks of at least 2. Setting task or transfer limits to zero, or retiring an in-flight seed after a policy change, prevents new cache work without itself evicting resident data that still fits the disk budget and headroom. Real-work admission and disk-limit reductions can still reclaim that data.

Each pool plan reports policy_revision, placements.immediate, placements.after_optional_reclaim, optional_occupancy, reclaiming_workers, local_headroom_slots_in_use and credited_hosts (host IDs and Worker epochs). Placements count complete compatible VM shapes before queued work is packed. Compatible pools can credit the same Worker, so their counts must not be added. Missing or stale resource observations make immediate placement and optional occupancy null and the plan incomplete. Post-reclaim counts remain advisory; physical cleanup must finish before a granted allocation starts. Optional backlog and occupancy do not create additional-Worker recommendations.

Retry scale-in safely

Submit POST /capacity/v1/worker-groups/{group}/pools/{pool}/scale-in with {"operation_key":"<UUIDv7>"}. Helmr selects an eligible host and commits its drain together with a durable receipt. A retry with the same key returns that selection; reusing it for another pool is a conflict. Only one unsettled automatic scale-in operation is allowed per group. Hosts occupied only by optional work may be selected, but cleanup must still finish before provider termination.

Read GET /capacity/v1/worker-groups/{group}/scale-in-operations/pending before starting a new operation, including after controller restart. It returns {"operation":null} or the outstanding receipt. Read a particular key at GET /capacity/v1/worker-groups/{group}/scale-in-operations/{operation_key}. Receipts retain the selected host, provider resource and drain claim. Their phase is draining, termination_ready, blocked or completed. Terminate only the exact resource with a current termination_ready receipt and matching inventory; a blocked receipt requires reconciliation. Completion follows confirmed physical provider absence. Response loss is not permission to select another host.

React to capacity changes

Connect to GET /capacity/v1/updates with the capacity token and Accept: text/event-stream. The stream emits event: change on connection and on committed capacity-relevant changes, with heartbeat comments during idle periods. Treat each event as an invalidation hint and re-read durable state. Events are coalesced, have no replay cursor, and do not contain a capacity plan. Reconnect after disconnects and reconcile periodically to cover lost hints and provider-only changes. Save and output progress alone do not trigger capacity notifications.

Select a primary pool

Use the claim_version returned by Worker-group resolution:

{
  "expected_group_claim_version": 4,
  "minimum_ready_hosts": 1,
  "pool_id": "019..."
}

PUT .../primary-pool returns worker_group and applied. Re-read the group and retry from current state when the claim fence is stale. Selection checks fresh, active, unpaused hosts at the current epoch under the same Group, Pool and Host fences used by drain. Selecting the current primary is an exact replay; it does not establish current fleet health.

Group and Pool resolution includes retained_profiles: up to 100 distinct profiles with live Instance, creating checkpoint, parked checkpoint and eligible Pool counts. Eligible Pool counts include the inspected Pool when it is active; they are not a count of alternatives after its retirement. complete: false means more profiles exist; a truncated result is not proof that retirement is safe. Valid parked checkpoints retain their profile when waits resolve or Commands queue. A Pool also reports its claim version and seal time. Retirement checks the complete database state atomically.

GET /capacity/v1/deployment reports this Control Plane replica’s release, source commit, worker protocol revision, applied database schema version and migration dirty flag. It reads database state directly and does not report provider fleet convergence.

Reconcile Worker inventory

GET /worker-hosts accepts these optional query parameters:

The response contains worker_hosts. Each item includes id, resource_id, worker_group_id, worker_pool_id, status, claim_version, created_at, and updated_at. drain_blockers counts unreclaimed Instances and unreconciled Session processes. Zero counts still require the worker’s local cleanup acknowledgment. current_epoch, draining_at, drain_reason, termination_ready_at, and lost_at appear when applicable.

Group statuses are active, paused, draining, or disabled. Pool statuses are pending, active, draining, or disabled.

Drain before termination

Never terminate a provider host until its Worker instance reaches termination_ready. Start drain with the current epoch and claim version:

printf 'Authorization: Bearer %s\n' "$CAPACITY_TOKEN" |
  curl --fail-with-body \
  --request POST \
  --header @- \
  --header 'Accept: application/json' \
  --header 'Content-Type: application/json' \
  --data "{\"expected_epoch\":$CURRENT_EPOCH,\"expected_claim_version\":$CLAIM_VERSION,\"reason\":\"idle_scale_in\"}" \
  "$CONTROL_PLANE_URL/capacity/v1/worker-hosts/$INSTANCE_ID/drain"

The epoch and claim version fence the exact Worker ownership observed by the scaler. On conflict, discard the attempted mutation, read the instance again, and reconcile from current state. For reason idle_scale_in, Helmr returns 409 with code queued_demand_present instead of beginning scale-in while that group has queued work. This demand snapshot is advisory; new work may enqueue afterward. Use replacement or capacity_reduction for planned maintenance that must preserve queued work while draining. The first committed drain reason and epoch remain unchanged on replay.

After the provider confirms that a host is absent, call POST .../{id}/lost. This is a confirmation of observed provider state, not a request for Helmr to delete the host.

Errors and retries

Non-2xx responses use the standard error envelope:

{
  "error": {
    "code": "queued_demand_present",
    "message": "queued demand is present"
  }
}

details is optional and appears only when an error supplies structured details.

Retry reads and reconciliation loops from freshly observed state. Do not retry a provider mutation blindly. 404 with not_found means the referenced object is absent. A stale drain or claim fence returns 409; re-read before deciding whether any provider action is still needed. Mutating request bodies are limited to 16 KiB.

Compatibility

/capacity/v1 is the integration boundary. Deploy matching Control Plane and capacity-controller versions during pre-release. Clients must ignore unknown response fields; the request fields documented above are required for current capacity observations.

The historical Go module release capacity/v0.1.0 remains downloadable for reproducible builds, but it is frozen and receives no successor releases. New scalers should implement this HTTP protocol directly.

Rotate the token

Control Plane accepts one token and reads it at startup, so rotation has no overlap window:

1. Pause the scaler.
2. Generate and store a new token.
3. Update CAPACITY_TOKEN, replace every Control Plane replica, and wait until all old replicas have stopped and all replacements are ready.
4. Update the scaler secret and restart the scaler.
5. Confirm the scaler can resolve a Worker group and read a plan, then resume.

The old token stops working only after the last old Control Plane replica has stopped.

The capacity authority can create a pending pool with POST /capacity/v1/worker-groups/{workerGroupID}/pools using name and expected_group_claim_version, and disable an eligible pool with POST /capacity/v1/worker-groups/{workerGroupID}/pools/{workerPoolID}/disable using expected_pool_claim_version. These use the same group/pool locks, claim checks, primary protection and custody checks as administration. Both return the capacity WorkerPool projection. A disabled name cannot enroll again; use a new generation name. Controllers must reobserve after a conflict or lost response and must not infer a terminal fence from provider absence alone.