1. Introduction to srv

srv is a self-hosted control-plane service for creating Firecracker microVMs over SSH on a Tailscale tailnet.

It exposes an SSH command surface on one Linux host and manages Firecracker microVMs behind it. You create, inspect, stop, start, resize, back up, and delete VMs the same way you would run a remote command — through ssh srv <command>.

ssh srv new demo# demo created — state: provisioning# inspect:  ssh srv inspect demo# connect:  ssh root@demo

1.1 Where srv fits

srv is useful for both short-lived sandboxes and persistent isolated services.

• Throwaway debug VMs — spin up an isolated environment, break things, and delete it without affecting the host
• Remote Amp runners and sandboxed agent VMs — run normal web- and mobile-controlled Amp threads in dedicated cgroup-limited microVMs with private Tailscale TCP/UDP access and scoped provider API proxies
• Dev/test environments — fast reflink-based clones from a single base image, with backup/restore for instant reset
• Isolated workloads — run services in separate microVMs with per-VM networking, auth, and resource limits

All VMs get a full Linux system with systemd, a real kernel, and their own Tailscale identity. Where containers are not enough — package managers, services that need root, bespoke networking configurations, predictable boot and teardown — srv gives you real isolated machines in seconds.

1.2 Conceptual architecture

                    Tailscale tailnet                           │                     ┌─────┴─────┐                     │   tsnet   │  (joins tailnet as "srv")                     │  :22/tcp  │  (SSH API surface)                     └─────┬─────┘                           │             ┌─────────────┼─────────────┐             │             │             │      ┌──────┴──────┐ ┌──────┴──────┐ ┌──────┴──────┐      │ VM: demo    │ │ VM: ci      │ │ VM: test    │      │ /30 net     │ │ /30 net     │ │ /30 net     │      │ cgroup      │ │ cgroup      │ │ cgroup      │      │ TAP + NAT   │ │ TAP + NAT   │ │ TAP + NAT   │      └─────────────┘ └─────────────┘ └─────────────┘

Key components:

• tsnet — joins the tailnet as srv and exposes the control API on tailnet TCP port 22
• gliderlabs/ssh — handles exec requests and rejects shell sessions
• SQLite — stores instances, events, command audits, and authorization decisions
• Reflinks — clone the base rootfs for fast per-instance writable disks
• Network helper — root-only process owns TAP creation, iptables MASQUERADE, and FORWARD rules
• VM runner — root-owned process invokes Firecracker through the official jailer, drops to srv-vm:srv, and places each VM into its own cgroup v2 leaf
• MMDS — one-off Tailscale auth keys are injected through Firecracker metadata so guests self-bootstrap
• Provider gateways — per-instance HTTP proxies on the guest's gateway IP forward to LLM providers with host keys
• HTTP integrations — admin-defined host-side HTTP proxies inject headers or auth for selected guests without storing raw secrets inside the VM

1.3 Where you can run it

srv runs on Linux with the following requirements:

• Linux host with cgroup v2 and /dev/kvm
• IPv4 forwarding enabled (net.ipv4.ip_forward=1)
• SRV_DATA_DIR on a reflink-capable filesystem (btrfs or reflink-enabled xfs)
• Tailscale installed and working on the host
• Tailscale OAuth client credentials with permission to mint auth keys for guest tags
• ip, iptables, cp, e2fsck, and resize2fs available on the host
• Official static Firecracker and jailer release pair

1.4 Next steps

• Install srv
• Multi-host frontend
• Walkthrough
• Self-hosted Amp orbs
• Amp runners on srv
• Sandboxed AI coding agent
• Architecture
• SSH command reference
• RFC: Hybrid memory pools
• Testing
• HTTP integrations

2. Amp runners on srv

Run Amp threads in dedicated, self-hosted Firecracker microVMs without giving up Amp's remote experience.

This page describes the CLI runner plugin, not the native Orb integration. For Amp-managed pause/wake on srv, see Self-hosted Amp orbs. Comparisons below refer to Amp-hosted orbs; self-hosted providers can have different lifecycle capabilities.

Each New isolated thread action creates a VM with a fresh repository clone, its own resource limits and Tailscale identity, and a dedicated Amp runner. The resulting thread is still a normal Amp thread: open it from the web or your phone, review its conversation and changes, and use its remote terminal while srv provides the machine underneath.

2.1 One action creates the whole environment

Run srv Runner: New isolated thread from Amp's command palette. The plugin turns a task into a ready remote thread:

Task + mode + size        │        ▼┌─────────────────────┐│ Dedicated srv VM    ││ Firecracker + cgroup│└──────────┬──────────┘           │ clone + .agents/setup           │ workstation profile           ▼┌─────────────────────┐│ Dedicated Amp runner││ persistent systemd  │└──────────┬──────────┘           │           ▼┌─────────────────────┐│ Normal Amp thread   ││ web · mobile · diff ││ remote terminal     │└─────────────────────┘

The clone comes from the current branch on origin. Local uncommitted and untracked files stay on the workstation.

2.2 Keep the Amp thread experience

srv replaces the executor, not Amp's thread and control plane. The plugin starts amp --no-tui with a stable runner ID and --remote-control-terminal, so a thread running in the VM retains:

• Remote control from ampcode.com on desktop or mobile
• The normal Amp thread URL, conversation, visibility, and changes UI
• A web-accessible terminal while the runner is online
• Amp project and thread workflows that apply to CLI runner threads
• The ability for Amp plugins and agents to target the live runner for more threads that share that VM

The VM must be running for work to execute. srv has no Orb-style automatic wake integration: any schedule or external or plugin automation that targets this runner requires the VM to already be running. Amp's Multiplayer feature is currently restricted to Orb-backed threads. See Amp's Owner's Manual for the current runner, thread-sharing, and remote-control behavior.

2.3 Tailscale services instead of HTTP-only portals

Every VM joins the tailnet with its own identity and MagicDNS name. A development service can be reached directly by authorized tailnet devices without port forwarding:

http://:3000postgresql://:5432/appssh root@

This is broader than an HTTP preview tunnel: Tailscale carries TCP and UDP, supports multiple simultaneous services, works with native clients, and applies tailnet ACLs to the whole VM identity. The service must listen on an address reachable through Tailscale, and the VM must be running.

Amp Portals provide conveniences that raw Tailscale connectivity does not: Amp-managed HTTPS, embedding and annotation in the thread, access derived from thread visibility, and automatic coupling to Orb sleep and wake. srv instead provides private, general-purpose network access to the complete VM. You can put a Tailscale service address in the thread, but it remains accessible only to authorized tailnet devices.

2.4 Bring your environment with you

There are two setup layers:

• ~/.config/amp/srv-runner.json defines workstation-wide skills, binaries, private files, guest symlinks, and NO_PROXY entries to copy into every runner VM.
• An executable .agents/setup in the repository installs project-specific dependencies after the clone.

Configured files are validated before VM creation. The confirmation shows how many profile items will enter the persistent VM.

2.5 Host-controlled credentials and egress

The Amp credential is stored root-only in the VM. Optional provider and HTTP integration gateways keep upstream API credentials on the srv host and expose only per-VM proxy endpoints. General public HTTPS traffic from the runner uses a per-VM CONNECT gateway that rejects private, tailnet, host-local, and special-use destinations.

Treat the VM as sensitive persistent state: its Amp credential and configured private files remain until deletion.

2.6 Persistent when useful, disposable when done

Unlike an Orb, an srv VM is persistent and consumes capacity on hardware you operate.

• Stop it to release runtime resources while preserving the checkout and disk.
• Start it again and the systemd-managed Amp runner reconnects.
• Delete it to remove the VM, checkout, copied credentials, and plugin state.
• Inspect or SSH into failed provisioning attempts instead of losing the environment.

2.7 srv runner and Amp Orb at a glance

Choose srv when you want Amp's remote thread experience on hardware, networks, credentials, and VM boundaries you control. Choose Orbs when managed elasticity, automatic wake-up, Portals, or Multiplayer matter more than operating the executor yourself.

2.8 Get started

After meeting the runner prerequisites, install the bundled plugin from the srv checkout:

mkdir -p ~/.config/amp/pluginsln -sf "$PWD/contrib/amp/plugins/srv-runner.ts" ~/.config/amp/plugins/srv-runner.ts

Reload plugins from Amp's command palette, then run srv Runner: New isolated thread.

Continue with the AI coding-agent guide for detailed provisioning behavior, profile configuration, secure branch pushing, resource limits, and cleanup.

3. Installation

3.1 System requirements

• Linux host with cgroup v2 and /dev/kvm
• IPv4 forwarding enabled: net.ipv4.ip_forward=1
• SRV_DATA_DIR on a reflink-capable filesystem (btrfs or reflink-enabled xfs), with SRV_BASE_ROOTFS on the same filesystem
• Tailscale installed and working on the host
• Tailscale OAuth client credentials (TS_CLIENT_ID and TS_CLIENT_SECRET) with permission to mint auth keys for the configured guest tags
• ip, iptables, cp, e2fsck, and resize2fs available on the host
• Official static Firecracker and jailer release pair (the installer can download these for you)

3.2 Build

Build the three binaries from source:

go build ./cmd/srvgo build ./cmd/srv-net-helpergo build ./cmd/srv-vm-runner

Or install them to a standard location with the provided installer (see below).

3.3 Build the guest image

The Arch base image builder produces the vmlinux kernel and rootfs-base.img rootfs that srv provisions from.

On an Arch Linux host:

sudo OUTPUT_DIR=/var/lib/srv/images/arch-base ./images/arch-base/build.sh

On a non-Arch host, use the podman workflow:

sudo podman run --rm --privileged --network host \  -v "$PWD":/work \  -v /var/lib/srv/images/arch-base:/var/lib/srv/images/arch-base \  -w /work \  docker.io/library/archlinux:latest \  bash -lc '    set -euo pipefail    pacman -Sy --noconfirm archlinux-keyring    pacman -Syu --noconfirm arch-install-scripts base-devel bc e2fsprogs rsync curl systemd    OUTPUT_DIR=/var/lib/srv/images/arch-base ./images/arch-base/build.sh  '

See Building a custom guest image and the guest image reference for details on what the image includes and how to customize it.

3.4 Install and configure

The systemd installer handles binary installation, unit setup, and downloading the official static Firecracker/jailer pair:

sudo ./contrib/systemd/install.sh

Enable IPv4 forwarding:

sudo tee /etc/sysctl.d/90-srv-ip-forward.conf >/dev/null <<'EOF'net.ipv4.ip_forward = 1EOFsudo sysctl --system

Write the environment file. At minimum you need Tailscale credentials and guest artifact paths:

sudoedit /etc/srv/srv.env

Key entries to fill in:

TS_AUTHKEY=tskey-auth-xxxxxxxxxxxxxxxxTS_CLIENT_ID=your-oauth-client-idTS_CLIENT_SECRET=your-oauth-client-secretTS_TAILNET=your-tailnet.ts.netSRV_BASE_KERNEL=/var/lib/srv/images/arch-base/vmlinuxSRV_BASE_ROOTFS=/var/lib/srv/images/arch-base/rootfs-base.imgSRV_GUEST_AUTH_TAGS=tag:microvm

See the full configuration reference for every variable.

Then start the services:

sudo ./contrib/systemd/install.sh --enable-now

3.5 Validate

Run the end-to-end smoke test to confirm everything works:

sudo ./contrib/smoke/host-smoke.sh

The smoke test verifies systemd units, SSH reachability, guest creation, readiness polling, backup/restore, cgroup limits, and cleanup. It is the supported validation gate after install, restore, and upgrade.

See the smoke test reference for details on overrides and failure artifacts.

3.6 Next steps

• Walkthrough — create your first VM
• Running as a daemon — systemd details

4. Multi-host frontend

srv-front provides one SSH endpoint in front of independent srv hosts. It keeps only a static host configuration and its own Tailscale/SSH identity, not a VM registry. It can run on a small separate Linux server without KVM, Firecracker, a reflink filesystem, or the srv helper services.

Each backend still owns VM state, permissions, admission, images, pools, and backups. An unavailable frontend does not stop VMs; direct backend SSH remains available. There is no HA failover, replication, or automatic rebalancing.

4.1 Configure trust and start the frontend

Upgrade the backends to a version supporting forwarding, placement, and migration before adding them. Give each backend a distinct SRV_HOSTNAME (for example srv-a, srv-b); the frontend can take the friendly name srv.

Build and install the separate binary:

go build -o srv-front ./cmd/srv-frontsudo install -m 0755 srv-front /usr/local/bin/srv-frontsudo install -m 0644 contrib/systemd/srv-front.service /etc/systemd/system/

Create /etc/srv-front.json with explicit allowed user logins and pinned backend host keys. Addresses must include the SSH port:

{  "hostname": "srv",  "data_dir": "/var/lib/srv-front",  "listen_addr": ":22",  "allowed_users": ["alice@example.com", "ops@example.com"],  "hosts": [    {"name": "a", "address": "srv-a:22", "host_key": "ssh-ed25519 REPLACE_WITH_HOST_A_PUBLIC_KEY"},    {"name": "b", "address": "srv-b:22", "host_key": "ssh-ed25519 REPLACE_WITH_HOST_B_PUBLIC_KEY"}  ]}

Obtain each public host key through a trusted administrative connection, not an unauthenticated first-use network scan. Each upgraded backend logs its public key at startup. Read the public_key value from its service journal:

sudo journalctl -u srv -b --no-pager | grep 'ssh host key'

Copy the ssh-ed25519 ... value without the surrounding log-field quotes. Only the public key, never /var/lib/srv/state/host_key, goes into the frontend configuration. Changing a backend key requires updating its pin.

Put the frontend's node-join credentials in /etc/srv-front.env (root-readable, mode 0600). TS_AUTHKEY, TS_CLIENT_ID, TS_CLIENT_SECRET, TS_CONTROL_URL, and SRV_ADVERTISE_TAGS are supported. Use a dedicated frontend node identity; it does not need credentials for creating guest nodes. Restrict tailnet access so the frontend can reach the configured backend SSH ports and intended users can reach the frontend.

sudo systemctl daemon-reloadsudo systemctl enable --now srv-frontsudo journalctl -u srv-front

The frontend logs its stable Tailscale node ID at startup. Configure that exact ID on each backend in /etc/srv/srv.env:

SRV_TRUSTED_FRONTENDS=nREPLACE_WITH_FRONTEND_STABLE_NODE_ID

Restart each backend after updating its configuration. This trust is powerful: the frontend can assert user identities, including administrators. Protect its state directory and credentials accordingly. Re-enrolling it as a new node requires updating the backend trust entries.

The frontend resolves the original user with Tailscale WhoIs; tagged client nodes cannot impersonate human users. Backends accept forwarding only from the configured stable node IDs, then apply their usual SRV_ALLOWED_USERS, SRV_ADMIN_USERS, and per-instance ownership checks. Audit records retain the user identity, actual frontend peer address, and forwarding node ID. No SSH username grants delegation rights. A trusted frontend itself may be tagged.

4.2 Commands and placement

ssh srv node listssh srv -- --json node listssh srv listssh srv statusssh srv -- --json listssh srv new demo --ram 8G --rootfs-size 20Gssh srv new other --host bssh srv inspect demossh srv stop a/demossh srv logs -f a/demo serialssh srv pool list --host a

node list shows one row per configured server: running/total VMs, allocated vCPUs versus CPU budget, one-minute load average, RAM and disk reservations versus their budgets, and actual available filesystem space. These are allocations and capacity, not instantaneous CPU/RAM utilization percentages. Unreachable or unauthorized hosts remain visible with an error rather than disappearing. Use node list --host a to query just one host; status gives more detail.

node list, list, and status query hosts concurrently and label their results. node list and status require backend admin access. With --json, aggregate responses contain a hosts array with host, data, and optional error. Partial failures retain successful results, print errors, and exit nonzero. Host-local operations such as pools, integrations, snapshots, and raw imports require --host.

Short VM names resolve only when all hosts answer and exactly one active source matches. Use host/name or --host when ambiguous or when another host is down. Completed migration sources remain visible but are excluded from short-name resolution. Keep guest names unique within the tailnet; the frontend does not reserve names globally or rewrite guest DNS names. Interactive top/PTY sessions must go directly to a backend.

Automatic new placement asks each backend whether the requested image, shape, pool, and integrations fit. It ranks eligible hosts by remaining RAM budget, then remaining disk space, then host name. RAM includes VM overhead; disk checks both logical reservations and available filesystem space with the host reserve. CPU remains advisory, consistent with srv's existing overcommit policy.

Preflight does not reserve capacity. The selected host still performs normal admission. A failed create is not retried on another host: after a network failure the original create may already have succeeded. Inspect that host before retrying. --host bypasses automatic selection, not backend permissions or admission checks.

4.3 Cold migration

ssh srv migrate a/demo --to b

The frontend checks the destination before stopping the source, records a durable source start fence, streams export into import through bounded pipes, confirms the transfer, marks the source completed, and starts the destination only if the source was previously running. An already stopped VM stays stopped. migrate currently reports text and progress, and rejects --json before doing any work. The frontend needs no disk-sized buffer, but all transfer traffic crosses its network connection. This is cold migration: downtime includes transfer and startup.

For pooled VMs, explicitly select an existing destination pool or convert to fixed memory. A fixed-memory VM can also be assigned to a destination pool:

ssh srv migrate a/demo --to b --pool sharedssh srv migrate a/demo --to b --fixed-memory

Pools are host-local memory reservations and shared limits, not shared disks. Pool IDs are never copied and pools are never created automatically. The target pool must support this VM's guest-visible RAM; it is charged once at host level, not again for every member. This does not guarantee spare memory under concurrent pool usage. The source pool stays allocated until an operator removes it. Without an explicit policy, pooled migration is rejected before stopping the VM. Raw imports of pooled exports likewise require import --pool NAME or import --fixed-memory. Upgrade both backends: pooled exports use a format that older importers reject instead of silently changing the memory policy.

VMs with enabled integrations remain blocked: credentials, gateway routes, and bindings are host-local and are not part of the artifact. An administrator must configure the intended integration on the destination, disable every source binding, migrate, and explicitly enable the destination bindings:

ssh srv integration disable demo openai --host assh srv migrate a/demo --to bssh srv integration enable demo openai --host b

Expect integration-dependent workloads to be unavailable during this sequence. The destination must have a compatible kernel/initrd under the same image name; its rootfs template is not used for migration. Matching names do not prove kernel compatibility; operators must deploy compatible image versions. Per-VM backups are not copied.

The source is retained and fenced, not automatically deleted. After verifying the destination, explicitly delete the completed source:

ssh srv delete a/demo

Deleting a completed source removes its local copy without revoking the shared guest Tailscale identity. Never start both copies: the rootfs contains the same durable guest identity.

4.3.1 Interrupted migrations

There is no automatic rollback or automatic resumption. Keep the reported source, target, and migration token. Inspect both hosts directly:

ssh srv-a -- --json migration status demossh srv-b -- --json migration receipt demossh srv-b inspect demo

An unknown target outcome must leave the source fenced. A successful import is recorded durably on the destination. Only after verifying the destination receipt belongs to the intended transfer may an operator complete the source and start the destination. A completed source cannot be aborted.

Recovery mutations go directly to the backends; srv-front intentionally has no automatic recovery loop. migration status records the selected memory policy. When manually resuming a transfer, pass that same policy to import. After abort, any fresh migration needs a new token.

If abandoning an incomplete migration, an administrator must first establish that all destination copies of the guest identity are destroyed, including saved transfer artifacts that might be booted. Then the explicit recovery command on the source removes the fence without starting the VM:

ssh srv-a migration abort demo TOKEN --confirm-destination-destroyed

This confirmation is an operator assertion, not a distributed proof. Do not use it merely because the destination is unreachable. Preserve the source until the outcome is understood; do not manually edit migration database rows.

5. Walkthrough

This walkthrough assumes you have completed the installation and the smoke test passes.

5.1 Create a VM

ssh srv new demo

demo created — state: provisioninginspect:  ssh srv inspect democonnect:  ssh root@demo

The control plane creates a reflink clone of the base rootfs, allocates a /30 network, mints a one-off Tailscale auth key, injects it through MMDS, and boots the VM through the jailer.

With custom sizing:

ssh srv new demo --cpus 4 --ram 8G --rootfs-size 20G

5.2 Check status

ssh srv list

NAME     STATE   CPUS  MEMORY   DISK     TAILSCALEdemo     ready   1     1.0 GiB  10.0 GiB 100.64.0.2

For detailed info:

ssh srv inspect demo

Machine-readable output for scripting:

ssh srv -- --json listssh srv -- --json inspect demo

5.3 Connect to the VM

Once the VM reports state: ready and shows a Tailscale IP, connect over the tailnet:

ssh root@demo

Because the guest image bootstraps tailscale up --ssh, Tailscale SSH handles authentication without needing per-user OpenSSH keys in the guest.

You can also get the IP from inspect:

ssh srv inspect demo# look for tailscale-ip and tailscale-name fields

5.4 View logs

ssh srv logs demossh srv logs demo serialssh srv logs demo firecrackerssh srv logs -f demo serial

The serial log shows guest boot, bootstrap, and tailscaled output. The Firecracker log shows VMM lifecycle events.

5.5 Stop, restart, and delete

ssh srv stop demossh srv start demossh srv restart demossh srv delete demo

Stopping a VM does a graceful shutdown. The guest rootfs persists across stop/start cycles.

5.6 Resize a VM

Resize requires the VM to be stopped. CPU and memory can be increased or decreased within limits, while rootfs can only grow:

ssh srv stop demossh srv resize demo --cpus 4 --ram 8G --rootfs-size 20Gssh srv start demo

See Resize a VM for details.

5.7 Back up and restore

ssh srv stop demossh srv backup create demossh srv backup list demossh srv restore demo 

See Backup and restore for the full workflow.

5.8 Move a VM between hosts

ssh srv-a export demo | ssh srv-b import

See Export and import for the semantics.

5.9 Next steps

• Instance lifecycle — full command reference
• Networking overview — how VM networking works
• Configuration — all environment variables

6. Self-hosted Amp orbs on srv

srv-orb-dispatch connects Amp's self-hosted orb provider to existing srv SSH commands. It works with one srv host or with srv-front for automatic placement. It does not replace srv's control plane or the separate Amp runner plugin.

This is an initial integration: subprocess tests cover the dispatch contract, but image boot and the Amp connection still need validation on a real srv host.

Orbs use the same Firecracker, capacity accounting and placement infrastructure as VMs, but have a separate workload kind and default inventory:

ssh srv list              # ordinary VMs onlyssh srv orb list          # Orbs onlyssh srv list --all        # both, with their kindssh srv orb new demo --image amp-orb

Names remain unique across both kinds on each backend. Existing records remain VMs; they are not inferred or converted from image names. Previously created Tailscale-based Orbs need a deliberate rebuild rather than automatic adoption. Common inspect, start, stop, delete, backup and capacity commands still operate on both kinds. The same list separation applies through srv-front.

There is no RAM suspend or automatic migration. New orbs reuse a known ORB_SNAPSHOT_ID on its recorded backend; unknown IDs fall back to the base image. Existing orbs always retain their own disk and use Amp's resume workflow. Processes, Terminal sessions and in-memory state do not survive a pause. Let Amp manage development services so they can be restarted after wake-up.

6.1 Project snapshots

Amp requests a snapshot by sending pause with ORB_SNAPSHOT_ID, then wakes the same orb again. srv stops the source, reflink-clones its disk, sanitizes the copy, and publishes it atomically in an owner-scoped template namespace. The source disk remains unchanged. A later new orb gets a clone of this prepared template and fresh srv allocation, rather than a migration of the source VM. Templates retain their workload kind: an Orb cannot clone a normal VM template or vice versa.

Templates are host-local. The dispatcher persists their backend mapping and pins new template-based VMs to that host, even when SRV_ORB_HOST changes. There is no cross-host template replication or failover. A known but incomplete mapping, image mismatch, unavailable backend or missing template fails rather than silently claiming snapshot reuse. A missing mapping permits Amp's base-image fallback.

Snapshot IDs are immutable: repeat requests from the same source reuse the saved copy; another source cannot replace it. Templates survive source-orb deletion. Back up both dispatcher mappings and backend templates. Restoring only one side does not restore usable snapshots.

Amp documents no snapshot-delete dispatch operation. Retire a snapshot in Amp first, stop the provider and wait for outstanding jobs, then delete the backend template and its project-*.json dispatcher mapping. The mapping contains the backend and template name. Keep lock files. Backend commands, executed as the same srv owner, are:

ssh srv-front -- --json project-template create ORB_VM_NAME TEMPLATE_ID --host assh srv-front -- --json orb new NEW_ORB_NAME --image amp-orb --template TEMPLATE_ID --host assh srv-front -- --json project-template delete TEMPLATE_ID --host a

Use no --host when connecting directly to srv. Template creation requires a stopped source. No RAM, running processes or terminal sessions are copied.

Backends need debugfs and e2fsck from e2fsprogs. Cleanup operates on the copied ext4 disk in userspace; it never mounts the guest filesystem on the host. It removes machine/Tailscale identities and named Amp authentication/session files, while retaining prepared settings, skills, CLI binaries and project data. Spaces and Unicode names are supported; ambiguous debugfs output and escape/control characters in cleanup paths fail closed. Filesystem checks honor job cancellation. Templates remain confidential artifacts in the source's trust domain: removing active identity files is not secure erasure of free blocks/journals and does not remove arbitrary project secrets. Do not distribute them as public images.

6.2 Prepare an image

Build an Arch or Debian image with SRV_AMP_ORB=1. This adds user:user, passwordless sudo, Amp, Node.js, Python, tmux, the Desktop stack and a narrow host-only bootstrap agent. Orb builds omit Tailscale and mask native srv enrollment; ordinary image builds retain their existing boot and enrollment behavior. Use an Orb-enabled kernel with built-in virtio-vsock support and rebuild older Orb images for this transport.

Both image builders include Docker and Compose v2; the supported command is docker compose. Amp runs as user with passwordless sudo, not as root: use sudo docker compose for daemon operations with the default root-owned Docker socket. This is an access convention, not a privilege boundary against the agent, which already has sudo. No Docker-group grant or socket-mode change is required. Browser/test dependencies belong in the repository's .agents/setup. Podman, Nix and legacy Compose v1 are not interchangeable provider contracts; projects that need them must prepare and qualify those environments explicitly.

Register the prepared kernel/rootfs as amp-orb in SRV_IMAGES_FILE on every eligible backend. Keep any existing catalog entries. Example additional entry:

{  "amp-orb": {    "kernel": "/var/lib/srv/images/amp-orb/vmlinux",    "rootfs": "/var/lib/srv/images/amp-orb/rootfs-base.img"  }}

Check the configuration reference for the catalog's complete format. Do not use the raw Amp OCI image as a Firecracker rootfs. In this adapter, ORB_IMAGE is an explicit prepared srv image name, not an OCI URL. Set it in the provider's environment as shown below, otherwise Amp supplies its default OCI URL and the adapter rejects the job before creating a VM.

Desktop preparation copies /opt/amp-wayland from a checksum-pinned layer of Amp's official amd64 image and installs its distro-specific runtime dependencies. It needs build-time access to us-central1-docker.pkg.dev and blob redirects, but no Docker daemon. The source digest is retained inside the guest at /opt/amp-wayland/srv-layer.sha256. Bootstrap and login shells load the bundled program paths from /etc/amp-orb.env. The bundled executables use private RPATHs for their libraries; a global LD_LIBRARY_PATH would break newer distro clients.

The catalog name is your choice. If you register the image as amp-orb-desktop, normal srv clients can use:

ssh srv orb new demo --image amp-orb-desktop

For the Amp provider, set both ORB_IMAGE=amp-orb-desktop and SRV_ORB_IMAGE=amp-orb-desktop. srv-front already considers image availability when placing new VMs. This is provider configuration, not a new per-thread Amp image selector. Existing VMs keep their original rootfs; rebuild and create a new orb to pick up Desktop support. Desktop processes restart after pause rather than resuming their old in-memory session.

6.3 Provider machine and identity

Use a Linux machine with Amp, OpenSSH, network access to the control endpoint, and a persistent local filesystem supporting flock, atomic rename and fsync. It does not need KVM if the backends run elsewhere.

Give the provider a dedicated Tailscale node, optionally tagged, and explicitly map its stable node ID to a service owner. For example, on srv backends:

SRV_PROVIDER_NODES='{"nPROVIDER1":"amp-orbs"}'SRV_ALLOWED_USERS='service:amp-orbs'

Append to existing allowlists rather than replacing their entries. With srv-front, also configure its provider_nodes map and allowed_users for the same service owner; see provider identities. The frontend remains a separate trusted forwarding node, not a provider node. Tags or SSH usernames alone never grant a service identity. Unconfigured tagged clients remain rejected. Service owners cannot be configured as admins.

All orbs and project templates from this provider belong to service:amp-orbs, isolated from normal users' VM ownership. The dispatch contract contains no Amp member ID, so this does not separate members within one Amp workspace. Use separate provider identities for separate trust domains. An existing untagged user provider still works; switching its identity does not transfer existing VMs or templates automatically.

Configure ACLs so the provider can reach:

• The srv or srv-front SSH endpoint, under the existing allowed-user rules.
• ampcode.com and static.ampcode.com for provider traffic and CLI updates.

Guests need outbound HTTPS/WSS to ampcode.com, tunnel.ampcode.com and static.ampcode.com, plus their Git hosts and package registries. The dispatcher uses the per-VM Amp CONNECT gateway reported by inspect when present; otherwise bootstrap uses direct egress. The gateway's existing private-address restrictions still apply. Private Git hosts or a private AMP_URL require an appropriate network/proxy policy; this integration does not weaken it automatically.

Proxy bypass is provider configuration, not a security boundary. Bootstrap writes a fixed bypass list for guest-local services, MMDS, the CGNAT range and .ts.net tailnet MagicDNS names. To send additional destinations past the gateway — an internal registry, a private DNS suffix — set SRV_ORB_PROXY_BYPASS in the provider environment to a comma-separated list of up to eight entries (512 characters joined). Each entry is a hostname, an IPv4 literal, or a leading-dot DNS suffix; an optional trailing dot (the FQDN form) is accepted verbatim but only matches clients that compare the same form. The entries are client-native NO_PROXY tokens, not a universal matcher: label-boundary subdomain matching agreed across urllib, curl and Go's httpproxy in the runs verified for this change, but apex handling does not — urllib and curl match the bare apex host for a leading-dot entry while Go's httpproxy does not, so configure the bare hostname (which matched apex and subdomains in all three) when the apex itself must be covered. CIDR ranges and IPv6 literals are not accepted in this field for the same reason: Python's urllib ignores CIDR entries entirely (including the built-in CGNAT entry), while curl and Go honor them for IP literals. Empty entries, duplicates and repeats of built-in entries are configuration errors. The dispatcher validates the list and the guest independently re-validates every entry, before any bootstrap generation is consumed, and appends the list to the unchanged built-in entries; unset keeps exactly the built-in list, and repository setup must not be used to repair this provider networking. Bypass only removes the gateway from the path: the guest still needs actual routing to the destination.

Setting SRV_ORB_PROXY_BYPASS adds a proxy_bypass field to the bootstrap payload. A new dispatcher with the variable unset, and an older dispatcher against a new guest image, both keep working because the field is simply absent; but a set variable needs the dispatcher, srv backend and orb guest image upgraded together, since older components reject the extra field and the bootstrap fails rather than silently dropping the destination.

The control endpoint's SSH host key must be installed in the service user's known_hosts through a trusted channel. Control connections use strict host-key checking. The provider never connects to guest SSH: the authenticated backend forwards bootstrap over a local Firecracker vsock endpoint. Its guest listener accepts only host CID 2 and only ping/bootstrap requests, not arbitrary commands. SSH username selection does not grant control-plane permissions.

6.3.1 Optional project Tailscale

Guests do not require a Tailscale account, auth key or enrollment for readiness, bootstrap, Terminal, Desktop or portals. The srv control plane and provider still use Tailscale for their existing authenticated transport.

For project-specific tailnet access, follow Amp's Tailscale setup: install the client in .agents/setup, then start/authenticate it in .agents/resume using Amp OIDC and an ephemeral node. Tailscale is not preinstalled or masked; the project owns its installation and lifecycle. Do not bake reusable credentials into the image or snapshot. Keep the Amp connection independent of project networking; avoid an exit node and configure proxy bypasses for private destinations when necessary.

6.4 Install and run

Build and install the adapter:

go build -o /tmp/srv-orb-dispatch ./cmd/srv-orb-dispatchsudo install -m 0755 /tmp/srv-orb-dispatch /usr/local/bin/srv-orb-dispatch

Create a dedicated local account srv-orbs with home /var/lib/srv-orbs. Install a current Amp CLI accessible as /usr/local/bin/amp; for self-updates, keep its actual installation writable by this account. Configure its SSH trust and test ssh srv -- --json orb list as that account before starting the provider.

Initialize a new empty dispatcher directory explicitly, using the same endpoint and account as the service:

sudo -u srv-orbs env SRV_ORB_ENDPOINT=srv SRV_ORB_STATE_DIR=/var/lib/srv-orbs/dispatch \  /usr/local/bin/srv-orb-dispatch init

Never initialize over lost provider state. Restore its backup instead. Update srv, srv-front (when used), srv-vm-runner, srv-net-helper and the prepared guest image together before enabling this protocol. Older components fail closed; there is no fallback to ordinary delete or unbound bootstrap. Preexisting Orb records without a backend bootstrap counter require deliberate retirement and recreation, not a counter reset or automatic adoption. JSON inspection remains available for diagnostics, omitting bootstrap_generation when its row is missing; this does not initialize a counter or permit managed bootstrap.

Create a provider in Amp workspace Settings → Orbs → Add Orb Provider. Store its token in /etc/srv-orbs.env, owned by root with mode 0600. Copy the non-secret settings from srv-orbs.env.example and add AMP_ORB_PROVIDER_TOKEN privately. Do not put tokens in shell arguments, Git, images or thread messages.

sudo install -m 0644 contrib/amp/orbs/srv-orbs.service /etc/systemd/system/sudo systemctl daemon-reloadsudo systemctl enable --now srv-orbssudo journalctl -u srv-orbs -f

The service runs:

amp orb provider /usr/local/bin/srv-orb-dispatch

Select this provider in Amp's thread composer, or use:

amp -ox "Inspect this repository" --orb-provider YOUR_PROVIDER_NAME

6.4.1 Configuration

Sizes are operator configuration, not dispatch fields supplied by Amp. Changing these settings affects new VMs only. Existing VMs keep their disks and placement. AMP_URL, when provided by Amp on spawn, is passed to bootstrap.

Only one provider installation may own an Orb ID. Do not run independent providers with separate state directories against the same orbs. The dispatcher serializes same-orb jobs with local file locks; it is not an HA controller. Different orbs can run concurrently. Jobs time out after 270 seconds, including lock waits, leaving time inside Amp's five-minute dispatch limit.

6.5 Recovery and security

The state directory contains an identity/host mapping, operation phase and deletion record. Bindings include the backend's own SSH host-key fingerprint, VM UUID, creation timestamp, workload kind and image, not just a frontend placement label. Back it up and keep it private. It never stores the provider or orb tokens. The bootstrap token travels over control SSH stdin and the host-only vsock channel, briefly enters a root-only environment file under /run, and is removed after systemd starts bootstrap. It remains in the bootstrap process environment; root inside the VM is trusted. Guest bootstrap output is not relayed to provider logs because it could contain credentials.

The srv daemon and dispatcher disable process dumpability. For the daemon this also restricts ptrace and changes /proc ownership/access; debugging must account for those effects. Both token-bearing systemd units set LimitCORE=0. The example provider unit additionally refuses a piped host core_pattern: not all external collectors honor the process limit. This intentionally excludes a stock systemd-coredump configuration until the operator supplies a reviewed no-dump policy; the integration does not change host-wide core settings. Guest root and its configured collector are trusted; the guest unit's limit is not a guarantee against a privileged piped collector. The srv-front forwarding process, SSH client and Amp CLI remain part of the trusted transport/runtime boundary. Avoid diagnostic dumps of those processes.

The adapter records creating before sending orb new. If the SSH response is lost, a later job refuses automatic adoption or replay. It never automatically retries a missing, uncertain create. Initial discovery through srv-front requires every host to answer; after placement is recorded, only the selected host is queried, so an unrelated backend outage does not prevent wake-up.

An explicit srv-orb-dispatch bind ORB_ID UUID, under the provider's existing configuration, can bind a visible native Orb on the already recorded host with no import/migration history and bootstrap generation zero. A lost reply to automatic frontend placement has no recorded host and cannot use this escape hatch. Use a pinned host when deterministic create reconciliation is required. Missing dispatch records, counters or lock-associated records fail closed. Do not delete or hand-edit JSON/lock files to release a fence. A VM found without a record is not silently adopted: restore the state backup instead. An unknown allocation that cannot satisfy these checks remains unavailable pending a separate reconciliation design.

6.5.1 Retire compute and explicitly start a fresh workspace

An operator can permanently discard an old Orb's compute and workspace without requesting deletion of its Amp thread. Run this on the provider machine, using the same account, endpoint and SRV_ORB_STATE_DIR as the provider:

srv-orb-dispatch retire ORB_ID EXPECTED_INSTANCE_UUID

Take orb_id and instance_id from the existing dispatch record and confirm that its name, host and instance identity match the intended Orb in srv inspection. The UUID is an explicit confirmation, not a name lookup or force option. Unknown, unbound or mismatched instances are rejected. This is an operator command, not an additional ORB_OPERATION value.

Retirement destroys the Orb's disk, including uncommitted work. Save any required files first. Unlike pause, it is not resumable. Unlike deleting the thread in Amp, it makes no Amp API call: conversation history and attachments already stored by Amp remain subject to Amp's own access and retention rules. They are not a backup of the whole filesystem. Amp's UI may not immediately reflect provider-side retirement; subsequent dispatched spawn jobs explicitly refuse to resume until an operator authorizes a fresh workspace for that thread.

The command takes the existing per-Orb lock and uses srv's delete-proof API, including the complete backend identity and v3 gateway/runtime/network/storage proof. It records retiring before sending the request and retired only after verifying and saving the exact cleanup receipt. Project templates remain retained as listed in the receipt; backup and snapshot exclusions still apply. Ordinary Amp deletion records are not relabeled or upgraded to retirement.

If the reply is lost, repeat the same command and UUID. A pending retirement only queries the original receipt; it does not send a second deletion request. A crash before admission, failed preflight or incomplete cleanup stays fenced for operator reconciliation rather than reporting success from an absent VM. Uncertain start/stop operations still block retirement. Later Amp delete may finish reading the same retirement receipt, but makes the disposition ordinary deleted, not eligible for fresh start. Amp delete after completed retirement also closes that lifecycle permanently. pause after retirement is a no-op. The original identity, receipt and lock files remain.

To continue the same retained thread with a new workspace:

1. Quiesce the thread's upstream work and stop provider admission. With the example service this requires stopping srv-orbs and ensuring its dispatcher children and lock waiters have exited. Resolve outstanding srv operations and complete retirement first; killing a client is not proof that its backend work stopped.

2. While admission remains stopped, run under the same provider account/config:

   srv-orb-dispatch fresh-start ORB_ID RETIRED_INSTANCE_UUID

3. Resume provider admission only after old upstream work is drained. Send a new message in the retained Amp thread to request a fresh spawn and token.

The authorization command itself starts no VM and needs no token. It requires the completed retirement and its full v3 receipt, then records one successor with a distinct srv name. The next spawn uses normal base-image or ready project snapshot selection, creates a new instance UUID, and bootstraps it with the new token. Amp's documented fresh-disk contract runs repository setup again when needed; prepared snapshots may skip setup. This does not restore the retired thread's uncommitted changes. Subsequent pause/wake retains the successor's own disk and follows normal resume behavior.

Job-ordering limitation: Amp's documented dispatch contract has no upstream incarnation or sequence field. A local job that observed the old incarnation before waiting for its lock is rejected after the switch; old identity-bound srv commands cannot target the new name/UUID. But an old upstream job first delivered after the switch is indistinguishable from a current job. Stopping the local provider alone does not prove the upstream queue is empty. Establish that drain boundary before using fresh start; if it cannot be established, leave the Orb retired. This is not a guarantee against arbitrary upstream reordering.

Fresh authorization first archives the exact retired record as orb-<original-hash>.json.retired-<UUID>, leaving its cleanup receipt untouched, then atomically publishes the successor in the original root record. The stable lock is never replaced. Cleanup files are scoped to each incarnation's srv name; all predecessor identity/counter records and receipts must remain backed up. A crash before publication leaves the old retirement current; retry validates the archive and finishes publication. A lost reply after publication is safe to retry. Repeating any previous authorization never resets a successor, even after that successor has itself retired or been deleted. Each operation validates its immediate predecessor; fresh-start authorization also validates the history it traverses. Missing or conflicting records or proof on that path fail closed. Unknown create outcomes retain their existing fences.

A successor can consume a ready project template but cannot capture into its predecessor's snapshot ID. New mappings bind snapshot authorship to the instance UUID. Legacy mappings may only be completed by the original, non-replacement incarnation; a successor must use a new snapshot ID for new content.

Do not substitute bare ssh srv delete NAME, remove provider records, or reuse the managed instance name: those actions do not reconcile provider state. Neither operator command is an ORB_OPERATION value. There is no automatic expiry, bulk deletion or automatic recreation of missing disks. Older dispatchers reject the new phases or successor names; use the updated dispatcher after using these commands. The existing v3 backend receipt contract is unchanged.

6.5.2 Operation receipts and bootstrap generations

The dispatcher persists a fresh request ID before start, stop, bootstrap or delete. Bound start/stop accept --expect REQUEST UUID CREATED_AT BACKEND_ID IMAGE; unbound human start/stop commands retain their previous behavior. srv checks the binding and admission under the instance lock. Repeating an admitted request never reexecutes it. operation-result REQUEST only reads its stored result. After a lost outer SSH reply, a matching done result is sufficient to recover.

srv-orb-dispatch cancel ORB_ID REQUEST attempts cancellation of a pending start or stop. srv takes the same instance lock and installs an exactly bound cancel tombstone only if the request was never admitted. If the worker won the race, cancel reports its actual phase and never overwrites it. No operator flag asserts quiescence. Failed, executing or interrupted operations mean unavailable pending later reconciliation, not safe to retry. A stopped/started observation or srv reboot alone is insufficient: downstream runner work is not durably bound to the ingress request at every crash boundary. Stop/start remain fenced, and the dispatcher refuses delete while either is uncertain.

Bootstrap has a narrower recovery protocol. Before sending credentials the dispatcher fsyncs its next generation and request ID. srv atomically consumes a strictly increasing generation for the exact UUID/creation timestamp, together with request admission. It holds the instance lock through the vsock exchange. A new spawn may supersede an uncertain bootstrap with a higher generation; an equal generation never executes again. Lower/missing/inconsistent provider or backend counters fail closed, rather than catching up or resetting. Pause remains refused while bootstrap is uncertain. Receipt-based deletion is allowed without an intervening spawn, provided the normal cleanup proof succeeds.

The guest serializes generation check, consumption and effects with a lock in /run/srv-orb. It first records executing, installs KillMode=control-group, synchronously stops any old unit/job with systemd's replace job mode, and checks inactive state, no queued job and an empty cgroup. A completed forced kill may leave systemd's unit in failed state; only after checking no processes remain is that bookkeeping cleared and inactive state checked again. Only then does it write the new environment, start the unit, verify active, unlink the environment file, record done and acknowledge the exact generation. Consuming a generation is never reported as success. The acknowledgment certifies the old unit's processes are gone and the replacement exec is active with new credentials; it does not certify Amp registration.

The journal survives agent restarts within a guest boot and is never part of disk templates. A guest reboot starts a new Firecracker process, discarding the old guest's vsock/backlog and unit processes; srv's durable high-water excludes older host-queued requests. This argument depends on that runtime boundary, not on arbitrary state rollback. Deleting all /run journal state within a live guest, restoring coherent old copies of all stores, or cloning backend host keys is outside the supported recovery model. Preserve state and distinct host keys. Partial journal loss refuses initialization. No reset/adoption command exists. Old guests without the echoed generation acknowledgment are rejected.

6.5.3 Authoritative native-Orb deletion

delete-proof REQUEST NAME UUID CREATED_AT BACKEND_ID IMAGE is native-Orb-only; ordinary VM deletion is unchanged. cleanup-receipt REQUEST reads the stored result without invoking deletion. Both require an explicit --host through srv-front. A failed or lost delete reply is queried by that same request ID, never retried with another ID and never downgraded to plain delete.

New deletions publish v3 receipts with gateways_absent: true. After preflight and durable cleanup admission, srv closes this instance's Amp CONNECT, provider and integration listeners, cancels their upstream requests and upgraded streams, and waits for admitted handlers (including pending dials) to finish. Gateway reconciliation cannot reopen the fenced instance from stale inventory. A timeout leaves cleanup fenced without a receipt; a preflight rejection leaves the live gateways untouched. Neighboring instances' gateways remain available. This proves absence of per-instance listeners and active handlers/streams, not absence of every host socket: shared idle upstream transport pools are retained.

Upgrade srv and srv-orb-dispatch together: the dispatcher now requires v3 to finalize a pending deletion. Stored v2 receipts remain readable, unchanged; they do not prove gateway quiescence and cannot finalize a deleting dispatcher record under the new version. Already-finalized deleted records still make repeat delete a no-op and reject spawn; that no-op does not upgrade the historical receipt to v3. Do not rewrite receipts or tombstones to bypass this distinction. The existing coordinated runner/nethelper/guest-image protocol requirements still apply; this change does not deploy or validate any installed host binary.

The backend fences ingress and drains gateways, then asks the VM runner to prove runtime/jailer/cgroup absence. The exact recorded start receipt must be closed, and the runner scans the entire name's receipt journal for corrupt identities or nonclosed attempts. It checks for no live VM with that name, no jailer workspace or runtime socket, and no delegated cgroup. This is name-scoped proof, not a separate UUID-bound enumeration of every historical runtime path. Nethelper proves network absence only. Missing/older helper endpoints and unreadable resources fail closed. It removes and fsyncs instance storage, checks backups and local data snapshots, and atomically commits the receipt, deleted tombstone and release of the cleanup fence. Failed cleanup remains charged by the existing srv capacity accounting. There is no added scheduler or independent slot counter.

never-minted is derived only for native Orbs with no enrollment-key row and no node ID. It says nothing about project-managed Tailscale or external copies. Sorted source-UUID-matched project templates are explicitly retained in the receipt, never erased or called absent. Local data snapshots containing source disks, backups or template copies block proof; they need a separate retention decision. Receipt/request records survive name reuse. An uncertain bootstrap can be deleted through this path: the runtime proof closes the guest work and the backend tombstone prevents late host admission. Uncertain start/stop cannot.

Do not manually migrate, delete/recreate or reuse managed orb names. A missing managed disk causes spawn to fail rather than creating an empty replacement. Migration state is rejected for operator recovery. Completed deletion records prevent a late spawn from resurrecting a deleted orb. Only a receipt-backed retirement can authorize a distinct successor through fresh-start; ordinary deletion cannot. This does not solve upstream reordering of jobs: the documented contract has no sequence number.

For bootstrap errors, inspect the VM and run, through a trusted guest session:

systemctl status srv-orb-bootstrap.servicejournalctl -u srv-orb-bootstrap.servicesystemctl status srv-orb-guest.service

Treat journal contents as sensitive. No service is enabled at guest boot with an old token: the next spawn supplies fresh credentials and restarts only bootstrap.

6.6 Validate on a real host

Before relying on this provider, create a disposable Amp thread and verify:

1. The guest boots and registers with Amp without Tailscale or guest SSH; setup runs successfully. Verify separate VM/Orb lists and common capacity accounting.
2. Write a file and install a dependency; pause and wake the thread. Both survive, while terminal processes do not. Verify .agents/resume behavior.
3. Open an Amp Terminal, Portal and Desktop; launch a graphical application and verify reconnect/service startup after wake (the old desktop session is lost).
4. With srv-front, take an unrelated backend out of service and wake the orb on its recorded healthy host.
5. Let Amp save a project snapshot; create a second thread from it. Verify prepared dependencies survive, setup is skipped, and machine ID and bootstrap credentials differ. If the project enables Tailscale, verify a fresh node identity too. Verify the original still wakes.
6. Delete the source thread, then create another from its snapshot. Verify the template survives while the source VM/disk is removed.
7. Under the configured service identity, verify foreign users' VMs are invisible and admin commands are rejected, both directly and through srv-front.
8. On a disposable retained thread, follow the retirement and drain procedure, then authorize fresh start. Verify the thread/history remain readable, the replacement has a different name/UUID and fresh bootstrap credentials, old disk-only files are absent, and repository setup or snapshot activation runs. Change a replacement file, pause/wake and confirm it survives. Repeat the old authorization and confirm no reset. Repeat retirement/fresh start, then delete the thread and verify terminal cleanup with all prior receipts retained.

Local automated checks:

go test ./cmd/srv-orb-dispatch ./internal/store ./internal/service ./internal/provision ./internal/front ./internal/vmrunner ./internal/nethelper ./internal/orbguestpython3 -m unittest discover -s contrib/amp/orbs -p 'test_*.py'# Disposable root/systemd environment only; unique temporary unit, synthetic token:sudo python3 contrib/amp/orbs/probe_guest_systemd.py

The Go dispatch tests use a subprocess SSH stand-in, not Firecracker or Amp. The root probe exercises the real guest handler, systemd jobs, token-holding descendants and environment lifecycle over a socketpair, not real vsock. Neither proves Amp/Tunnel interoperability or the guest-reset runtime boundary. Provision tests additionally check real temporary ext4 images, identity cleanup, retained settings/skills, immutable templates and cancellation. Actual Amp setup-skipping still requires the live cycle above; its internal completion-marker format is not a documented provider contract.

7. Sandboxed AI coding agent

srv makes it straightforward to run AI coding agents in isolated microVMs. Each VM gets its own cgroup limits, per-instance Tailscale identity, and optional provider API proxies that inject host API keys without exposing them inside the guest.

For Amp users, the repository also includes an opinionated plugin that creates the VM, installs a dedicated Amp runner, opens a thread on that runner, and later pushes the thread's branch from the VM.

For an outcome-oriented overview, including Amp remote control, thread URLs, Tailscale development services, and a comparison with Orbs, start with Amp runners on srv.

7.1 Create a VM for an agent

ssh srv new agent-1 --cpus 4 --ram 8G --rootfs-size 30G

Wait for it to report ready:

ssh srv inspect agent-1

Look for state: ready and a tailscale-ip.

7.2 Provider API proxies

When provider API keys such as SRV_ZEN_API_KEY or SRV_DEEPSEEK_API_KEY are configured on the host, srv binds per-instance HTTP proxies on the guest's gateway IP and the provider gateway ports. The proxies:

• Only accept requests from that VM's guest IP
• Forward /v1/... requests to upstream provider APIs with host keys injected
• The guest bootstrap writes /root/.config/opencode/opencode.json and Pi config under /root/.pi/agent/ pointing at these gateways

This means the agent inside the VM can use opencode, pi, or any OpenAI-compatible client against the per-provider gateway URLs without ever seeing the real API keys.

7.3 Connect the agent

ssh root@agent-1

The preinstalled opencode and pi CLIs are already configured to target the per-VM gateway. Chromium and agent-browser are also installed; agent-browser is configured to launch the system browser at /usr/bin/chromium instead of downloading a separate browser build.

If you are using a different agent framework, point its API client at:

http://:11434/v1

The gateway IP is the default route inside the VM. You can read it from the inspect output under host-addr.

7.4 Run Amp on a dedicated VM runner

Install the bundled Amp plugin on the workstation where Amp is running:

mkdir -p ~/.config/amp/pluginsln -sf "$PWD/contrib/amp/plugins/srv-runner.ts" ~/.config/amp/plugins/srv-runner.ts

Reload plugins from Amp's command palette after installing or updating it. The plugin expects:

• A Git repository with an origin remote, a checked-out branch that already exists on origin, and no detached HEAD
• Working ssh srv ... and ssh root@<vm-name> access from the workstation
• An authenticated Amp installation at ~/.local/share/amp/secrets.json
• A local SSH agent when the repository uses an SSH remote

Run srv Runner: New isolated thread from the command palette. The plugin asks for the task, Amp mode, VM size, and explicit consent to copy the Amp credential and configured defaults. It then:

1. Creates an amp-<repository>-<suffix> VM and waits for guest SSH.
2. Copies the Amp credential and optional workstation-wide profile into the persistent VM.
3. Installs Amp, makes a fresh clone of the current branch under /workspace/repository, initializes submodules, and runs .agents/setup when that executable exists.
4. Configures amp-runner.service to use the VM's per-instance Amp CONNECT gateway, starts a dedicated Amp runner, creates a thread on it, and sends the requested task.

The clone comes from origin; uncommitted and untracked workstation changes are not copied. Credentials remain in the persistent guest until the VM is deleted, so treat the VM as sensitive state.

The plugin also provides commands to show the VM for the current thread, list all managed VMs, start or stop the current VM, retry failed provisioning, and permanently delete a managed VM. Failed provisioning leaves the VM intact for inspection.

7.4.1 Amp remote control and thread features

The dedicated process runs with --remote-control-terminal. It is a normal Amp runner thread rather than a separate agent UI, so the thread remains available through ampcode.com on desktop or mobile with its conversation, visibility, changes UI, and remote terminal while the VM is online.

Amp plugins and agents can target the live runner for additional threads. Scheduled or plugin-created work needs the runner to be online, and Amp cannot currently wake a stopped srv VM; start it first with srv Runner: Start current thread VM. Amp currently limits Multiplayer to Orb-backed threads.

7.4.2 Development services over Tailscale

The VM's Tailscale name is a private network endpoint for more than SSH. Services that listen on a Tailscale-reachable address can be opened directly from authorized tailnet devices over TCP or UDP—for example a web app on http://<vm-name>:3000 or a database on <vm-name>:5432.

This covers the connectivity role of an Amp Portal and supports more protocols, but it does not embed the service in the Amp thread, create Amp-managed HTTPS, or grant access based on thread visibility. Tailnet ACLs control access. See Networking overview.

7.5 Push an isolated thread's branch

From a plugin-managed thread, run srv Runner: Push current VM branch…. This command only supports SSH Git remotes. Before asking for confirmation, it reads the VM's current branch and commit over host-key-verified SSH. The confirmation shows the repository, branch, and abbreviated commit that will be pushed.

After confirmation, the plugin temporarily forwards the workstation's SSH agent and pushes that exact commit to the same branch. The push:

• Aborts if the VM's branch or commit changes after confirmation
• Uses a temporary bare repository and disables repository hooks and global/system Git configuration for the operation
• Requires strict host-key checking for the Git server
• Ends agent forwarding when the command completes, with a two-minute overall timeout

The push does not include uncommitted changes. Commit all intended VM changes first.

7.6 Resource limits

Each VM runs in its own cgroup v2 leaf with:

• cpu.max — capped at the vCPU count
• memory.max — capped at the requested guest RAM plus a small Firecracker overhead reserve
• memory.swap.max — set to 0 (no swap)
• pids.max — default 512, configurable via SRV_VM_PIDS_MAX

This prevents a misbehaving agent from consuming the entire host.

7.7 Clean up

ssh srv delete agent-1

For a plugin-managed thread, srv Runner: Delete current thread VM removes the VM, persistent checkout, copied Amp credential, workstation profile files, and local plugin record.

7.8 Multiple agents

Create as many agent VMs as the host can hold. Each gets independent networking, identity, and resource limits:

ssh srv new agent-2 --cpus 2 --ram 4Gssh srv new agent-3 --cpus 2 --ram 4G

Use ssh srv status to check remaining host capacity before creating more.

8. Architecture

srv is a single-host control plane that manages Firecracker microVMs through an SSH command surface on a Tailscale tailnet.

8.1 Control plane

┌────────────────────────────────────────────────────────┐│                      srv process                       ││                                                        ││  ┌──────────┐  ┌──────────────┐  ┌──────────────────┐  ││  │ tsnet    │  │ gliderlabs/  │  │ authorization    │  ││  │ :22/tcp  │──│ ssh          │──│ (Tailscale       │  ││  └──────────┘  └──────────────┘  │ WhoIs)           │  ││                                  └──────────────────┘  ││                                                        ││  ┌──────────┐  ┌──────────────┐  ┌──────────────────┐  ││  │ SQLite   │  │ reflink      │  │ Tailscale API    │  ││  │ store    │  │ cloner       │  │ (auth key        │  ││  └──────────┘  └──────────────┘  │ minting)         │  ││                                  └──────────────────┘  ││                                                        ││  ┌─────────────┐  ┌─────────────────────────────────┐  ││  │ Provider    │  │ per-VM HTTP proxies             │  ││  │ gateways    │  │ (inject host API keys)          │  ││  └─────────────┘  └─────────────────────────────────┘  │└────────────────────────────────────────────────────────┘         │                    │                  │    ┌────┴────┐          ┌────┴────┐        ┌────┴─────┐    │ net-    │          │ vm-     │        │ Tailscale│    │ helper  │          │ runner  │        │ coord.   │    │ (root)  │          │ (root)  │        │ server   │    └────┬────┘          └────┬────┘        └──────────┘         │                    │    ┌────┴────┐          ┌────┴─────────────┐    │ TAP +   │          │ Firecracker      │    │ iptables│          │ + jailer         │    │ + NAT   │          │ → srv-vm:srv     │    └─────────┘          │ → cgroup v2      │                         └──────────────────┘

8.2 Key components

8.2.1 tsnet + gliderlabs/ssh

srv joins the tailnet as the hostname configured in SRV_HOSTNAME (default srv) and listens on the tailnet TCP port configured in SRV_LISTEN_ADDR (default :22). The gliderlabs/ssh library handles SSH exec requests and rejects shell sessions.

Caller identity comes from Tailscale WhoIs data resolved from the incoming tailnet connection — not from the SSH username.

8.2.2 SQLite store

All instance state, event history, command audits, and authorization decisions are stored in SQLite under SRV_DATA_DIR/state/app.db. Migrations run during startup and are additive.

8.2.3 Reflink cloning

When a new VM is created, the base rootfs is cloned using filesystem reflinks (on btrfs or reflink-enabled xfs). This gives each VM its own writable copy without actually copying the data — the copy is instantaneous on a reflink-capable filesystem.

8.2.4 Network helper

A root-only helper process owns all TAP device creation, iptables MASQUERADE rules, and FORWARD rules. The main srv process communicates with it over a unix socket.

8.2.5 VM runner

A root-owned process invokes Firecracker through the official jailer binary. After the jailer sets up the chroot and drops privileges, the microVM process runs as srv-vm:srv. Each VM is placed into its own cgroup v2 leaf under firecracker-vms/<name> with enforced limits.

8.2.6 Tailscale integration

For each new VM, the control plane:

1. Mints a one-off Tailscale auth key using the configured OAuth credentials and tags
2. Injects the key into the VM's MMDS payload
3. The guest bootstrap service reads the key and runs tailscale up --auth-key=... --ssh

On warm reboots (after a stop + start or restart), the guest reuses its persisted tailscaled state instead of minting a new key.

8.2.7 Provider gateways

When SRV_ZEN_API_KEY or SRV_DEEPSEEK_API_KEY is set, srv binds a per-instance HTTP proxy on each VM's gateway IP and the provider's configured port (SRV_ZEN_GATEWAY_PORT for Zen, SRV_DEEPSEEK_GATEWAY_PORT for DeepSeek). The proxies forward guest requests to the upstream APIs while injecting the host keys. See Provider gateways for details.

8.3 Data paths

SRV_DATA_DIR/├── state/│   ├── app.db              # SQLite store│   ├── tsnet/               # Tailscale persistent state│   └── host_key             # SSH host key├── images/│   └── arch-base/│       ├── vmlinux          # Guest kernel│       └── rootfs-base.img  # Base rootfs (reflink source)├── instances/│   └── /│       ├── rootfs.img       # Writable reflink clone│       ├── serial.log       # Serial console output│       └── firecracker.log  # VMM log├── backups/│   └── /│       └── /│           └── rootfs.img   # Backup copy├── jailer/                 # Jailer workspaces└── .snapshots/              # btrfs host snapshots    └── /

8.4 Service architecture

The srv-vm-runner process starts as root, and the jailer drops the Firecracker process to srv-vm:srv within each VM's cgroup v2 leaf.

9. SSH command reference

All srv commands are invoked through the SSH surface. The service treats SSH as command transport only — there is no shell session.

9.1 Syntax

ssh srv  [args]

For machine-readable output, use --json with supported non-streaming commands such as list, inspect, status, integration ..., and pool .... With OpenSSH, terminate local option parsing first:

ssh srv -- --json listssh srv -- --json inspect demossh srv -- --json status

9.2 Instance commands

9.3 Lifecycle commands

Lifecycle commands work for both VMs and Orbs. Names share one namespace per backend; the image name does not determine the workload kind. Existing instances remain VMs. See Amp orbs for bootstrap and optional project networking. orb bootstrap <name> is a streaming provider command: credentials are JSON on stdin, never command arguments; it does not accept --json.

9.4 Resize command

All flags can be combined. Omitted flags keep the current value.

For pooled VMs, resize --ram keeps the VM in the same pool. In v1 it is allowed only while the VM is stopped and only up to the pool's reserved size.

9.5 Backup commands

9.6 Project templates

These commands support --json and require explicit --host through srv-front. Unlike backups, templates create new identities rather than restore an existing VM. Templates outlive their source VM. See Amp orbs for storage, security and retention requirements.

9.7 Transfer commands

Usage: ssh srv-a export demo | ssh srv-b import

Pooled exports require an explicit destination memory policy. For coordinated cold migration, host-qualified names, and node list across servers, see the multi-host frontend.

9.8 Admin and host commands

9.9 Integration commands

All integration commands are admin-only.

9.10 Notes

• new accepts --cpus, --ram, --pool, and --rootfs-size in any combination that matches the memory mode contract
• new --pool <name> requires --ram <size> because guest-visible RAM stays explicit in pooled mode
• new also accepts repeated --integration <name> flags; the create request fails if any requested integration cannot be enabled
• Fixed mode remains the default. Without --pool, --ram means guest-visible RAM; the hard per-VM host cgroup cap is that RAM plus max(256 MiB, RAM/16) Firecracker overhead reserve
• In pooled mode, --ram means guest-visible RAM while host reservation comes from the pool created with pool create
• resize requires the VM to be stopped; CPU and RAM may increase or decrease within limits, while rootfs is grow-only
• Pooled VMs cannot move between pools or switch memory mode through resize in v1
• resize, fsck, backup, and restore all require the VM to be stopped
• Backups are tied to the original VM record — they cannot be restored onto a different VM
• Export requires the source VM to be stopped
• Import recreates the VM under the same name and leaves it stopped
• top refreshes continuously by default; use ssh -t srv top --interval 2s or similar to slow the redraw rate
• inspect shows memory-mode and whether host reservation is dedicated or shared via a pool
• status memory accounting now splits fixed reservations from pool reservations so pooled member VMs are not double-counted
• top shows live/configured RAM for every VM and appends the pool name for pooled rows
• Integration targets are intentionally narrow in v1: HTTP only, operator-managed, no guest-supplied raw secrets, and no automatic outbound interception

10. Testing

srv separates fast correctness checks, generated lifecycle models, process-crash matrices, fuzzing, and mutation checks. Use the narrowest tier that exercises the changed boundary.

10.1 Fast

go test ./...go vet ./...gofmt -l cmd internal

The default suite covers the store, provision, service, vmrunner, nethelper, config, front, and CLI packages. It needs no root, no KVM, and no network.

10.2 Race

GORACE=atexit_sleep_ms=0 go test -race -count=1 -timeout=15m ./...

cmd/srv-orb-dispatch re-execs the race-instrumented test binary for every fake SSH call. Disabling the race runtime's atexit sleep keeps that harness inside the package timeout; without it the package can exceed the default limit.

10.3 Generated lifecycle model

go test ./internal/store -run '^TestLifecycleStateMachine$' -count=1

Hegel generates up to 15 operations per case over a fresh SQLite store: create, admit fresh/same/changed requests, begin/finish start requests, begin/commit cleanup, reserve/fresh/existing/collision imports, create an import instance, finalize, and interrupt. An independent model tracks instance state, request binding/phase/exit, fence, cleanup receipt, and import receipt. Eligible IDs are sorted before drawing so replay and shrinking act on the same rows.

The model is sequential and store-level. It does not cover provision/service admission policy, migration, concurrency, or crash behavior. Set HEGEL_STATISTICS=1 to print rule-event counts.

10.4 Provision policy matrix

go test ./internal/provision -run '^TestProvisionPolicyMatrix$' -count=1

Exhaustively combines migration phase (none/preparing/prepared/completed/aborted), import receipt (none/importing/imported), and cleanup fence (none/fenced). It asserts PrepareMigration and DeleteWithProof outcomes, file removal, tombstones, and receipts. Interrupted preparing retries are exercised with the matching token/target and with a changed binding. The preparing/prepared/completed plus importing combinations are unreachable through the public import path and are skipped.

10.5 Process-crash matrices

• internal/store: SIGKILL at every scripted lifecycle boundary, reopen, and exact durable-prefix checks.
• internal/vmrunner: SIGKILL while holding the name lock, SIGKILL after the accepted receipt is durable, receipt-scan recovery, and a started receipt whose runtime cannot be verified.
• internal/nethelper: setup/cleanup failures at every command, retry idempotence, missing-state tolerance, and existing-rule skipping.
• internal/provision: import staging/commit-failure, cleanup admission window, restore retry after a partial replacement, and missing-rootfs start refusal.

These cover process crashes, not power-loss durability or fsync ordering.

10.6 Fuzzing

go test -run='^$' -fuzz='^FuzzSetupHandler$' -fuzztime=20s ./internal/nethelper

Targets:

A returned error is a normal result only when the target's contract expects one; a panic, hang, or violated contract fails. The committed seeds run in the normal go test tier. CI runs a 20-second campaign for nine high-value targets.

10.7 Mutation audit

python3 test/mutation_audit.py

The audit copies the worktree, verifies the unmodified focused tests pass, then disables one durable guard at a time. Each mutant must fail with a focused test assertion; a surviving mutant, missing anchor, build/setup failure, or timeout fails the audit. It covers cleanup fence deletion, deleting-state commit, request rebinding, begin-phase gating, reserve collision, finalize receipt publication, and the migration import-fence policy guard. The original worktree is not modified.

10.8 Benchmarks

make bench

Benchmarks cover store reads/list/admission and service parsing. They are not a CI gate; run them when changing those paths.

10.9 Known limits

• Process-crash tests do not model loss of kernel-cached writes or fsync ordering.
• The nethelper tests use a recording runner; real iptables/tap behavior still needs a disposable host.
• The Hegel model is store-level and sequential.
• TestDeleteProofQuiescesGatewaysBeforeReceipt/late-dial validates an implicit HTTP response only on the pending CONNECT path; established tunnels must close without any bytes.

11. HTTP integrations

srv can expose selected upstream HTTP APIs to a VM through a host-side integration gateway without storing the real upstream credentials inside the guest.

This feature is intentionally narrow in v1:

• Admin or operator managed only
• HTTP integrations only
• No guest-side secret submission over SSH
• No discovery, tagging, or attachment system
• No transparent interception of arbitrary outbound HTTPS

11.1 How it works

┌──────────────────────────────────────────────────────┐│                       Host                           ││                                                      ││  srv control plane                                   ││      │                                               ││      │ integration add/enable                        ││      ▼                                               ││  SQLite metadata + /etc/srv/srv.env                  ││      │                                               ││      ▼                                               ││  Integration gateway :11435 on VM gateway IP         ││      │                                               ││      │ /integrations/openai/...                      ││      │ injects headers/auth from SRV_SECRET_*        ││      ▼                                               ││  upstream HTTP API                                   │└──────────────────────────────────────────────────────┘                       │                       │ /30 network                       │              ┌────────┴────────┐              │ Guest VM        │              │                 │              │ curl http://    │              │ :11435 │              │ /integrations/  │              │ openai/...      │              └─────────────────┘

Each enabled VM gets access to a host-side HTTP listener on its gateway IP and SRV_INTEGRATION_GATEWAY_PORT (default 11435). Requests are routed by path prefix:

• http://<gateway-ip>:11435/integrations/<name>/...

The host looks up the named integration, rewrites the request to the configured upstream target, injects the configured auth or headers, and forwards the request.

11.2 Supported auth and header modes

An integration can use any combination of:

• Static headers via --header NAME:VALUE
• Env-backed headers via --header-env NAME:SRV_SECRET_FOO
• Bearer auth via --bearer-env SRV_SECRET_FOO
• Basic auth via --basic-user USER --basic-password-env SRV_SECRET_BAR

Secrets are referenced by env var name only. The raw values are expected to live on the host, typically in /etc/srv/srv.env.

For --header-env, the env var value becomes the exact header value sent upstream.

11.3 Security behavior

The gateway is designed so the guest gets access to the upstream capability, not the underlying secret material.

• The gateway only accepts requests from the owning guest IP
• Guest-supplied Authorization and Proxy-Authorization headers are stripped on the host
• Env-backed secrets are read on the host at request time
• If a referenced SRV_SECRET_* env var is missing, the gateway returns 502 Bad Gateway
• Request paths are normalized before routing so traversal attempts cannot escape the configured integration prefix

11.4 Example setup

Add host secrets to /etc/srv/srv.env:

SRV_INTEGRATION_GATEWAY_PORT=11435SRV_SECRET_OPENAI_PROD=sk-live-redactedSRV_SECRET_VENDOR_API_KEY=vendor-key-redacted

Create and enable integrations from the control plane:

ssh srv integration add http openai --target https://api.openai.com/v1 --bearer-env SRV_SECRET_OPENAI_PRODssh srv integration add http vendor --target https://vendor.example/api --header-env X-API-Key:SRV_SECRET_VENDOR_API_KEYssh srv integration enable demo openaissh srv integration enable demo vendorssh srv inspect demo

The VM's inspect output will include URLs such as:

integrations:- openai: http://172.28.0.1:11435/integrations/openai- vendor: http://172.28.0.1:11435/integrations/vendor

From inside the guest, requests go to the gateway IP:

curl http://$(ip route show default | awk '{print $3}'):11435/integrations/openai/modelscurl http://$(ip route show default | awk '{print $3}'):11435/integrations/vendor/ping

11.5 Related references

• SSH command reference
• Configuration reference
• Networking overview

12. Running as a daemon

srv is designed to run as a set of systemd services on a prepared host. The installer sets up three units that must all be active for the control plane to function.

12.1 Systemd units

12.2 Common systemd commands

# Check statussudo systemctl status srv srv-net-helper srv-vm-runner# View logssudo journalctl -u srv -fsudo journalctl -u srv-vm-runner -f# Restart all servicessudo systemctl stop srv srv-net-helper srv-vm-runnersleep 5sudo systemctl start srv-vm-runner srv-net-helper srv

12.3 Host reboot recovery

When srv-vm-runner.service stops, its systemd ExecStop drain discovers running fixed and pooled Firecracker VMs, asks up to eight guests at a time to shut down cleanly, and falls back to terminating guests that do not exit. The drain is bounded by the unit's 120-second stop timeout. It cleans runtime cgroups and sockets without changing the control plane's desired instance state.

Same-host reboot recovery is built into the control plane. Because the drain preserves that desired state, previously active instances are restarted automatically when srv comes back under systemd.

You do not need to manually restart VMs after a host reboot.

Check journalctl -u srv-vm-runner if a service stop reports a drain failure. After the timeout, systemd still terminates processes left in the unit cgroup; the next control-plane startup reconciles persisted state.

12.4 Environment file

Configuration lives in /etc/srv/srv.env. See the configuration reference for every variable.

If /etc/srv/srv.env already exists before an upgrade, the installer keeps it by default. After upgrade, verify that SRV_FIRECRACKER_BIN and SRV_JAILER_BIN still point at the intended static binaries.

12.5 Upgrading

1. Take a quiesced backup or a host snapshot.

2. Build and install the new srv, srv-net-helper, and srv-vm-runner binaries.

3. If you also refreshed Firecracker, verify /etc/srv/srv.env still points at the correct binaries.

4. Restart:

   sudo systemctl stop srv srv-net-helper srv-vm-runnersleep 5sudo systemctl start srv-vm-runner srv-net-helper srv

5. Run the smoke test:

   sudo ./contrib/smoke/host-smoke.sh

See the operations reference for full upgrade and rollback procedures.

12.6 Smoke test as a validation gate

The smoke test is part of the supported workflow after install, restore, and upgrade — not an optional extra:

sudo ./contrib/smoke/host-smoke.sh

Overrides:

• ENV_PATH=/etc/srv/srv.env.alt — alternate environment file
• SMOKE_SSH_HOST=srv-test — alternate control-plane hostname
• INSTANCE_NAME=smoke-manual — force a predictable instance name
• KEEP_FAILED=1 — leave a failed instance intact for debugging
• READY_TIMEOUT_SECONDS=300 — override guest-ready timeout

13. Authorization

srv uses Tailscale identity for authentication and authorization. The SSH surface does not use traditional SSH username/password authentication — instead, it resolves the caller's Tailscale identity from the incoming tailnet connection.

13.1 How it works

1. A user connects to srv over the tailnet: ssh srv <command>
2. tsnet resolves the connection using WhoIs to get the Tailscale login and node
3. The control plane checks the resolved identity against the configured allow and admin lists
4. If authorized, the command runs; the decision is recorded in the audit store

13.2 User roles

13.2.1 Regular users

If SRV_ALLOWED_USERS is unset or empty, any authenticated tailnet user can invoke commands. Regular users can:

• Create VMs (new)
• Manage their own VMs (inspect, logs, stop, start, restart, fsck, delete)
• View their own VMs in list
• Create backups and restores for their own VMs
• Resize their own VMs
• Export their own VMs

13.2.2 Provider services

SRV_PROVIDER_NODES maps explicit stable Tailscale node IDs to non-human owners such as service:amp-orbs. Configured tagged nodes can use this path; they do not become human users or admins. Existing allowlists and VM ownership checks still apply. See provider identity configuration. Project templates are owner-scoped too. A provider does not distinguish the Amp members who submit jobs; use separate identities for separate trust domains.

13.2.3 Admin users

Users listed in SRV_ADMIN_USERS can additionally:

• View and manage all VMs, not just their own
• Run status (host capacity summary)
• Run snapshot create (host-level btrfs snapshot)
• Import VMs

13.3 Configuration

Example /etc/srv/srv.env:

SRV_ALLOWED_USERS=alice@example.com,bob@example.comSRV_ADMIN_USERS=ops@example.comSRV_GUEST_USER_AUTH_TAGS=alice@example.com=tag:alice,bob@example.com=tag:bob

Creator-specific guest tags are selected only from this operator-controlled mapping after WhoIs succeeds. The new command does not accept caller-supplied tags.

For the optional multi-host frontend, delegation is bound to the frontend's stable node ID, not its SSH username or display name. Backends still enforce the forwarded user's allowlist, admin role, and ownership. Trusted frontends can assert any user identity; protect their credentials and state like administrative access. Tagged nodes need explicit provider-node mapping or trusted-frontend delegation rather than being treated as human users.

13.4 Audit trail

Every command invocation is recorded in the SQLite store with:

• Actor Tailscale login and display name
• Actor node name
• Remote address
• SSH user
• Command and arguments
• Whether the command was allowed
• Reason for allow/deny
• Duration in milliseconds
• Error text if any

This audit trail is visible through ssh srv -- --json inspect <name> in the instance events.

14. Configuration reference

srv is configured through environment variables, read from /etc/srv/srv.env by the systemd units.

14.1 Provider service owners

Provider nodes can have a fixed, non-human owner identity, including tagged Tailscale nodes. This is opt-in and uses stable Tailscale node IDs, never hostnames, tags, SSH usernames, Dispatch parameters or Amp member IDs.

SRV_PROVIDER_NODES='{"nPROVIDER1":"amp-dispatch","nPROVIDER2":"amp-dispatch"}'SRV_ALLOWED_USERS='alice@example.com,service:amp-dispatch'

srv --provider-nodes '{"nPROVIDER1":"amp-dispatch"}' overrides the environment value. The JSON object maps each node ID to a service name matching [a-z0-9][a-z0-9_-]{0,63}. The resulting owner is service:amp-dispatch; values cannot be human logins. Configured nodes use this owner even if their WhoIs profile belongs to a human/admin. Unconfigured tagged nodes remain denied; unconfigured ordinary user nodes keep normal user authentication.

For srv-front, add the same mapping to its static --config JSON:

{  "provider_nodes": {"nPROVIDER1": "amp-dispatch", "nPROVIDER2": "amp-dispatch"},  "allowed_users": ["alice@example.com", "service:amp-dispatch"]}

These are additional fields, not a complete frontend config. Configure the mapping on each backend too: forwarded service owners must be explicitly known there. Existing frontend trust (SRV_TRUSTED_FRONTENDS) still applies, and frontend and backend allowlists still apply. Provider nodes must not be trusted frontends; backend configuration rejects overlap. Trust only the actual srv-front node, which authenticates the original peer and forwards its owner. As before, a trusted frontend is a security boundary that can assert identities.

The service: namespace is reserved: direct human profiles cannot claim it and SRV_ADMIN_USERS cannot contain it (startup fails). Service owners receive normal owner-scoped operations, never admin privileges or another user's instances. Multiple nodes mapped to the same name intentionally share ownership; use separate names for isolation. Dispatch supplies no Amp member ID, so this provides no per-human Amp ownership. Removing a mapping revokes that node's service identity after restart, but does not delete or reassign existing instances. Rotate node IDs explicitly; do not grant a tag-wide wildcard. Protect configuration as admin-owned.

Host-local project-template create NAME ID, project-template delete ID, and new NAME --template ID require explicit --host HOST through srv-front. Template creation and use must target the same backend; automatic template placement is rejected.

14.2 Tailscale credentials

Use either TS_AUTHKEY or TS_CLIENT_ID/TS_CLIENT_SECRET for the control-plane node. The OAuth flow is preferred for guest auth key minting.

14.3 Core settings

14.4 Guest artifacts

For example, /etc/srv/images.json can register Debian alongside the existing Arch defaults:

{  "debian-13": {    "kernel": "/var/lib/srv/images/debian-base/vmlinux",    "rootfs": "/var/lib/srv/images/debian-base/rootfs-base.img"  }}

Set SRV_IMAGES_FILE=/etc/srv/images.json in the service environment. Upgrade both srv and srv-vm-runner, then restart both services; an old runner rejects image-aware starts rather than silently using its global kernel. Paths must be absolute; initrd is optional. Selected kernels/initrds must resolve within SRV_IMAGES_DIR (except the exact legacy configured kernel/initrd pair). Keep this tree and its ancestors root-owned and not writable by the srv service or guests; do not store secrets in it. Symlinks escaping the tree are rejected. Every base rootfs must be reflink-cloneable into SRV_DATA_DIR. Names use lowercase letters, digits, dots, underscores and hyphens, starting with a letter or digit. Unknown names are rejected before creating a VM.

ssh srv new demo --image debian-13 selects Debian; omitting --image selects arch. The arch entry falls back to SRV_BASE_KERNEL, SRV_BASE_INITRD and SRV_BASE_ROOTFS unless explicitly defined in the catalog. These legacy variables are not needed when all required paths are supplied by catalog entries.

The selected image and boot paths are stored with the VM. Non-Arch guests keep their saved kernel/initrd paths across starts, even if the catalog changes or the entry is removed. Keep those artifacts available and use versioned paths when replacing kernels. Arch retains its existing behavior: current configured Arch boot paths are picked up on start/restart. No existing writable rootfs is replaced by changing the catalog. inspect and JSON responses expose image; older VM records are treated as arch.

Portable exports preserve the image name. To import a named image on another host, register that name with a compatible kernel/module build first; imports use the destination's boot paths. Older exports without an image name use Arch. Non-Arch exports use format version 2, so older servers reject them instead of importing them as Arch; Arch exports remain compatible with format version 1.

14.5 Guest defaults

User tag mappings are matched case-insensitively against the login returned by Tailscale WhoIs; callers cannot request arbitrary tags. For example:

SRV_GUEST_AUTH_TAGS=tag:microvmSRV_GUEST_USER_AUTH_TAGS=rene.michaelis@antares.net=tag:rene

VMs created by rene.michaelis@antares.net receive only tag:rene; unmapped creators continue to receive tag:microvm. Every configured tag must exist in the tailnet policy and be assignable by the OAuth client used by srv.

14.6 Networking

14.7 Authorization

14.8 Provider gateways

14.9 HTTP integrations

The SRV_SECRET_* entries are a naming convention rather than a fixed list. They are meant to live in /etc/srv/srv.env so the SSH control surface only ever receives secret names such as SRV_SECRET_OPENAI_PROD, not the raw credential values.

14.10 Alternate Tailscale endpoints

14.11 Misc

14.12 Path constraints

Several paths must share the same reflink-capable filesystem (typically btrfs or reflink-enabled xfs):

• SRV_DATA_DIR — host state directory
• SRV_BASE_ROOTFS — base guest image
• SRV_JAILER_BASE_DIR — jailer workspaces (hard-links log files into the jail)

Cross-filesystem hard-links fail, so keeping these on the same filesystem is required.

Host-side clients dial SRV_JAILER_BASE_DIR/<firecracker binary name>/<vm name>/root/firecracker.sock and SRV_DATA_DIR/instances/<vm name>/firecracker.sock (plus orb.vsock for Orbs) by absolute path, and Unix socket paths are limited to 107 bytes. With the binary named firecracker, the default SRV_DATA_DIR/jailer is too long for 60-character Orb names; SRV_JAILER_BASE_DIR must then be at most 12 bytes (9 bytes for 63-character names), for example /srvj on the root filesystem when it also holds SRV_DATA_DIR. The VM runner rejects a start whose socket paths are too long before launching anything. Choose the base before the first start: start receipts and lifecycle locks live under it, so moving it later abandons the records of existing attempts.

15. Guest image reference

The default guest image is an Arch Linux rootfs designed for srv. It is built by images/arch-base/build.sh and contains a full Linux userspace with developer tooling, Chromium, and the agent-browser automation CLI preinstalled.

15.1 Artifacts

The builder produces:

15.2 Included packages

The image is intentionally not minimal — it includes tooling for development and AI agent workflows:

• Docker, docker-compose with docker.socket activation instead of an idle dockerd
• Chromium and the pinned native agent-browser CLI with matching runtime skill data; /root/.agent-browser/config.json selects /usr/bin/chromium
• Go, gopls
• Odin, odinfmt, OLS
• Neovim with prewarmed LazyVim (BMW heritage amber theme)
• OpenCode and Pi CLIs with per-VM provider gateway bootstrap
• Git, fd, ripgrep, tree-sitter-cli, gcc, perf, valgrind
• iptables-nft with IPv4/IPv6 nftables support
• Kernel module tree matching the custom kernel (overlay, br_netfilter, Docker-related modules)

15.3 Bootstrap service

The guest includes srv-bootstrap.service, which runs on every boot:

1. Discovers the primary virtio interface from the kernel-provided default route
2. Adds a route to Firecracker MMDS at 169.254.169.254/32
3. Reads the MMDS payload from http://169.254.169.254/ with Accept: application/json
4. Sets the hostname from srv.hostname
5. Starts tailscaled
6. Runs tailscale up --auth-key=... --hostname=... --ssh on the first authenticated boot (relies on persisted state on later boots)
7. Writes /root/.config/opencode/opencode.json plus Pi config under /root/.pi/agent/ to the per-VM host provider gateways when provider API keys are configured, or removes those managed defaults when all provider gateways are disabled
8. Writes /var/lib/srv/bootstrap.done with the latest successful bootstrap timestamp

The --ssh flag on tailscale up is intentional — it enables Tailscale SSH so the control plane's connect: ssh root@<name> output works through the tailnet without per-user OpenSSH keys.

Docker is installed but not started during boot. The image enables docker.socket, so the first Docker CLI/API use starts dockerd and containerd on demand. Getty/serial-getty and udev units are masked to avoid idle daemons that are not needed for the Firecracker bootstrap path.

15.4 Kernel details

• Starts from Firecracker's microvm-kernel-ci-x86_64-6.1.config and runs olddefconfig against the selected source tree
• Enables CONFIG_PCI=y for ACPI initialization (required by current Firecracker x86 builds)
• Disables CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES so the kernel prefers ACPI discovery
• Enables Landlock and adds it to CONFIG_LSM (keeps pacman's download sandbox working)
• Enables virtio ballooning and page reporting so fixed and pooled VMs can return idle guest pages to the host
• Builds real .ko module files for Docker, overlay, br_netfilter, and nftables

15.5 DNS

Guest bootstrap configures systemd-resolved with the nameservers supplied through the kernel ip= boot parameter in /proc/net/pnp. /etc/resolv.conf points at the systemd-resolved stub, allowing Tailscale to install MagicDNS and split-DNS routes after joining the tailnet.

15.6 Logging

journald is configured to forward logs to ttyS0, making the guest bootstrap flow visible in each instance's serial log (ssh srv logs <name> serial).

16. Operations runbook

This runbook is for the supported prepared-host path: systemd-managed srv, srv-net-helper, and srv-vm-runner on a Linux host with cgroup v2, /dev/kvm, SRV_DATA_DIR on a reflink-capable filesystem such as btrfs or reflink-enabled xfs, SRV_BASE_ROOTFS on the same filesystem, and the official static Firecracker/jailer release pair.

Same-host reboot recovery is already built into the control plane: when srv comes back under systemd, previously active instances are restarted automatically. The steps below cover the larger operator workflows that were previously only implied.

Hybrid memory pools are now part of the supported runtime model:

• Fixed memory remains the default and preserves the original dedicated-reservation behavior.
• Pools reserve host memory immediately at pool create time.
• Pooled member VMs consume guest-visible RAM from that shared reservation and are not charged against host memory a second time.
• Pooled guest RAM changes are stopped-only and must stay within the current pool's reserved size.

16.1 Supported Upgrade Lanes

• Guest-local maintenance is still guest-local. Running pacman -Syu inside a guest is supported when you want that guest to drift independently, but it is not the control plane's golden-image rollout path.
• Kernel roll-forward is a host-managed lane. Existing stopped guests pick up the current SRV_BASE_KERNEL and optional SRV_BASE_INITRD on their next start or restart.
• Rootfs golden-image rollout is a new-guest lane. Updating SRV_BASE_ROOTFS changes what future new clones from, but it does not rewrite existing guests' writable disks.
• Schema rollout is tied to the control-plane binary. SQLite migrations run during srv startup and are currently additive; rollback means restoring the pre-upgrade backup together with the previous binary set.

16.2 Backup

Take backups from a quiesced host so SRV_DATA_DIR and the SQLite WAL state are self-consistent.

Include SRV_JAILER_BASE_DIR/.start-receipts in the same backup and restore it with the database. These runner-owned launch records and tombstones fence delayed requests. They can live outside SRV_DATA_DIR; a data-directory snapshot alone does not capture them. Do not delete receipts to unblock a quarantined launch. For the archive command below, add the configured receipt directory as a path relative to / if it is not already covered by var/lib/srv.

If you need a fast host-local point-in-time copy without shutting the control plane down first, use the built-in snapshot barrier instead:

ssh srv snapshot create

That command is admin-only. It briefly rejects every other SSH command, waits for already admitted commands to finish, checkpoints SQLite, flushes the filesystem, and then creates a readonly btrfs snapshot of SRV_DATA_DIR under SRV_DATA_DIR/.snapshots/<timestamp>.

Snapshot semantics are intentionally limited and explicit:

• control-plane consistent
• stopped guests fully safe
• running guests crash-consistent

Important caveats for this path:

• SRV_DATA_DIR itself must be a btrfs subvolume root. A plain directory on btrfs is not enough.
• The app snapshots SRV_DATA_DIR only. /etc/srv, environment files, and unit overrides still need the existing operator-managed backup flow below.
• Remote btrfs send/receive replication is intentionally out of the barrier path. If you use it for DR or warm standby, run it after the local snapshot already exists.

1. Stop the services.

sudo systemctl stop srv srv-net-helper srv-vm-runner

2. Capture the state directory, environment file, and any unit overrides.

sudo tar --xattrs --acls --numeric-owner \  --ignore-failed-read \  -C / \  -czf /var/tmp/srv-backup-$(date -u +%Y%m%dT%H%M%SZ).tar.gz \  etc/srv \  var/lib/srv \  etc/systemd/system/srv.service.d \  etc/systemd/system/srv-net-helper.service.d \  etc/systemd/system/srv-vm-runner.service.d \  etc/systemd/system.control

3. Start the services again if this was only a backup window.

sudo systemctl start srv-vm-runner srv-net-helper srv

Notes:

• Preserve the configured paths in /etc/srv/srv.env. Instance rows store absolute runtime paths such as SRV_DATA_DIR/instances/<name>/rootfs.img, so changing SRV_DATA_DIR, SRV_JAILER_BASE_DIR, or the base artifact paths during restore is not a supported relocation workflow.
• Keep SRV_JAILER_BASE_DIR on the same filesystem as SRV_DATA_DIR; the runner hard-links log files into the jail and cross-filesystem links fail.

16.3 Move One Stopped VM Between Hosts

For single-VM cutover, use the portable stopped-VM stream instead of copying SQLite rows or the whole host state directory:

ssh srv-a export demo | ssh srv-b import

Operational notes:

• The source VM must be stopped before export.
• Import recreates the VM under the same name and leaves it stopped. Start it explicitly on the destination after the stream completes.
• The artifact preserves portable metadata such as name, creator, machine shape, rootfs size, and last-known Tailscale name or IP.
• Serial and Firecracker logs are included when present, but each is capped to the newest 256 MiB during export.
• Import regenerates destination-local runtime state such as absolute file paths, tap device wiring, guest MAC, and VM subnet allocation.
• The copied rootfs carries the guest's durable Tailscale identity, so do not boot the source and destination copies at the same time.
• The destination host uses its currently configured SRV_BASE_KERNEL and optional SRV_BASE_INITRD on the first later start; only the writable guest disk and optional logs come from the streamed artifact.

If the exported VM was in pooled mode, import preserves that metadata. The destination host must already have the referenced memory pool state available through the restored control-plane database before the VM can be started successfully.

16.4 Restore Or Rebuild A Host

1. Prepare a fresh host with the normal prerequisites: Tailscale, cgroup v2, /dev/kvm, reflink-capable storage such as btrfs or reflink-enabled xfs shared by SRV_DATA_DIR and SRV_BASE_ROOTFS, and the repo checkout.
2. Reinstall the managed assets.

sudo ./contrib/systemd/install.sh

3. Re-enable IPv4 forwarding for guest NAT on the rebuilt host.

sudo tee /etc/sysctl.d/90-srv-ip-forward.conf >/dev/null <<'EOF'net.ipv4.ip_forward = 1EOFsudo sysctl --system

4. Restore /etc/srv/srv.env from backup and verify that SRV_FIRECRACKER_BIN, SRV_JAILER_BIN, SRV_DATA_DIR, SRV_BASE_KERNEL, SRV_BASE_ROOTFS, and any optional SRV_BASE_INITRD still point at the intended paths.
5. Restore the saved SRV_DATA_DIR tree to the same path.
6. Reload systemd and start the services.

sudo systemctl daemon-reloadsudo systemctl enable --now srv-vm-runner srv-net-helper srv

7. Run the prepared-host validation gate before handing the host back to users.

sudo ./contrib/smoke/host-smoke.sh

That smoke pass is part of the supported restore workflow now, not an optional extra.

16.5 Upgrade And Rollback

16.5.1 Control Plane And Schema

The launch-outcome protocol requires matching srv and srv-vm-runner binaries. Do not run the new control plane with an old helper or mix lifecycle writers. Stopping the runner can stop its guests: use an approved maintenance window.

Cold enrollment also requires a rebuilt golden rootfs containing the updated srv-bootstrap script. It reports the guest's stable Tailscale node ID with a per-launch nonce over the serial channel. An old base image will leave new guests pending rather than accepting an unrelated same-name Tailscale record. Existing enrolled guests do not need an in-place rootfs update for warm starts.

1. Take the quiesced backup above before starting the upgraded srv binary for the first time.
2. Build and install the new srv, srv-net-helper, and srv-vm-runner binaries.
3. If you also refreshed Firecracker, use the matching official static firecracker and jailer pair and verify /etc/srv/srv.env still points at those paths.
4. Restart the services.

sudo systemctl stop srv srv-net-helper srv-vm-runnersleep 5sudo systemctl start srv-vm-runner srv-net-helper srv

5. Run the host smoke test.

sudo ./contrib/smoke/host-smoke.sh

Rollback for control-plane or schema regressions is restore-based:

If rolling back from a version that records stopping, first finish pending stops with ssh srv stop <name> under that version. Older versions do not recognize the state and may automatically start those guests. Never mix old and new control-plane writers against the same database. Swapping only the binary is not a substitute for the restore-based procedure below.

1. Stop the services.
2. Reinstall the previous binaries and previous static Firecracker/jailer pair if those changed.
3. Restore the pre-upgrade backup of /etc/srv, SRV_DATA_DIR, and the matching runner receipt directory. Confirm no post-backup guest remains running before replacing state; never combine old database rows with newer runtime state.
4. Restart the services.
5. Run the same host smoke test again.

16.5.2 Kernel Rollout For Existing Guests

1. Rebuild the kernel artifact under images/arch-base/.
2. Update SRV_BASE_KERNEL and optional SRV_BASE_INITRD in /etc/srv/srv.env.
3. Restart the units if needed so the runner sees the new base paths.
4. Stop and start guests one at a time, or let already stopped guests pick up the new boot artifacts on their next start.
5. Use a canary guest first, then roll wider once it passes the workload check you care about.

Rollback is just pointing SRV_BASE_KERNEL or SRV_BASE_INITRD back to the previous artifact and restarting the affected guests again.

16.5.3 Golden Rootfs Rollout

1. Rebuild rootfs-base.img under images/arch-base/.
2. Point SRV_BASE_ROOTFS at the new image.
3. Create a canary guest with ssh srv new <name> and validate it.
4. After the canary passes, new guests will clone from the new base image.

Rollback for the golden rootfs lane is also path-based: point SRV_BASE_ROOTFS back to the previous image before creating more guests.

Important caveat: existing guests keep their own writable rootfs.img. There is no host-driven in-place existing-rootfs conversion workflow yet. For an existing guest that needs OS updates today, the supported choices are:

• manage that guest locally with pacman -Syu, accepting guest-local drift
• create a replacement guest from the refreshed golden image and migrate the workload or data to it

16.6 Host Hardening And Caveats

• cgroup v2 is required. The runner depends on a delegated cgroup v2 subtree for both fixed and pooled layouts:
• Fixed VMs use firecracker-vms/<name> leaves with enforced cpu.max, memory.max, memory.swap.max, and pids.max. The guest sees the configured RAM; the host memory.max adds a small Firecracker overhead reserve (max(256 MiB, RAM/16)) so VMM/page-table overhead does not consume guest-visible memory headroom.
• Pooled VMs use firecracker-pools/<pool>/<name> leaves. The pool parent holds the hard memory reservation with memory.max=<pool size> and memory.swap.max=0, while child VM cgroups still carry CPU and PID limits.
• IPv4 forwarding must stay enabled on the host. Guest egress depends on forwarding packets from each TAP device through the host's outbound interface after the helper installs MASQUERADE and FORWARD rules.
• srv-vm-runner.service must keep User=root, Group=srv, Delegate=cpu memory pids, DelegateSubgroup=supervisor, and a group-accessible socket under /run/srv-vm-runner/.
• Do not add NoNewPrivileges=yes to srv-vm-runner.service; the jailer must drop privileges and exec Firecracker on real hosts.
• Keep using the official static Firecracker and jailer release pairing. Distro-provided dynamically linked binaries can fail after chroot before the API socket appears.
• Preserve /etc/srv/srv.env across reinstall or upgrade unless you are intentionally changing configuration and have accounted for the stored absolute paths.
• Keep SRV_DATA_DIR and SRV_BASE_ROOTFS on the same reflink-capable filesystem, such as btrfs or reflink-enabled xfs. Fast per-instance provisioning still depends on reflink cloning the configured base rootfs.
• Use pooled mode only for workloads that tolerate an elastic memory contract. Fixed mode remains the supported choice for databases, JVMs, and other memory-sensitive services.
• status is now the operator view for host memory accounting: it separates fixed reservations from pool reservations so pooled members do not look like additional dedicated host claims.
• inspect and top are the operator views for VM memory behavior: inspect shows memory-mode and host reservation style, while top shows host-resident live memory against configured guest RAM. Fixed and pooled VMs both use a virtio balloon with free-page reporting so idle guest pages can be returned to the host; pooled rows also show the pool name.
• On reboot recovery, pooled VMs recreate their pool-parent cgroups before restart; if a recovered VM references a missing pool record, treat that as a control-plane state restore problem rather than a guest runtime issue.
• Run sudo ./contrib/smoke/host-smoke.sh after install, restore, control-plane upgrade, and base-image changes.

17. Troubleshooting

17.1 VM is stuck in provisioning or failed

Check the control-plane view first:

ssh srv inspect 

Look for the last_error field and state. Then check the guest and VMM logs:

ssh srv logs  serialssh srv logs  firecracker

Common causes:

• Missing or misconfigured base rootfs/kernel
• Tailscale OAuth credentials expired or invalid
• Network helper not running (systemctl status srv-net-helper)
• VM runner not running (systemctl status srv-vm-runner)
• /dev/kvm not accessible

17.2 Running VM remains awaiting-tailnet after a start timeout

A disconnected caller or readiness timeout can leave a running VM in awaiting-tailnet. Once the VM has booted, srv preserves it rather than stopping it because the caller disconnected. The background reconciler retries readiness automatically. Check last_error and the serial log; for an immediate retry:

ssh srv start 

For an awaiting-tailnet VM whose recorded process is still running, this checks the existing runtime and tailnet peer without rebooting or re-enrolling it. srv matches the API socket to the recorded process, checks the enrolled node against the local tailnet peer map, and sends a bounded TSMP probe. An unchanged Tailscale LastSeen, an idle peer with InEngine=false, or a zero reported RTT does not prevent success. Missing or ambiguous identity and failed probes leave the running VM pending. If the process has exited, start retains its normal behavior of starting the VM.

Warm-start ready means the guest responded on the tailnet. It does not prove SSH access or application health; test those separately. Allow the SSH caller more time than SRV_GUEST_READY_TIMEOUT (default 2m), plus transport overhead. list and inspect can wait behind an in-progress lifecycle operation because their runtime cleanup uses the same instance lock.

After startup recovery, a background sweep runs every 30 seconds. Busy instances are skipped until a later sweep; snapshots exclude background mutations. Readiness and outcome lookup get a 10-second attempt budget per instance. Shutdown and network cleanup retain their own bounded completion deadlines. Slow sweeps can delay later checks; 30 seconds is not a recovery-time guarantee.

The sweep resumes stopping and deleting intent and checks pending readiness. It does not start stopped guests, restart crashed guests, or monitor application health. Startup recovery remains the separate path for reboot-time auto-start.

Do not repair this state with raw SQL or restart shared services just to refresh it. Deployments and rollback still require the maintenance and backup procedure in the operations reference.

17.3 Stop was interrupted

stopping records an unfinished stop. The PID and resource reservations remain until the VM runner and network cleanup succeed. The background sweep retries the stop; ssh srv stop <name> also retries immediately. Likewise, deleting resumes an already requested deletion after failures. Backup/restore refuses instances with unfinished shutdown cleanup; neither start nor startup recovery may override this state, even if the process has already exited.

Before downgrading to a version without stopping, finish all pending stops under the new version. Older versions can treat this unknown state as eligible for automatic startup. Do not run old and new control-plane writers together. This is an additional rollback constraint, not a replacement for the documented maintenance and backup procedure.

17.4 Launch outcome is unknown

Each new launch has a durable request ID in SQLite and a runner receipt under SRV_JAILER_BASE_DIR/.start-receipts. After a lost HTTP response, srv asks the runner about that same attempt. It adopts a verified running VM, or cleans up only after the runner confirms absence. Unknown outcomes preserve disks, networking, and reservations; another start cannot launch a duplicate.

If the helper dies between accepting a launch and recording its process identity, the attempt remains quarantined for that host boot. Restoring helper connectivity can recover committed launches, but does not make an unverified launch safe. Inspect the runner logs and runtime before planning recovery. Do not clear the receipt, edit the PID in SQLite, or restart the host without a maintenance plan. A host reboot makes old-boot attempts recoverable only after the runner verifies that no conflicting runtime occupies the name.

Cold guests need a serial srv-enrollment report from the updated base image. Missing reports, wrong nonces, and mismatched node IDs leave them pending even if a same-name device responds. Warm starts require a stored node ID or a unique existing API device; a missing legacy device must be investigated before retrying. Deleting a legacy VM without a stored node ID leaves its Tailscale device for manual identification and cleanup rather than deleting a potentially unrelated same-name device.

17.5 VM boots but Tailscale SSH doesn't work

If ssh srv inspect <name> shows state: ready and a Tailscale IP but you can't SSH in:

1. Check the serial log for bootstrap errors: ssh srv logs <name> serial
2. Look for srv-bootstrap.service failures
3. Verify that SRV_GUEST_AUTH_TAGS matches a tag your OAuth client can mint keys for
4. Verify that the guest can reach the Tailscale coordination server (check DNS and outbound connectivity from inside the VM if possible)

17.6 Guest can't reach the internet

Guest egress depends on IPv4 forwarding and MASQUERADE rules:

# Check forwardingsysctl net.ipv4.ip_forward# Check iptables rulessudo iptables -t nat -L -nsudo iptables -L FORWARD -n# Check the network helpersudo systemctl status srv-net-helper

If net.ipv4.ip_forward is 0, re-enable it:

sudo sysctl -w net.ipv4.ip_forward=1

17.7 VM runs out of disk space

Check from inside the VM:

df -h

If the rootfs is full, you can resize the VM:

ssh srv stop ssh srv resize  --rootfs-size 20Gssh srv start 

On the host, check SRV_DATA_DIR usage:

df -h /var/lib/srv

17.8 Check cgroup limits

If a VM is being throttled or OOM-killed, check its cgroup v2 limits:

cat /sys/fs/cgroup/firecracker-vms//cpu.maxcat /sys/fs/cgroup/firecracker-vms//memory.maxcat /sys/fs/cgroup/firecracker-vms//memory.swap.maxcat /sys/fs/cgroup/firecracker-vms//pids.max

17.9 Firecracker or jailer errors

The VM runner logs to journald:

sudo journalctl -u srv-vm-runner -f

Common issues:

• Jailer chroot setup failures — check that SRV_JAILER_BASE_DIR is on the same filesystem as SRV_DATA_DIR
• Permission errors — verify srv-vm-runner.service keeps User=root, Group=srv, Delegate=cpu memory pids, DelegateSubgroup=supervisor, and no NoNewPrivileges=yes
• /dev/kvm not available — check permissions and that KVM is loaded

17.10 Smoke test

The end-to-end smoke test validates the full host setup:

sudo ./contrib/smoke/host-smoke.sh

Overrides:

On failure, artifacts are written to /var/tmp/srv-smoke/<instance>/ including inspect, logs, systemctl status, and journalctl output.

18. Backup and restore

srv provides in-place backup and restore for VMs. This is useful for checkpointing VMs before risky changes and rolling them back when something goes wrong.

18.1 Create a backup

ssh srv backup create demo

This copies the current rootfs image into the backup store under SRV_DATA_DIR/backups/demo/<backup-id>/.

Stopping the VM first is strongly recommended. If you back up a running VM, the resulting rootfs is crash-consistent only — like pulling the power cord on a running machine — and the guest filesystem may need journal replay on restore. A crash-consistent backup is still better than no backup at all.

18.2 List backups

ssh srv backup list demo

Each backup has a unique ID and timestamp.

18.3 Delete a backup

ssh srv backup delete demo 

This permanently removes the selected backup. The VM may be running or stopped.

18.4 Restore from a backup

ssh srv restore demo 

This replaces the current rootfs with the backup's rootfs. The VM must be stopped.

18.5 Full workflow

# Create and configure a VMssh srv new demossh root@demo  # install packages, configure services# Checkpoint before risky changesssh srv stop demossh srv backup create demossh srv start demo# ... make changes that break things ...# Reset to the checkpointssh srv stop demossh srv backup list demossh srv restore demo ssh srv start demo# Verify the restore worked — the VM should be back to the checkpoint state

18.6 Constraints

• Restore requires stopped: restoring onto a running VM is not supported
• Stopped recommended for backup: backing up a stopped VM produces a fully safe, clean-filesystem snapshot. Backups of running VMs are crash-consistent only
• In-place only: backups are tied to the original VM record. They cannot be restored onto a newly created VM that reuses the same name
• Single host: backups live on the same host. For cross-host migration, use export/import
• Rootfs only: backups capture the writable rootfs. The kernel and initrd come from the host's current configuration on the next start

18.7 Backup storage

Backups are stored under:

SRV_DATA_DIR/backups///

Monitor disk usage if you create many backups — each one is a full copy of the rootfs at that point in time.

19. Build a custom guest image

The default guest image is an Arch Linux rootfs with Docker, Go, Neovim, OpenCode, Pi, and common development tools. You can customize it by modifying the overlay or building a completely different rootfs.

For a basic Debian 13 guest without the Arch developer/agent tools, use the Debian builder. It reuses the custom Firecracker kernel and matching modules, plus the existing srv bootstrap. Register its artifacts in the image catalog and select it with ssh srv new demo --image debian-13.

19.1 Default image builder

The images/arch-base/ directory contains the official builder:

sudo OUTPUT_DIR=/var/lib/srv/images/arch-base ./images/arch-base/build.sh

This produces two artifacts:

• vmlinux — an x86_64 Firecracker-compatible kernel built from the 6.12 LTS source
• rootfs-base.img — a sparse ext4 image populated via pacstrap

19.1.1 What the image includes

• Docker, docker-compose
• Go, gopls, Odin, odinfmt, OLS
• Neovim with a prewarmed LazyVim config (BMW heritage amber theme)
• OpenCode and Pi CLIs with per-VM provider gateway bootstrap
• Git, fd, ripgrep, tree-sitter-cli, gcc, perf, valgrind
• iptables-nft with IPv4/IPv6 nftables support
• srv-bootstrap.service for Tailscale and MMDS setup

19.1.2 Build overrides

# Change kernel versionsudo KERNEL_VERSION=6.1.167 OUTPUT_DIR=/var/lib/srv/images/arch-base ./images/arch-base/build.sh# Change rootfs size (default 10G)ROOTFS_SIZE=20G sudo OUTPUT_DIR=/var/lib/srv/images/arch-base ./images/arch-base/build.sh# Reduce kernel build parallelismsudo KERNEL_JOBS=2 OUTPUT_DIR=/var/lib/srv/images/arch-base ./images/arch-base/build.sh

19.2 Podman build on non-Arch hosts

If your host is not Arch Linux and does not provide pacstrap:

sudo podman run --rm --privileged --network host \  -v "$PWD":/work \  -v /var/lib/srv/images/arch-base:/var/lib/srv/images/arch-base \  -w /work \  docker.io/library/archlinux:latest \  bash -lc '    set -euo pipefail    pacman -Sy --noconfirm archlinux-keyring    pacman -Syu --noconfirm arch-install-scripts base-devel bc e2fsprogs rsync curl systemd    OUTPUT_DIR=/var/lib/srv/images/arch-base ./images/arch-base/build.sh  '

• --privileged is required because the builder uses losetup, mkfs.ext4, and mount
• --network host keeps mirror and kernel downloads simple

19.3 Customizing the overlay

The images/arch-base/overlay/ directory contains files that are copied into the guest rootfs during the build. Changes here only affect new guests after you rebuild rootfs-base.img and refresh SRV_BASE_ROOTFS.

After modifying the overlay:

sudo OUTPUT_DIR=/var/lib/srv/images/arch-base ./images/arch-base/build.sh# Then update SRV_BASE_ROOTFS in /etc/srv/srv.env to point at the new image

19.4 Rolling out a new image

Rootfs changes only affect newly created guests. Existing guests keep their own writable rootfs.img. There is no host-driven in-place rootfs update.

To migrate an existing guest to a new base image:

1. Rebuild and update SRV_BASE_ROOTFS
2. Create a new guest with ssh srv new <name>
3. Migrate workload data to the new guest
4. Delete the old guest

Or manage the existing guest locally with pacman -Syu, accepting guest-local drift.

19.5 Kernel rollout

Kernel roll-forward is simpler. Existing stopped guests pick up the currently configured SRV_BASE_KERNEL and optional SRV_BASE_INITRD on their next start or restart:

1. Rebuild the kernel artifact
2. Update SRV_BASE_KERNEL (and SRV_BASE_INITRD if applicable) in /etc/srv/srv.env
3. Restart the units if needed
4. Stop and start guests, or let stopped guests pick up the new kernel on their next start

Rollback: point the path back to the previous artifact and restart guests again.

20. Export and import

The export and import commands stream a portable VM artifact between hosts. This is the supported way to move a VM from one srv host to another.

20.1 Export

On the source host:

ssh srv stop demossh srv export demo

The command writes a tar stream to stdout. The artifact contains:

• A versioned manifest
• The writable rootfs.img
• Serial and Firecracker logs when present (each capped to the newest 256 MiB)

20.2 Import

Pipe the export stream directly into the destination host:

ssh srv-a export demo | ssh srv-b import

Import creates the VM under the same name on the destination host, allocates new runtime paths and network state, and leaves the VM stopped.

20.3 Start after import

ssh srv-b start demo

The destination host uses its currently configured SRV_BASE_KERNEL and optional SRV_BASE_INITRD — only the writable disk and optional logs come from the streamed artifact.

20.4 Important semantics

Because the guest's durable Tailscale identity lives in the copied rootfs:

• Do not boot the source and destination VMs at the same time — this would cause a Tailscale key conflict
• Treat export/import as cutover or move semantics, not cloning semantics

The destination host regenerates:

• Absolute file paths (runtime directories)
• TAP device wiring
• Guest MAC address
• VM /30 subnet allocation

The artifact preserves:

• VM name
• Creator identity
• Machine shape (vCPUs, memory, rootfs size)
• Last-known Tailscale name and IP (as cached state)

20.5 Save to a file

You can also save the export to a file for later or offline transfer:

ssh srv export demo > demo-backup.tar

Then on the destination:

cat demo-backup.tar | ssh srv import

21. Host snapshots

The snapshot create command is a host-level disaster-recovery primitive that creates a consistent point-in-time copy of SRV_DATA_DIR.

21.1 How it works

ssh srv snapshot create

This is an admin-only command that:

1. Briefly rejects all other SSH commands
2. Waits for already admitted commands to finish
3. Checkpoints SQLite
4. Flushes the filesystem
5. Creates a readonly btrfs snapshot of SRV_DATA_DIR under SRV_DATA_DIR/.snapshots/<timestamp>

21.2 Snapshot semantics

The snapshot is intentionally simple:

• Control-plane consistent: SQLite state is checkpointed
• Stopped guests fully safe: rootfs data is on disk and consistent
• Running guests crash-consistent only: like pulling the power on a running VM — the filesystem may need journal replay on restore

21.3 Prerequisites

• SRV_DATA_DIR must be a btrfs subvolume root — a plain directory on btrfs is not enough
• The snapshot covers SRV_DATA_DIR only. /etc/srv, environment files, and unit overrides still need your existing operator-managed backup flow

21.4 Use with btrfs send/receive

You can combine snapshots with btrfs send/receive for off-host replication:

# After creating a snapshotsudo btrfs send SRV_DATA_DIR/.snapshots/ | \  ssh backup-host btrfs receive /backup/srv/

Run btrfs send/receive after the local snapshot already exists — the snapshot barrier is not involved in the replication step.

21.5 Not included

• /etc/srv/srv.env — back this up separately
• Systemd unit overrides in /etc/systemd/system/srv*.service.d/
• Firecracker and jailer binaries

See the operations reference for the full host backup and restore workflow.

22. Instance lifecycle

This page covers the full lifecycle of a srv VM from creation to deletion.

22.1 Create

ssh srv new 

Arch is the default. To select a configured image at creation:

ssh srv new debian-vm --image debian-13

The image stays with the VM; start does not change its operating system. See image configuration for the host-side catalog and the Debian builder for the basic Debian 13 rootfs without Arch's optional developer/agent tools.

With custom sizing:

ssh srv new  --cpus  --ram  --rootfs-size 

--ram and --rootfs-size accept units like 2G, 512M, or plain MiB integers. --cpus must be 1 or an even number, up to 32.

The control plane:

1. Clones the base rootfs as a reflink
2. Allocates a /30 network subnet and TAP device
3. Mints a one-off Tailscale auth key via MMDS
4. Boots the VM through the jailer with cgroup v2 enforcement
5. Polls until the guest reports state: ready

22.2 Inspect

ssh srv inspect 

Shows instance state, vCPU count, memory, rootfs size, network addresses, Tailscale name and IP, and event history.

Machine-readable:

ssh srv -- --json inspect 

22.3 List and status

ssh srv listssh srv status

list shows the VMs visible to the caller: all VMs for admins, or only the caller's own VMs for regular users. status is admin-only and reports host capacity — instance counts plus CPU, memory, and disk headroom.

The storage rows answer two different questions:

• DISK RSV is logical rootfs reservation headroom. This is the budget used to admit new, start, and rootfs-growing resize operations.
• FS SPACE is the filesystem's actual used and available space. It also reflects backups, reflink divergence, logs, and other files outside rootfs reservations; shared reflink extents count only once until they diverge.

Use FS SPACE to detect physical storage pressure even when the logical reservation budget still has room.

22.4 Logs

ssh srv logs ssh srv logs  serialssh srv logs  firecrackerssh srv logs -f  serialssh srv logs -f  firecracker

The serial log is append-only across boots. The Firecracker log is reset when the VMM starts so it only contains the current Firecracker process's output.

22.5 Stop

ssh srv stop 

Performs a graceful shutdown through Firecracker. The rootfs is preserved on disk.

22.6 Start

ssh srv start 

Boots a previously stopped VM. Stopped guests pick up the currently configured SRV_BASE_KERNEL and optional SRV_BASE_INITRD on their next start.

22.7 Restart

ssh srv restart 

Stops and starts the VM in one command. Also picks up the current kernel and initrd.

22.8 Check the root filesystem

ssh srv stop ssh srv fsck  && ssh srv start 

fsck runs e2fsck -f -p against the stopped VM's ext4 rootfs. It forces a full check and automatically repairs problems that can be fixed safely. Fatal or uncorrected filesystem errors make the command fail and leave the VM stopped.

22.9 Delete

ssh srv delete 

Removes the VM's rootfs, runtime directory, TAP device, cgroup, and jailer workspace. This is irreversible.

22.10 Warm boot behavior

When a VM that already has tailscaled state reboots (via start or restart), it reuses its existing Tailscale identity instead of minting a new auth key. This is called a warm boot.

On cold boot (first new), the guest bootstrap service reads the Tailscale auth key from MMDS and runs tailscale up --auth-key=... --ssh exactly once.

23. Resize a VM

Resize lets you change the vCPU count, memory, and rootfs size of a stopped VM.

23.1 Prerequisites

• The VM must be stopped — resize is rejected for running VMs
• vCPUs and memory can be increased or decreased as long as the requested values stay within the supported limits
• Rootfs size is grow-only — shrink requests are rejected
• The srv host must have both e2fsck and resize2fs available for a rootfs grow (and when creating a VM with a custom rootfs size)

23.2 Resize

ssh srv stop demossh srv resize demo --cpus 4 --ram 8G --rootfs-size 20Gssh srv start demo

You can specify any combination of flags. Omitted flags keep the current value:

# Only increase RAMssh srv stop demossh srv resize demo --ram 16Gssh srv start demo

23.3 How it works

• vCPU count: stored in the instance record and applied on the next boot
• Memory: stored in the instance record and applied on the next boot
• Rootfs size: runs e2fsck -f -p against the stopped ext4 image, expands the underlying file, and then uses resize2fs to grow the filesystem. Fatal or uncorrected filesystem errors stop the resize before the image is expanded. This operation only increases the filesystem — it never shrinks

If an earlier attempt expanded the image file but did not finish growing the filesystem, retrying the same resize request checks the filesystem and resumes the pending filesystem grow.

23.4 Limits

vCPU count must be 1 or an even number. CPU and memory changes are applied on the next boot.

24. View logs

srv provides two log sources for each VM: serial output and Firecracker VMM logs.

24.1 Serial log

Shows kernel boot, srv-bootstrap.service, tailscaled, and general guest console output:

ssh srv logs demossh srv logs demo serialssh srv logs -f demo serial

24.2 Firecracker log

Shows VMM lifecycle events, API requests, and microVM-level errors:

ssh srv logs demo firecrackerssh srv logs -f demo firecracker

24.3 Both logs at once (default)

Without a log source argument, ssh srv logs <name> shows the serial log by default.

24.4 Where logs live on the host

The serial log is append-only across boots. The Firecracker log is reset when the VMM starts so it only contains the current Firecracker process's output.

24.5 Systemd logs

The VM runner and network helper also log to journald:

sudo journalctl -u srv-vm-runner -fsudo journalctl -u srv-net-helper -fsudo journalctl -u srv -f

24.6 Debugging a failed VM

When a VM is stuck or has failed:

1. ssh srv inspect <name> — control-plane view, state, and recorded events
2. ssh srv logs <name> serial — guest boot and bootstrap errors
3. ssh srv logs <name> firecracker — VMM errors
4. journalctl -u srv-vm-runner --no-pager — jailer and stop-time cleanup failures
5. Check cgroup limits: cat /sys/fs/cgroup/firecracker-vms/<name>/memory.max

25. Dev/test environment

srv's reflink-based cloning makes it fast to spin up a VM from a base image, modify it, and then use backup/restore to reset it to a known-good state.

25.1 Create a dev VM

ssh srv new dev --cpus 4 --ram 8G --rootfs-size 30G

25.2 Install your toolchain

ssh root@dev# Inside the VMpacman -Syupacman -S nodejs npm# ... set up your project ...

25.3 Back up the clean state

Once the VM is configured the way you like, take a checkpoint before making risky changes:

ssh srv stop devssh srv backup create devssh srv start dev

25.4 Wipe and reset

After a bad experiment:

ssh srv stop devssh srv backup list devssh srv restore dev ssh srv start dev

The restore rolls the rootfs back to the exact state captured at backup time. This is fast because it replaces the writable disk image rather than copying file by file.

25.5 Repeat

You can create multiple backups at different points:

ssh srv stop devssh srv backup create dev    # backup 1: cleanssh srv start dev# ... make changes ...ssh srv stop devssh srv backup create dev    # backup 2: with toolchainssh srv start dev

Then restore whichever snapshot you need:

ssh srv stop devssh srv restore dev ssh srv start dev

25.6 Key constraints

• Backups and restores only work on stopped VMs
• Backups are tied to the original VM record — they cannot be restored onto a differently created VM even if the name is reused
• Restore is in-place: it overwrites the VM's current rootfs with the backup's rootfs

26. Isolated service

Each srv VM runs in its own cgroup v2 leaf with its own /30 network subnet, TAP device, and Tailscale identity. This makes it straightforward to run services that need full isolation — container runtimes, network services, databases — without sharing the host's PID, network, or filesystem namespace.

26.1 Run a database

ssh srv new db --cpus 2 --ram 4G --rootfs-size 20G

Once the VM is ready:

ssh root@dbpacman -S postgresql# ... configure and start postgresql ...

The database is now reachable at its Tailscale IP from any other machine on the tailnet.

26.2 Run Docker workloads

The guest image includes Docker and docker-compose:

ssh srv new builder --cpus 4 --ram 8G --rootfs-size 40G

ssh root@builderdocker run ...docker compose up -d

The overlay and br_netfilter kernel modules are preloaded, and nftables supports both IPv4 and IPv6 families.

26.3 Per-VM resource isolation

Each VM is enforced by cgroup v2:

You can verify the live cgroup limits:

cat /sys/fs/cgroup/firecracker-vms//cpu.maxcat /sys/fs/cgroup/firecracker-vms//memory.maxcat /sys/fs/cgroup/firecracker-vms//memory.swap.maxcat /sys/fs/cgroup/firecracker-vms//pids.max

26.4 Per-VM networking

Each VM gets its own /30 subnet, TAP device, and NAT rules. VMs cannot reach each other's private networks. They can reach the host and the internet through MASQUERADE rules.

From any tailnet machine, you can SSH directly to the VM's Tailscale name or IP — no port forwarding needed.

26.5 Clean separation

When the service is no longer needed:

ssh srv delete db

The rootfs, TAP device, cgroup, jailer workspace, and iptables rules are all cleaned up.

27. Throwaway debug VM

One of the most immediate uses for srv is spinning up an isolated Linux environment where you can install packages, run risky commands, or reproduce a bug — then delete it without any trace on the host.

27.1 Create

ssh srv new debug-vm

By default the VM gets 1 vCPU, 1 GiB of RAM, and a 10 GiB rootfs. Adjust if you need more:

ssh srv new debug-vm --cpus 2 --ram 4G --rootfs-size 30G

27.2 Watch it boot

ssh srv logs -f debug-vm serial

You will see the kernel boot, srv-bootstrap.service set up networking and Tailscale, and tailscale up --ssh complete. Once inspect reports state: ready:

ssh srv inspect debug-vm

Look for the tailscale-name and tailscale-ip fields.

27.3 Connect and use

ssh root@debug-vm

The guest image comes with Docker, Go, Neovim, Git, perf, valgrind, and common development tools preinstalled. Install anything else with pacman -S.

27.4 Clean up

When you are done:

ssh srv delete debug-vm

This removes the VM's rootfs, runtime directory, TAP device, cgroup, and jailer workspace. No trace remains on the host.

27.5 Tips

• Use -- --json inspect <name> to get machine-readable output for scripting
• The serial log under ssh srv logs <name> serial is append-only — always check the newest lines
• If a VM gets stuck in provisioning or failed, check ssh srv inspect <name> for the last_error field and ssh srv logs <name> firecracker for VMM errors

28. Networking overview

Every srv VM gets its own isolated network stack: a dedicated /30 subnet, TAP device, and NAT rules. Guest egress is routed through the host's outbound interface after MASQUERADE.

28.1 How it works

Internet    │    │ (host outbound interface)    │┌───┴─────────────────────┐│      Linux host         ││  ┌──────────────────┐   ││  │ iptables MASQ    │   ││  │ + FORWARD rules  │   ││  └────────┬─────────┘   ││           │             ││  ┌────────┴─────────┐   ││  │ TAP device       │   ││  │ (per-VM /30)     │   ││  └────────┬─────────┘   ││           │             ││  ┌────────┴─────────┐   ││  │ Firecracker VM   │   ││  │ gateway = .1     │   ││  │ guest   = .2     │   ││  └──────────────────┘   │└─────────────────────────┘

When ssh srv new demo runs, the network helper:

1. Allocates the next free /30 from SRV_VM_NETWORK_CIDR (default 172.28.0.0/16)
2. Creates a TAP device for the VM
3. Installs MASQUERADE and FORWARD rules for guest egress
4. Configures the gateway address on the host side

The VM's bootstrap configures the guest interface with:

• Gateway: SRV_DATA_DIR-derived host address (first usable IP in the /30)
• Guest IP: second usable IP in the /30
• DNS: configurable via SRV_VM_DNS (default 1.1.1.1, 1.0.0.1)

28.2 Tailscale integration

Each VM gets its own Tailscale identity. The control plane mints a one-off auth key and injects it through Firecracker MMDS metadata. The guest bootstrap service:

1. Reads the MMDS payload
2. Starts tailscaled
3. Runs tailscale up --auth-key=... --ssh on the first authenticated boot
4. Persists tailscaled state for warm reboots

This means any machine on the tailnet can reach the VM by its Tailscale name or IP — no port forwarding needed.

28.2.1 SSH access

Guests expose SSH through Tailscale's --ssh flag, so ssh root@<tailscale-name> works from any tailnet machine. Per-user OpenSSH keys are not injected — Tailscale SSH handles authentication based on tailnet identity.

28.2.2 Application and development services

Tailscale connectivity is not limited to SSH or HTTP. A service listening on a Tailscale-reachable guest address can be reached by authorized tailnet devices over TCP or UDP, under the VM identity and the tailnet's ACLs. This supports web applications, databases, debuggers, custom protocols, and multiple ports without host-side port forwarding.

For Amp runner VMs, this provides the network-connectivity role of an Orb Portal with a broader protocol surface. It does not provide Amp-managed HTTPS, thread embedding or annotation, public/thread-visibility access, or automatic Orb sleep/wake integration. Service availability follows the srv VM lifecycle.

28.3 Configuration

28.4 IPv4 forwarding

Guest NAT depends on IP forwarding:

sudo tee /etc/sysctl.d/90-srv-ip-forward.conf >/dev/null <<'EOF'net.ipv4.ip_forward = 1EOFsudo sysctl --system

This must stay enabled. Disabling it breaks guest egress.

28.5 Network cleanup

When a VM is deleted, the network helper removes:

• The TAP device
• The MASQUERADE rule
• The FORWARD rule
• The gateway address from the host interface

28.6 Host-side API gateways

srv can also expose host-side HTTP gateways on each VM's gateway IP:

• Provider gateways on SRV_ZEN_GATEWAY_PORT and SRV_DEEPSEEK_GATEWAY_PORT proxy /v1/... to configured LLM upstreams with host API keys injected.
• The generic integration gateway on SRV_INTEGRATION_GATEWAY_PORT proxies /integrations/<name>/... to operator-defined HTTP integrations with host-managed auth or headers injected.
• The Amp CONNECT gateway on SRV_AMP_CONNECT_GATEWAY_PORT (default 3128) provides an HTTP proxy for public HTTPS destinations. inspect --json exposes its URL as instance.amp_connect_gateway while the VM is active.

All gateway types only accept requests from the owning guest IP. See Provider gateways and HTTP integrations.

28.6.1 Amp CONNECT gateway

The CONNECT gateway is intended for an Amp runner whose general HTTPS egress must pass through the per-VM host endpoint. Set both HTTP_PROXY and HTTPS_PROXY in the guest to the URL returned by inspect --json; the bundled Amp runner plugin does this automatically.

This is a tunnel, not a TLS-terminating proxy: srv does not inspect HTTPS payloads or inject credentials. To prevent the proxy from becoming a path into the host or tailnet, it accepts only the CONNECT method, only permits destination port 443, and rejects loopback, link-local, private, carrier-grade NAT, documentation, multicast, other special-use, and host-local addresses. Destinations are checked again after DNS resolution and connection.

Use NO_PROXY for guest-local services, the VM gateway services, tailnet destinations, and other endpoints that should not use public HTTPS proxying. The bundled Amp runner config excludes localhost, MMDS, the tailnet, and the guest gateway, then appends any noProxy entries from ~/.config/amp/srv-runner.json.

29. Provider gateways

When API keys are configured on the host, srv sets up per-instance HTTP proxies that allow guest VMs to reach LLM provider APIs without storing real API keys inside the guest.

29.1 Providers

Both gateways share the same architecture. Only providers with a configured API key are enabled.

29.2 How it works

┌───────────────────────────────────────────────────┐│                       Host                        ││                                                   ││  ┌───────────────┐   ┌────────────────────────┐   ││  │ srv control   │   │ Provider gateways      │   ││  │ plane         │   │ :11434 (Zen)           │   ││  │               │   │ :11436 (DeepSeek)      │   ││  └───────────────┘   │ on gateway IP          │   ││                      │                        │   ││                      │ injects API key        │   ││                      │ into Authorization     │   ││                      │ header                 │   ││                      └───────────┬────────────┘   ││                                  │                ││                     ┌────────────┴───────────┐    ││                     │ upstreams:              │    ││                     │ opencode.ai/zen/v1      │    ││                     │ api.deepseek.com/v1     │    ││                     └────────────────────────┘    │└───────────────────────────────────────────────────┘                  │                  │ /30 network                  │        ┌─────────┴─────────┐        │ Guest VM          │        │                   │        │ opencode / pi ->  │        │ http://gateway    │        │ :11434/v1 (Zen)   │        │ :11436/v1 (DS)    │        └───────────────────┘

For each VM and each enabled provider, srv binds an HTTP proxy on the VM's host/gateway IP address and the provider's configured port. The proxies:

• Only accept requests from that VM's guest IP
• Forward /v1/... requests to the upstream API
• Inject the host's API key into the Authorization header (Bearer auth)
• The real keys never leave the host

DeepSeek uses OpenAI-compatible Bearer auth exclusively. The Zen gateway additionally supports Anthropic (X-API-Key) and Google (X-Goog-Api-Key) auth styles based on request path.

29.3 Guest bootstrap

When any provider gateway is enabled, the guest srv-bootstrap.service writes /root/.config/opencode/opencode.json targeting the per-VM gateways:

{  "$schema": "https://opencode.ai/config.json",  "provider": {    "opencode": {      "options": {        "baseURL": "http://:11434/v1",        "apiKey": "srv-provider-gateway"      }    },    "deepseek": {      "options": {        "baseURL": "http://:11436/v1",        "apiKey": "srv-provider-gateway"      }    }  }}

The apiKey is a local placeholder only so the CLIs keep the paid model catalogs visible — the real credentials still live only on the host and are injected by the proxies.

Bootstrap also writes Pi config under /root/.pi/agent/ so the preinstalled pi CLI uses both gateways by default:

{  "providers": {    "opencode": {      "baseUrl": "http://:11434/v1",      "apiKey": "srv-provider-gateway"    },    "deepseek": {      "baseUrl": "http://:11436/v1",      "apiKey": "srv-provider-gateway",      "compat": {        "supportsDeveloperRole": false,        "supportsStore": false      }    }  }}

When all gateways are disabled, bootstrap removes those managed default config files.

29.4 Configuration

29.5 Using the gateways from the guest

The preinstalled opencode and pi CLIs work out of the box. For other agents or HTTP clients:

# Inside the VM — Zencurl http://$(ip route show default | awk '{print $3}'):11434/v1/models# Inside the VM — DeepSeekcurl http://$(ip route show default | awk '{print $3}'):11436/v1/models

29.6 Disabling gateways

Remove or leave the respective API key env var unset. After the next guest boot, the bootstrap service will remove the managed config files for disabled providers.

30. srv Cheatsheet

Quick reference for the srv control plane.

30.1 Commands (via SSH)

ssh srv -- [--json]  [args]

Use --json with the non-streaming instance and backup commands when you need machine-readable output.

Use fixed mode for databases, JVMs, caches, and other memory-sensitive services. Pooled mode is for workloads that tolerate elastic memory.

30.2 Integrations (admin only)

30.3 Quick Examples

# Create VMssh srv new demossh srv statusssh srv -- --json inspect demo# With sizingssh srv new demo --cpus 4 --ram 8G --rootfs-size 20G# Integrationsssh srv integration add http openai --target https://api.openai.com/v1 --bearer-env SRV_SECRET_OPENAI_PRODssh srv integration enable demo openaissh srv inspect demo# Resize (must be stopped)ssh srv stop demossh srv resize demo --cpus 4 --ram 8Gssh srv start demo# Force rootfs check (must be stopped)ssh srv stop demossh srv fsck demo && ssh srv start demo# Backup and restore (stop VM first for safety)ssh srv backup create demossh srv backup list demossh srv backup delete demo ssh srv restore demo # View logsssh srv logs demossh srv logs demo serialssh srv logs demo firecrackerssh srv logs -f demo serialssh srv logs -f demo firecracker# Live VM usagessh -t srv topssh -t srv top --interval 2s

30.4 Systemd Management

# Statussudo systemctl status srv srv-net-helper srv-vm-runner# Restartsudo systemctl stop srv srv-net-helper srv-vm-runnersleep 5sudo systemctl start srv-vm-runner srv-net-helper srv# View logssudo journalctl -u srv -fsudo journalctl -u srv-vm-runner -f

30.5 Smoke Test

sudo ./contrib/smoke/host-smoke.sh

30.6 Build Artifacts

# Binariesgo build ./cmd/srvgo build ./cmd/srv-net-helpergo build ./cmd/srv-vm-runner# Guest imagesudo OUTPUT_DIR=/var/lib/srv/images/arch-base ./images/arch-base/build.sh# Installsudo ./contrib/systemd/install.shsudoedit /etc/srv/srv.envsudo ./contrib/systemd/install.sh --enable-now

30.7 Key Env Variables

30.8 Backup & Restore

# Backupsudo systemctl stop srv srv-net-helper srv-vm-runnersudo tar -czf backup.tar.gz /etc/srv /var/lib/srv# Restoresudo ./contrib/systemd/install.sh# restore /etc/srv/srv.env and /var/lib/srvsudo systemctl daemon-reloadsudo systemctl enable --now srv-vm-runner srv-net-helper srvsudo ./contrib/smoke/host-smoke.sh

30.9 Debug Commands

# VM inspectionssh srv inspect # Recent logsssh srv logs  serialssh srv logs  firecracker# System logsjournalctl -u srv-vm-runner | tail# Check cgroupscat /sys/fs/cgroup/firecracker-vms//cpu.maxcat /sys/fs/cgroup/firecracker-vms//memory.max

30.10 Notes

• VM disks at: SRV_DATA_DIR/instances/<name>/rootfs.img
• VM backups live at: SRV_DATA_DIR/backups/<name>/<backup-id>/
• Resize requires a stopped VM; CPU and RAM can go up or down within limits, but rootfs shrink is rejected
• Resize and fsck only work on stopped VMs
• Backup and restore only work on stopped VMs and only restore onto the original VM record, not a newly recreated VM with the same name
• Creators manage their own VMs; admins manage all VMs
• Integration commands are admin-only, and integration secrets are referenced by host env name rather than passed as raw SSH arguments
• Warm start/restart reuses tailscaled state
• Host reboot auto-restarts active VMs