1. Introduction to srv

srv is a self-hosted control-plane service for creating Firecracker microVMs over SSH on a Tailscale tailnet.

It exposes an SSH command surface on one Linux host and manages Firecracker microVMs behind it. You create, inspect, stop, start, resize, back up, and delete VMs the same way you would run a remote command — through ssh srv <command>.

ssh srv new demo# demo created — state: provisioning# inspect:  ssh srv inspect demo# connect:  ssh root@demo

1.1 Where srv fits

srv is useful for both short-lived sandboxes and persistent isolated services.

• Throwaway debug VMs — spin up an isolated environment, break things, and delete it without affecting the host
• Sandboxed agent VMs — give AI coding agents their own cgroup-limited VM with per-instance Tailscale identity and a scoped Zen API proxy
• Dev/test environments — fast reflink-based clones from a single base image, with backup/restore for instant reset
• Isolated workloads — run services in separate microVMs with per-VM networking, auth, and resource limits

All VMs get a full Linux system with systemd, a real kernel, and their own Tailscale identity. Where containers are not enough — package managers, services that need root, bespoke networking configurations, predictable boot and teardown — srv gives you real isolated machines in seconds.

1.2 Conceptual architecture

                    Tailscale tailnet                           │                     ┌─────┴─────┐                     │   tsnet   │  (joins tailnet as "srv")                     │  :22/tcp  │  (SSH API surface)                     └─────┬─────┘                           │             ┌─────────────┼─────────────┐             │             │             │      ┌──────┴──────┐ ┌──────┴──────┐ ┌──────┴──────┐      │ VM: demo    │ │ VM: ci      │ │ VM: test    │      │ /30 net     │ │ /30 net     │ │ /30 net     │      │ cgroup      │ │ cgroup      │ │ cgroup      │      │ TAP + NAT   │ │ TAP + NAT   │ │ TAP + NAT   │      └─────────────┘ └─────────────┘ └─────────────┘

Key components:

• tsnet — joins the tailnet as srv and exposes the control API on tailnet TCP port 22
• gliderlabs/ssh — handles exec requests and rejects shell sessions
• SQLite — stores instances, events, command audits, and authorization decisions
• Reflinks — clone the base rootfs for fast per-instance writable disks
• Network helper — root-only process owns TAP creation, iptables MASQUERADE, and FORWARD rules
• VM runner — root-owned process invokes Firecracker through the official jailer, drops to srv-vm:srv, and places each VM into its own cgroup v2 leaf
• MMDS — one-off Tailscale auth keys are injected through Firecracker metadata so guests self-bootstrap
• Zen gateway — per-instance HTTP proxy on the guest's gateway IP forwards to OpenCode Zen with the host key
• HTTP integrations — admin-defined host-side HTTP proxies inject headers or auth for selected guests without storing raw secrets inside the VM

1.3 Where you can run it

srv runs on Linux with the following requirements:

• Linux host with cgroup v2 and /dev/kvm
• IPv4 forwarding enabled (net.ipv4.ip_forward=1)
• SRV_DATA_DIR on a reflink-capable filesystem (btrfs or reflink-enabled xfs)
• Tailscale installed and working on the host
• Tailscale OAuth client credentials with permission to mint auth keys for guest tags
• ip, iptables, cp, and resize2fs available on the host
• Official static Firecracker and jailer release pair

1.4 Next steps

• Install srv
• Walkthrough
• Architecture
• SSH command reference
• HTTP integrations

2. Installation

2.1 System requirements

• Linux host with cgroup v2 and /dev/kvm
• IPv4 forwarding enabled: net.ipv4.ip_forward=1
• SRV_DATA_DIR on a reflink-capable filesystem (btrfs or reflink-enabled xfs), with SRV_BASE_ROOTFS on the same filesystem
• Tailscale installed and working on the host
• Tailscale OAuth client credentials (TS_CLIENT_ID and TS_CLIENT_SECRET) with permission to mint auth keys for the configured guest tags
• ip, iptables, cp, and resize2fs available on the host
• Official static Firecracker and jailer release pair (the installer can download these for you)

2.2 Build

Build the three binaries from source:

go build ./cmd/srvgo build ./cmd/srv-net-helpergo build ./cmd/srv-vm-runner

Or install them to a standard location with the provided installer (see below).

2.3 Build the guest image

The Arch base image builder produces the vmlinux kernel and rootfs-base.img rootfs that srv provisions from.

On an Arch Linux host:

sudo OUTPUT_DIR=/var/lib/srv/images/arch-base ./images/arch-base/build.sh

On a non-Arch host, use the podman workflow:

sudo podman run --rm --privileged --network host \  -v "$PWD":/work \  -v /var/lib/srv/images/arch-base:/var/lib/srv/images/arch-base \  -w /work \  docker.io/library/archlinux:latest \  bash -lc '    set -euo pipefail    pacman -Sy --noconfirm archlinux-keyring    pacman -Syu --noconfirm arch-install-scripts base-devel bc e2fsprogs rsync curl systemd    OUTPUT_DIR=/var/lib/srv/images/arch-base ./images/arch-base/build.sh  '

See Building a custom guest image and the guest image reference for details on what the image includes and how to customize it.

2.4 Install and configure

The systemd installer handles binary installation, unit setup, and downloading the official static Firecracker/jailer pair:

sudo ./contrib/systemd/install.sh

Enable IPv4 forwarding:

sudo tee /etc/sysctl.d/90-srv-ip-forward.conf >/dev/null <<'EOF'net.ipv4.ip_forward = 1EOFsudo sysctl --system

Write the environment file. At minimum you need Tailscale credentials and guest artifact paths:

sudoedit /etc/srv/srv.env

Key entries to fill in:

TS_AUTHKEY=tskey-auth-xxxxxxxxxxxxxxxxTS_CLIENT_ID=your-oauth-client-idTS_CLIENT_SECRET=your-oauth-client-secretTS_TAILNET=your-tailnet.ts.netSRV_BASE_KERNEL=/var/lib/srv/images/arch-base/vmlinuxSRV_BASE_ROOTFS=/var/lib/srv/images/arch-base/rootfs-base.imgSRV_GUEST_AUTH_TAGS=tag:microvm

See the full configuration reference for every variable.

Then start the services:

sudo ./contrib/systemd/install.sh --enable-now

2.5 Validate

Run the end-to-end smoke test to confirm everything works:

sudo ./contrib/smoke/host-smoke.sh

The smoke test verifies systemd units, SSH reachability, guest creation, readiness polling, backup/restore, cgroup limits, and cleanup. It is the supported validation gate after install, restore, and upgrade.

See the smoke test reference for details on overrides and failure artifacts.

2.6 Next steps

• Walkthrough — create your first VM
• Running as a daemon — systemd details

3. Walkthrough

This walkthrough assumes you have completed the installation and the smoke test passes.

3.1 Create a VM

ssh srv new demo

demo created — state: provisioninginspect:  ssh srv inspect democonnect:  ssh root@demo

The control plane creates a reflink clone of the base rootfs, allocates a /30 network, mints a one-off Tailscale auth key, injects it through MMDS, and boots the VM through the jailer.

With custom sizing:

ssh srv new demo --cpus 4 --ram 8G --rootfs-size 20G

3.2 Check status

ssh srv list

NAME     STATE   CPUS  MEMORY   DISK     TAILSCALEdemo     ready   1     1.0 GiB  10.0 GiB 100.64.0.2

For detailed info:

ssh srv inspect demo

Machine-readable output for scripting:

ssh srv -- --json listssh srv -- --json inspect demo

3.3 Connect to the VM

Once the VM reports state: ready and shows a Tailscale IP, connect over the tailnet:

ssh root@demo

Because the guest image bootstraps tailscale up --ssh, Tailscale SSH handles authentication without needing per-user OpenSSH keys in the guest.

You can also get the IP from inspect:

ssh srv inspect demo# look for tailscale-ip and tailscale-name fields

3.4 View logs

ssh srv logs demossh srv logs demo serialssh srv logs demo firecrackerssh srv logs -f demo serial

The serial log shows guest boot, bootstrap, and tailscaled output. The Firecracker log shows VMM lifecycle events.

3.5 Stop, restart, and delete

ssh srv stop demossh srv start demossh srv restart demossh srv delete demo

Stopping a VM does a graceful shutdown. The guest rootfs persists across stop/start cycles.

3.6 Resize a VM

Resize requires the VM to be stopped. CPU and memory can be increased or decreased within limits, while rootfs can only grow:

ssh srv stop demossh srv resize demo --cpus 4 --ram 8G --rootfs-size 20Gssh srv start demo

See Resize a VM for details.

3.7 Back up and restore

ssh srv stop demossh srv backup create demossh srv backup list demossh srv restore demo 

See Backup and restore for the full workflow.

3.8 Move a VM between hosts

ssh srv-a export demo | ssh srv-b import

See Export and import for the semantics.

3.9 Next steps

• Instance lifecycle — full command reference
• Networking overview — how VM networking works
• Configuration — all environment variables

4. Architecture

srv is a single-host control plane that manages Firecracker microVMs through an SSH command surface on a Tailscale tailnet.

4.1 Control plane

┌────────────────────────────────────────────────────────┐│                      srv process                       ││                                                        ││  ┌──────────┐  ┌──────────────┐  ┌──────────────────┐  ││  │ tsnet    │  │ gliderlabs/  │  │ authorization    │  ││  │ :22/tcp  │──│ ssh          │──│ (Tailscale       │  ││  └──────────┘  └──────────────┘  │ WhoIs)           │  ││                                  └──────────────────┘  ││                                                        ││  ┌──────────┐  ┌──────────────┐  ┌──────────────────┐  ││  │ SQLite   │  │ reflink      │  │ Tailscale API    │  ││  │ store    │  │ cloner       │  │ (auth key        │  ││  └──────────┘  └──────────────┘  │ minting)         │  ││                                  └──────────────────┘  ││                                                        ││  ┌─────────────┐  ┌─────────────────────────────────┐  ││  │ Zen gateway │  │ per-VM HTTP proxy               │  ││  │ manager     │  │ (injects SRV_ZEN_API_KEY)       │  ││  └─────────────┘  └─────────────────────────────────┘  │└────────────────────────────────────────────────────────┘         │                    │                  │    ┌────┴────┐          ┌────┴────┐        ┌────┴─────┐    │ net-    │          │ vm-     │        │ Tailscale│    │ helper  │          │ runner  │        │ coord.   │    │ (root)  │          │ (root)  │        │ server   │    └────┬────┘          └────┬────┘        └──────────┘         │                    │    ┌────┴────┐          ┌────┴─────────────┐    │ TAP +   │          │ Firecracker      │    │ iptables│          │ + jailer         │    │ + NAT   │          │ → srv-vm:srv     │    └─────────┘          │ → cgroup v2      │                         └──────────────────┘

4.2 Key components

4.2.1 tsnet + gliderlabs/ssh

srv joins the tailnet as the hostname configured in SRV_HOSTNAME (default srv) and listens on the tailnet TCP port configured in SRV_LISTEN_ADDR (default :22). The gliderlabs/ssh library handles SSH exec requests and rejects shell sessions.

Caller identity comes from Tailscale WhoIs data resolved from the incoming tailnet connection — not from the SSH username.

4.2.2 SQLite store

All instance state, event history, command audits, and authorization decisions are stored in SQLite under SRV_DATA_DIR/state/app.db. Migrations run during startup and are additive.

4.2.3 Reflink cloning

When a new VM is created, the base rootfs is cloned using filesystem reflinks (on btrfs or reflink-enabled xfs). This gives each VM its own writable copy without actually copying the data — the copy is instantaneous on a reflink-capable filesystem.

4.2.4 Network helper

A root-only helper process owns all TAP device creation, iptables MASQUERADE rules, and FORWARD rules. The main srv process communicates with it over a unix socket.

4.2.5 VM runner

A root-owned process invokes Firecracker through the official jailer binary. After the jailer sets up the chroot and drops privileges, the microVM process runs as srv-vm:srv. Each VM is placed into its own cgroup v2 leaf under firecracker-vms/<name> with enforced limits.

4.2.6 Tailscale integration

For each new VM, the control plane:

1. Mints a one-off Tailscale auth key using the configured OAuth credentials and tags
2. Injects the key into the VM's MMDS payload
3. The guest bootstrap service reads the key and runs tailscale up --auth-key=... --ssh

On warm reboots (after a stop + start or restart), the guest reuses its persisted tailscaled state instead of minting a new key.

4.2.7 Zen gateway

When SRV_ZEN_API_KEY is set, srv binds a per-instance HTTP proxy on each VM's gateway IP and SRV_ZEN_GATEWAY_PORT. The proxy forwards guest requests to the upstream Zen API while injecting the host key. See Zen gateway for details.

4.3 Data paths

SRV_DATA_DIR/├── state/│   ├── app.db              # SQLite store│   ├── tsnet/               # Tailscale persistent state│   └── host_key             # SSH host key├── images/│   └── arch-base/│       ├── vmlinux          # Guest kernel│       └── rootfs-base.img  # Base rootfs (reflink source)├── instances/│   └── /│       ├── rootfs.img       # Writable reflink clone│       ├── serial.log       # Serial console output│       └── firecracker.log  # VMM log├── backups/│   └── /│       └── /│           └── rootfs.img   # Backup copy├── jailer/                 # Jailer workspaces└── .snapshots/              # btrfs host snapshots    └── /

4.4 Service architecture

The srv-vm-runner process starts as root, and the jailer drops the Firecracker process to srv-vm:srv within each VM's cgroup v2 leaf.

5. SSH command reference

All srv commands are invoked through the SSH surface. The service treats SSH as command transport only — there is no shell session.

5.1 Syntax

ssh srv  [args]

For machine-readable output, use --json with non-streaming instance and backup commands. With OpenSSH, terminate local option parsing first:

ssh srv -- --json listssh srv -- --json inspect demossh srv -- --json status

5.2 Instance commands

5.3 Lifecycle commands

5.4 Resize command

All flags can be combined. Omitted flags keep the current value.

5.5 Backup commands

5.6 Transfer commands

Usage: ssh srv-a export demo | ssh srv-b import

5.7 Host commands

5.8 Integration commands

All integration commands are admin-only.

5.9 Notes

• new accepts --cpus, --ram, and --rootfs-size in any combination
• new also accepts repeated --integration <name> flags; the create request fails if any requested integration cannot be enabled
• resize requires the VM to be stopped; CPU and RAM may increase or decrease within limits, while rootfs is grow-only
• resize, backup, and restore all require the VM to be stopped
• Backups are tied to the original VM record — they cannot be restored onto a different VM
• Export requires the source VM to be stopped
• Import recreates the VM under the same name and leaves it stopped
• top refreshes continuously by default; use ssh -t srv top --interval 2s or similar to slow the redraw rate
• Integration targets are intentionally narrow in v1: HTTP only, operator-managed, no guest-supplied raw secrets, and no automatic outbound interception

6. HTTP integrations

srv can expose selected upstream HTTP APIs to a VM through a host-side integration gateway without storing the real upstream credentials inside the guest.

This feature is intentionally narrow in v1:

• Admin or operator managed only
• HTTP integrations only
• No guest-side secret submission over SSH
• No discovery, tagging, or attachment system
• No transparent interception of arbitrary outbound HTTPS

6.1 How it works

┌──────────────────────────────────────────────────────┐│                       Host                           ││                                                      ││  srv control plane                                   ││      │                                               ││      │ integration add/enable                        ││      ▼                                               ││  SQLite metadata + /etc/srv/srv.env                  ││      │                                               ││      ▼                                               ││  Integration gateway :11435 on VM gateway IP         ││      │                                               ││      │ /integrations/openai/...                      ││      │ injects headers/auth from SRV_SECRET_*        ││      ▼                                               ││  upstream HTTP API                                   │└──────────────────────────────────────────────────────┘                       │                       │ /30 network                       │              ┌────────┴────────┐              │ Guest VM        │              │                 │              │ curl http://    │              │ :11435 │              │ /integrations/  │              │ openai/...      │              └─────────────────┘

Each enabled VM gets access to a host-side HTTP listener on its gateway IP and SRV_INTEGRATION_GATEWAY_PORT (default 11435). Requests are routed by path prefix:

• http://<gateway-ip>:11435/integrations/<name>/...

The host looks up the named integration, rewrites the request to the configured upstream target, injects the configured auth or headers, and forwards the request.

6.2 Supported auth and header modes

An integration can use any combination of:

• Static headers via --header NAME:VALUE
• Env-backed headers via --header-env NAME:SRV_SECRET_FOO
• Bearer auth via --bearer-env SRV_SECRET_FOO
• Basic auth via --basic-user USER --basic-password-env SRV_SECRET_BAR

Secrets are referenced by env var name only. The raw values are expected to live on the host, typically in /etc/srv/srv.env.

For --header-env, the env var value becomes the exact header value sent upstream.

6.3 Security behavior

The gateway is designed so the guest gets access to the upstream capability, not the underlying secret material.

• The gateway only accepts requests from the owning guest IP
• Guest-supplied Authorization and Proxy-Authorization headers are stripped on the host
• Env-backed secrets are read on the host at request time
• If a referenced SRV_SECRET_* env var is missing, the gateway returns 502 Bad Gateway
• Request paths are normalized before routing so traversal attempts cannot escape the configured integration prefix

6.4 Example setup

Add host secrets to /etc/srv/srv.env:

SRV_INTEGRATION_GATEWAY_PORT=11435SRV_SECRET_OPENAI_PROD=sk-live-redactedSRV_SECRET_VENDOR_API_KEY=vendor-key-redacted

Create and enable integrations from the control plane:

ssh srv integration add http openai --target https://api.openai.com/v1 --bearer-env SRV_SECRET_OPENAI_PRODssh srv integration add http vendor --target https://vendor.example/api --header-env X-API-Key:SRV_SECRET_VENDOR_API_KEYssh srv integration enable demo openaissh srv integration enable demo vendorssh srv inspect demo

The VM's inspect output will include URLs such as:

integrations:- openai: http://172.28.0.1:11435/integrations/openai- vendor: http://172.28.0.1:11435/integrations/vendor

From inside the guest, requests go to the gateway IP:

curl http://$(ip route show default | awk '{print $3}'):11435/integrations/openai/modelscurl http://$(ip route show default | awk '{print $3}'):11435/integrations/vendor/ping

6.5 Related references

• SSH command reference
• Configuration reference
• Networking overview

7. Running as a daemon

srv is designed to run as a set of systemd services on a prepared host. The installer sets up three units that must all be active for the control plane to function.

7.1 Systemd units

7.2 Common systemd commands

# Check statussudo systemctl status srv srv-net-helper srv-vm-runner# View logssudo journalctl -u srv -fsudo journalctl -u srv-vm-runner -f# Restart all servicessudo systemctl stop srv srv-net-helper srv-vm-runnersleep 5sudo systemctl start srv-vm-runner srv-net-helper srv

7.3 Host reboot recovery

Same-host reboot recovery is built into the control plane. When srv comes back under systemd, previously active instances are restarted automatically.

You do not need to manually restart VMs after a host reboot.

7.4 Environment file

Configuration lives in /etc/srv/srv.env. See the configuration reference for every variable.

If /etc/srv/srv.env already exists before an upgrade, the installer keeps it by default. After upgrade, verify that SRV_FIRECRACKER_BIN and SRV_JAILER_BIN still point at the intended static binaries.

7.5 Upgrading

1. Take a quiesced backup or a host snapshot.

2. Build and install the new srv, srv-net-helper, and srv-vm-runner binaries.

3. If you also refreshed Firecracker, verify /etc/srv/srv.env still points at the correct binaries.

4. Restart:

   sudo systemctl stop srv srv-net-helper srv-vm-runnersleep 5sudo systemctl start srv-vm-runner srv-net-helper srv

5. Run the smoke test:

   sudo ./contrib/smoke/host-smoke.sh

See the operations reference for full upgrade and rollback procedures.

7.6 Smoke test as a validation gate

The smoke test is part of the supported workflow after install, restore, and upgrade — not an optional extra:

sudo ./contrib/smoke/host-smoke.sh

Overrides:

• ENV_PATH=/etc/srv/srv.env.alt — alternate environment file
• SMOKE_SSH_HOST=srv-test — alternate control-plane hostname
• INSTANCE_NAME=smoke-manual — force a predictable instance name
• KEEP_FAILED=1 — leave a failed instance intact for debugging
• READY_TIMEOUT_SECONDS=300 — override guest-ready timeout

8. Authorization

srv uses Tailscale identity for authentication and authorization. The SSH surface does not use traditional SSH username/password authentication — instead, it resolves the caller's Tailscale identity from the incoming tailnet connection.

8.1 How it works

1. A user connects to srv over the tailnet: ssh srv <command>
2. tsnet resolves the connection using WhoIs to get the Tailscale login and node
3. The control plane checks the resolved identity against the configured allow and admin lists
4. If authorized, the command runs; the decision is recorded in the audit store

8.2 User roles

8.2.1 Regular users

If SRV_ALLOWED_USERS is unset or empty, any authenticated tailnet user can invoke commands. Regular users can:

• Create VMs (new)
• Manage their own VMs (inspect, logs, stop, start, restart, delete)
• View their own VMs in list
• Create backups and restores for their own VMs
• Resize their own VMs
• Export their own VMs

8.2.2 Admin users

Users listed in SRV_ADMIN_USERS can additionally:

• View and manage all VMs, not just their own
• Run status (host capacity summary)
• Run snapshot create (host-level btrfs snapshot)
• Import VMs

8.3 Configuration

Example /etc/srv/srv.env:

SRV_ALLOWED_USERS=alice@example.com,bob@example.comSRV_ADMIN_USERS=ops@example.com

8.4 Audit trail

Every command invocation is recorded in the SQLite store with:

• Actor Tailscale login and display name
• Actor node name
• Remote address
• SSH user
• Command and arguments
• Whether the command was allowed
• Reason for allow/deny
• Duration in milliseconds
• Error text if any

This audit trail is visible through ssh srv -- --json inspect <name> in the instance events.

9. Configuration reference

srv is configured through environment variables, read from /etc/srv/srv.env by the systemd units.

9.1 Tailscale credentials

Use either TS_AUTHKEY or TS_CLIENT_ID/TS_CLIENT_SECRET for the control-plane node. The OAuth flow is preferred for guest auth key minting.

9.2 Core settings

9.3 Guest artifacts

9.4 Guest defaults

9.5 Networking

9.6 Authorization

9.7 Zen gateway

9.8 HTTP integrations

The SRV_SECRET_* entries are a naming convention rather than a fixed list. They are meant to live in /etc/srv/srv.env so the SSH control surface only ever receives secret names such as SRV_SECRET_OPENAI_PROD, not the raw credential values.

9.9 Alternate Tailscale endpoints

9.10 Misc

9.11 Path constraints

Several paths must share the same reflink-capable filesystem (typically btrfs or reflink-enabled xfs):

• SRV_DATA_DIR — host state directory
• SRV_BASE_ROOTFS — base guest image
• SRV_JAILER_BASE_DIR — jailer workspaces (hard-links log files into the jail)

Cross-filesystem hard-links fail, so keeping these on the same filesystem is required.

10. Guest image reference

The default guest image is an Arch Linux rootfs designed for srv. It is built by images/arch-base/build.sh and contains a full Linux userspace with developer tooling preinstalled.

10.1 Artifacts

The builder produces:

10.2 Included packages

The image is intentionally not minimal — it includes tooling for development and AI agent workflows:

• Docker, docker-compose
• Go, gopls
• Odin, odinfmt, OLS
• Neovim with prewarmed LazyVim (BMW heritage amber theme)
• OpenCode and Pi CLIs with per-VM Zen gateway bootstrap
• Git, fd, ripgrep, tree-sitter-cli, gcc, perf, valgrind
• iptables-nft with IPv4/IPv6 nftables support
• Kernel module tree matching the custom kernel (overlay, br_netfilter, Docker-related modules)

10.3 Bootstrap service

The guest includes srv-bootstrap.service, which runs on every boot:

1. Discovers the primary virtio interface from the kernel-provided default route
2. Adds a route to Firecracker MMDS at 169.254.169.254/32
3. Reads the MMDS payload from http://169.254.169.254/ with Accept: application/json
4. Sets the hostname from srv.hostname
5. Starts tailscaled
6. Runs tailscale up --auth-key=... --hostname=... --ssh on the first authenticated boot (relies on persisted state on later boots)
7. Writes /root/.config/opencode/opencode.json plus Pi config under /root/.pi/agent/ to the per-VM host Zen gateway when SRV_ZEN_API_KEY is configured, or removes those managed defaults when the gateway is disabled
8. Writes /var/lib/srv/bootstrap.done with the latest successful bootstrap timestamp

The --ssh flag on tailscale up is intentional — it enables Tailscale SSH so the control plane's connect: ssh root@<name> output works through the tailnet without per-user OpenSSH keys.

10.4 Kernel details

• Starts from Firecracker's microvm-kernel-ci-x86_64-6.1.config and runs olddefconfig against the selected source tree
• Enables CONFIG_PCI=y for ACPI initialization (required by current Firecracker x86 builds)
• Disables CONFIG_VIRTIO_MMIO_CMDLINE_DEVICES so the kernel prefers ACPI discovery
• Enables Landlock and adds it to CONFIG_LSM (keeps pacman's download sandbox working)
• Builds real .ko module files for Docker, overlay, br_netfilter, and nftables

10.5 DNS

/etc/resolv.conf is symlinked to /proc/net/pnp so the kernel ip= boot parameter provides working DNS before tailscale up runs.

10.6 Logging

journald is configured to forward logs to ttyS0, making the guest bootstrap flow visible in each instance's serial log (ssh srv logs <name> serial).

11. Operations runbook

This runbook is for the supported prepared-host path: systemd-managed srv, srv-net-helper, and srv-vm-runner on a Linux host with cgroup v2, /dev/kvm, SRV_DATA_DIR on a reflink-capable filesystem such as btrfs or reflink-enabled xfs, SRV_BASE_ROOTFS on the same filesystem, and the official static Firecracker/jailer release pair.

Same-host reboot recovery is already built into the control plane: when srv comes back under systemd, previously active instances are restarted automatically. The steps below cover the larger operator workflows that were previously only implied.

11.1 Supported Upgrade Lanes

• Guest-local maintenance is still guest-local. Running pacman -Syu inside a guest is supported when you want that guest to drift independently, but it is not the control plane's golden-image rollout path.
• Kernel roll-forward is a host-managed lane. Existing stopped guests pick up the current SRV_BASE_KERNEL and optional SRV_BASE_INITRD on their next start or restart.
• Rootfs golden-image rollout is a new-guest lane. Updating SRV_BASE_ROOTFS changes what future new clones from, but it does not rewrite existing guests' writable disks.
• Schema rollout is tied to the control-plane binary. SQLite migrations run during srv startup and are currently additive; rollback means restoring the pre-upgrade backup together with the previous binary set.

11.2 Backup

Take backups from a quiesced host so SRV_DATA_DIR and the SQLite WAL state are self-consistent.

If you need a fast host-local point-in-time copy without shutting the control plane down first, use the built-in snapshot barrier instead:

ssh srv snapshot create

That command is admin-only. It briefly rejects every other SSH command, waits for already admitted commands to finish, checkpoints SQLite, flushes the filesystem, and then creates a readonly btrfs snapshot of SRV_DATA_DIR under SRV_DATA_DIR/.snapshots/<timestamp>.

Snapshot semantics are intentionally limited and explicit:

• control-plane consistent
• stopped guests fully safe
• running guests crash-consistent

Important caveats for this path:

• SRV_DATA_DIR itself must be a btrfs subvolume root. A plain directory on btrfs is not enough.
• The app snapshots SRV_DATA_DIR only. /etc/srv, environment files, and unit overrides still need the existing operator-managed backup flow below.
• Remote btrfs send/receive replication is intentionally out of the barrier path. If you use it for DR or warm standby, run it after the local snapshot already exists.

1. Stop the services.

sudo systemctl stop srv srv-net-helper srv-vm-runner

2. Capture the state directory, environment file, and any unit overrides.

sudo tar --xattrs --acls --numeric-owner \  --ignore-failed-read \  -C / \  -czf /var/tmp/srv-backup-$(date -u +%Y%m%dT%H%M%SZ).tar.gz \  etc/srv \  var/lib/srv \  etc/systemd/system/srv.service.d \  etc/systemd/system/srv-net-helper.service.d \  etc/systemd/system/srv-vm-runner.service.d \  etc/systemd/system.control

3. Start the services again if this was only a backup window.

sudo systemctl start srv-vm-runner srv-net-helper srv

Notes:

• Preserve the configured paths in /etc/srv/srv.env. Instance rows store absolute runtime paths such as SRV_DATA_DIR/instances/<name>/rootfs.img, so changing SRV_DATA_DIR, SRV_JAILER_BASE_DIR, or the base artifact paths during restore is not a supported relocation workflow.
• Keep SRV_JAILER_BASE_DIR on the same filesystem as SRV_DATA_DIR; the runner hard-links log files into the jail and cross-filesystem links fail.

11.3 Move One Stopped VM Between Hosts

For single-VM cutover, use the portable stopped-VM stream instead of copying SQLite rows or the whole host state directory:

ssh srv-a export demo | ssh srv-b import

Operational notes:

• The source VM must be stopped before export.
• Import recreates the VM under the same name and leaves it stopped. Start it explicitly on the destination after the stream completes.
• The artifact preserves portable metadata such as name, creator, machine shape, rootfs size, and last-known Tailscale name or IP.
• Serial and Firecracker logs are included when present, but each is capped to the newest 256 MiB during export.
• Import regenerates destination-local runtime state such as absolute file paths, tap device wiring, guest MAC, and VM subnet allocation.
• The copied rootfs carries the guest's durable Tailscale identity, so do not boot the source and destination copies at the same time.
• The destination host uses its currently configured SRV_BASE_KERNEL and optional SRV_BASE_INITRD on the first later start; only the writable guest disk and optional logs come from the streamed artifact.

11.4 Restore Or Rebuild A Host

1. Prepare a fresh host with the normal prerequisites: Tailscale, cgroup v2, /dev/kvm, reflink-capable storage such as btrfs or reflink-enabled xfs shared by SRV_DATA_DIR and SRV_BASE_ROOTFS, and the repo checkout.
2. Reinstall the managed assets.

sudo ./contrib/systemd/install.sh

3. Re-enable IPv4 forwarding for guest NAT on the rebuilt host.

sudo tee /etc/sysctl.d/90-srv-ip-forward.conf >/dev/null <<'EOF'net.ipv4.ip_forward = 1EOFsudo sysctl --system

4. Restore /etc/srv/srv.env from backup and verify that SRV_FIRECRACKER_BIN, SRV_JAILER_BIN, SRV_DATA_DIR, SRV_BASE_KERNEL, SRV_BASE_ROOTFS, and any optional SRV_BASE_INITRD still point at the intended paths.
5. Restore the saved SRV_DATA_DIR tree to the same path.
6. Reload systemd and start the services.

sudo systemctl daemon-reloadsudo systemctl enable --now srv-vm-runner srv-net-helper srv

7. Run the prepared-host validation gate before handing the host back to users.

sudo ./contrib/smoke/host-smoke.sh

That smoke pass is part of the supported restore workflow now, not an optional extra.

11.5 Upgrade And Rollback

11.5.1 Control Plane And Schema

1. Take the quiesced backup above before starting the upgraded srv binary for the first time.
2. Build and install the new srv, srv-net-helper, and srv-vm-runner binaries.
3. If you also refreshed Firecracker, use the matching official static firecracker and jailer pair and verify /etc/srv/srv.env still points at those paths.
4. Restart the services.

sudo systemctl stop srv srv-net-helper srv-vm-runnersleep 5sudo systemctl start srv-vm-runner srv-net-helper srv

5. Run the host smoke test.

sudo ./contrib/smoke/host-smoke.sh

Rollback for control-plane or schema regressions is restore-based:

1. Stop the services.
2. Reinstall the previous binaries and previous static Firecracker/jailer pair if those changed.
3. Restore the pre-upgrade backup of /etc/srv and SRV_DATA_DIR.
4. Restart the services.
5. Run the same host smoke test again.

11.5.2 Kernel Rollout For Existing Guests

1. Rebuild the kernel artifact under images/arch-base/.
2. Update SRV_BASE_KERNEL and optional SRV_BASE_INITRD in /etc/srv/srv.env.
3. Restart the units if needed so the runner sees the new base paths.
4. Stop and start guests one at a time, or let already stopped guests pick up the new boot artifacts on their next start.
5. Use a canary guest first, then roll wider once it passes the workload check you care about.

Rollback is just pointing SRV_BASE_KERNEL or SRV_BASE_INITRD back to the previous artifact and restarting the affected guests again.

11.5.3 Golden Rootfs Rollout

1. Rebuild rootfs-base.img under images/arch-base/.
2. Point SRV_BASE_ROOTFS at the new image.
3. Create a canary guest with ssh srv new <name> and validate it.
4. After the canary passes, new guests will clone from the new base image.

Rollback for the golden rootfs lane is also path-based: point SRV_BASE_ROOTFS back to the previous image before creating more guests.

Important caveat: existing guests keep their own writable rootfs.img. There is no host-driven in-place existing-rootfs conversion workflow yet. For an existing guest that needs OS updates today, the supported choices are:

• manage that guest locally with pacman -Syu, accepting guest-local drift
• create a replacement guest from the refreshed golden image and migrate the workload or data to it

11.6 Host Hardening And Caveats

• cgroup v2 is required. The runner now depends on a delegated cgroup v2 subtree to place each VM into its own firecracker-vms/<name> leaf with enforced cpu.max, memory.max, memory.swap.max, and pids.max.
• IPv4 forwarding must stay enabled on the host. Guest egress depends on forwarding packets from each TAP device through the host's outbound interface after the helper installs MASQUERADE and FORWARD rules.
• srv-vm-runner.service must keep User=root, Group=srv, Delegate=cpu memory pids, DelegateSubgroup=supervisor, and a group-accessible socket under /run/srv-vm-runner/.
• Do not add NoNewPrivileges=yes to srv-vm-runner.service; the jailer must drop privileges and exec Firecracker on real hosts.
• Keep using the official static Firecracker and jailer release pairing. Distro-provided dynamically linked binaries can fail after chroot before the API socket appears.
• Preserve /etc/srv/srv.env across reinstall or upgrade unless you are intentionally changing configuration and have accounted for the stored absolute paths.
• Keep SRV_DATA_DIR and SRV_BASE_ROOTFS on the same reflink-capable filesystem, such as btrfs or reflink-enabled xfs. Fast per-instance provisioning still depends on reflink cloning the configured base rootfs.
• Run sudo ./contrib/smoke/host-smoke.sh after install, restore, control-plane upgrade, and base-image changes.

12. Troubleshooting

12.1 VM is stuck in provisioning or failed

Check the control-plane view first:

ssh srv inspect 

Look for the last_error field and state. Then check the guest and VMM logs:

ssh srv logs  serialssh srv logs  firecracker

Common causes:

• Missing or misconfigured base rootfs/kernel
• Tailscale OAuth credentials expired or invalid
• Network helper not running (systemctl status srv-net-helper)
• VM runner not running (systemctl status srv-vm-runner)
• /dev/kvm not accessible

12.2 VM boots but Tailscale SSH doesn't work

If ssh srv inspect <name> shows state: ready and a Tailscale IP but you can't SSH in:

1. Check the serial log for bootstrap errors: ssh srv logs <name> serial
2. Look for srv-bootstrap.service failures
3. Verify that SRV_GUEST_AUTH_TAGS matches a tag your OAuth client can mint keys for
4. Verify that the guest can reach the Tailscale coordination server (check DNS and outbound connectivity from inside the VM if possible)

12.3 Guest can't reach the internet

Guest egress depends on IPv4 forwarding and MASQUERADE rules:

# Check forwardingsysctl net.ipv4.ip_forward# Check iptables rulessudo iptables -t nat -L -nsudo iptables -L FORWARD -n# Check the network helpersudo systemctl status srv-net-helper

If net.ipv4.ip_forward is 0, re-enable it:

sudo sysctl -w net.ipv4.ip_forward=1

12.4 VM runs out of disk space

Check from inside the VM:

df -h

If the rootfs is full, you can resize the VM:

ssh srv stop ssh srv resize  --rootfs-size 20Gssh srv start 

On the host, check SRV_DATA_DIR usage:

df -h /var/lib/srv

12.5 Check cgroup limits

If a VM is being throttled or OOM-killed, check its cgroup v2 limits:

cat /sys/fs/cgroup/firecracker-vms//cpu.maxcat /sys/fs/cgroup/firecracker-vms//memory.maxcat /sys/fs/cgroup/firecracker-vms//memory.swap.maxcat /sys/fs/cgroup/firecracker-vms//pids.max

12.6 Firecracker or jailer errors

The VM runner logs to journald:

sudo journalctl -u srv-vm-runner -f

Common issues:

• Jailer chroot setup failures — check that SRV_JAILER_BASE_DIR is on the same filesystem as SRV_DATA_DIR
• Permission errors — verify srv-vm-runner.service keeps User=root, Group=srv, Delegate=cpu memory pids, DelegateSubgroup=supervisor, and no NoNewPrivileges=yes
• /dev/kvm not available — check permissions and that KVM is loaded

12.7 Smoke test

The end-to-end smoke test validates the full host setup:

sudo ./contrib/smoke/host-smoke.sh

Overrides:

On failure, artifacts are written to /var/tmp/srv-smoke/<instance>/ including inspect, logs, systemctl status, and journalctl output.

13. Backup and restore

srv provides in-place backup and restore for stopped VMs. This is useful for checkpointing VMs before risky changes and rolling them back when something goes wrong.

13.1 Create a backup

ssh srv stop demossh srv backup create demo

This copies the current rootfs image into the backup store under SRV_DATA_DIR/backups/demo/<backup-id>/.

13.2 List backups

ssh srv backup list demo

Each backup has a unique ID and timestamp.

13.3 Restore from a backup

ssh srv restore demo 

This replaces the current rootfs with the backup's rootfs. The VM must be stopped.

13.4 Full workflow

# Create and configure a VMssh srv new demossh root@demo  # install packages, configure services# Checkpoint before risky changesssh srv stop demossh srv backup create demossh srv start demo# ... make changes that break things ...# Reset to the checkpointssh srv stop demossh srv backup list demossh srv restore demo ssh srv start demo# Verify the restore worked — the VM should be back to the checkpoint state

13.5 Constraints

• Stopped only: both backup and restore require the VM to be stopped
• In-place only: backups are tied to the original VM record. They cannot be restored onto a newly created VM that reuses the same name
• Single host: backups live on the same host. For cross-host migration, use export/import
• Rootfs only: backups capture the writable rootfs. The kernel and initrd come from the host's current configuration on the next start

13.6 Backup storage

Backups are stored under:

SRV_DATA_DIR/backups///

Monitor disk usage if you create many backups — each one is a full copy of the rootfs at that point in time.

14. Build a custom guest image

The default guest image is an Arch Linux rootfs with Docker, Go, Neovim, OpenCode, Pi, and common development tools. You can customize it by modifying the overlay or building a completely different rootfs.

14.1 Default image builder

The images/arch-base/ directory contains the official builder:

sudo OUTPUT_DIR=/var/lib/srv/images/arch-base ./images/arch-base/build.sh

This produces two artifacts:

• vmlinux — an x86_64 Firecracker-compatible kernel built from the 6.12 LTS source
• rootfs-base.img — a sparse ext4 image populated via pacstrap

14.1.1 What the image includes

• Docker, docker-compose
• Go, gopls, Odin, odinfmt, OLS
• Neovim with a prewarmed LazyVim config (BMW heritage amber theme)
• OpenCode and Pi CLIs with per-VM Zen gateway bootstrap
• Git, fd, ripgrep, tree-sitter-cli, gcc, perf, valgrind
• iptables-nft with IPv4/IPv6 nftables support
• srv-bootstrap.service for Tailscale and MMDS setup

14.1.2 Build overrides

# Change kernel versionsudo KERNEL_VERSION=6.1.167 OUTPUT_DIR=/var/lib/srv/images/arch-base ./images/arch-base/build.sh# Change rootfs size (default 10G)ROOTFS_SIZE=20G sudo OUTPUT_DIR=/var/lib/srv/images/arch-base ./images/arch-base/build.sh# Reduce kernel build parallelismsudo KERNEL_JOBS=2 OUTPUT_DIR=/var/lib/srv/images/arch-base ./images/arch-base/build.sh

14.2 Podman build on non-Arch hosts

If your host is not Arch Linux and does not provide pacstrap:

sudo podman run --rm --privileged --network host \  -v "$PWD":/work \  -v /var/lib/srv/images/arch-base:/var/lib/srv/images/arch-base \  -w /work \  docker.io/library/archlinux:latest \  bash -lc '    set -euo pipefail    pacman -Sy --noconfirm archlinux-keyring    pacman -Syu --noconfirm arch-install-scripts base-devel bc e2fsprogs rsync curl systemd    OUTPUT_DIR=/var/lib/srv/images/arch-base ./images/arch-base/build.sh  '

• --privileged is required because the builder uses losetup, mkfs.ext4, and mount
• --network host keeps mirror and kernel downloads simple

14.3 Customizing the overlay

The images/arch-base/overlay/ directory contains files that are copied into the guest rootfs during the build. Changes here only affect new guests after you rebuild rootfs-base.img and refresh SRV_BASE_ROOTFS.

After modifying the overlay:

sudo OUTPUT_DIR=/var/lib/srv/images/arch-base ./images/arch-base/build.sh# Then update SRV_BASE_ROOTFS in /etc/srv/srv.env to point at the new image

14.4 Rolling out a new image

Rootfs changes only affect newly created guests. Existing guests keep their own writable rootfs.img. There is no host-driven in-place rootfs update.

To migrate an existing guest to a new base image:

1. Rebuild and update SRV_BASE_ROOTFS
2. Create a new guest with ssh srv new <name>
3. Migrate workload data to the new guest
4. Delete the old guest

Or manage the existing guest locally with pacman -Syu, accepting guest-local drift.

14.5 Kernel rollout

Kernel roll-forward is simpler. Existing stopped guests pick up the currently configured SRV_BASE_KERNEL and optional SRV_BASE_INITRD on their next start or restart:

1. Rebuild the kernel artifact
2. Update SRV_BASE_KERNEL (and SRV_BASE_INITRD if applicable) in /etc/srv/srv.env
3. Restart the units if needed
4. Stop and start guests, or let stopped guests pick up the new kernel on their next start

Rollback: point the path back to the previous artifact and restart guests again.

15. Export and import

The export and import commands stream a portable VM artifact between hosts. This is the supported way to move a VM from one srv host to another.

15.1 Export

On the source host:

ssh srv stop demossh srv export demo

The command writes a tar stream to stdout. The artifact contains:

• A versioned manifest
• The writable rootfs.img
• Serial and Firecracker logs when present (each capped to the newest 256 MiB)

15.2 Import

Pipe the export stream directly into the destination host:

ssh srv-a export demo | ssh srv-b import

Import creates the VM under the same name on the destination host, allocates new runtime paths and network state, and leaves the VM stopped.

15.3 Start after import

ssh srv-b start demo

The destination host uses its currently configured SRV_BASE_KERNEL and optional SRV_BASE_INITRD — only the writable disk and optional logs come from the streamed artifact.

15.4 Important semantics

Because the guest's durable Tailscale identity lives in the copied rootfs:

• Do not boot the source and destination VMs at the same time — this would cause a Tailscale key conflict
• Treat export/import as cutover or move semantics, not cloning semantics

The destination host regenerates:

• Absolute file paths (runtime directories)
• TAP device wiring
• Guest MAC address
• VM /30 subnet allocation

The artifact preserves:

• VM name
• Creator identity
• Machine shape (vCPUs, memory, rootfs size)
• Last-known Tailscale name and IP (as cached state)

15.5 Save to a file

You can also save the export to a file for later or offline transfer:

ssh srv export demo > demo-backup.tar

Then on the destination:

cat demo-backup.tar | ssh srv import

16. Host snapshots

The snapshot create command is a host-level disaster-recovery primitive that creates a consistent point-in-time copy of SRV_DATA_DIR.

16.1 How it works

ssh srv snapshot create

This is an admin-only command that:

1. Briefly rejects all other SSH commands
2. Waits for already admitted commands to finish
3. Checkpoints SQLite
4. Flushes the filesystem
5. Creates a readonly btrfs snapshot of SRV_DATA_DIR under SRV_DATA_DIR/.snapshots/<timestamp>

16.2 Snapshot semantics

The snapshot is intentionally simple:

• Control-plane consistent: SQLite state is checkpointed
• Stopped guests fully safe: rootfs data is on disk and consistent
• Running guests crash-consistent only: like pulling the power on a running VM — the filesystem may need journal replay on restore

16.3 Prerequisites

• SRV_DATA_DIR must be a btrfs subvolume root — a plain directory on btrfs is not enough
• The snapshot covers SRV_DATA_DIR only. /etc/srv, environment files, and unit overrides still need your existing operator-managed backup flow

16.4 Use with btrfs send/receive

You can combine snapshots with btrfs send/receive for off-host replication:

# After creating a snapshotsudo btrfs send SRV_DATA_DIR/.snapshots/ | \  ssh backup-host btrfs receive /backup/srv/

Run btrfs send/receive after the local snapshot already exists — the snapshot barrier is not involved in the replication step.

16.5 Not included

• /etc/srv/srv.env — back this up separately
• Systemd unit overrides in /etc/systemd/system/srv*.service.d/
• Firecracker and jailer binaries

See the operations reference for the full host backup and restore workflow.

17. Instance lifecycle

This page covers the full lifecycle of a srv VM from creation to deletion.

17.1 Create

ssh srv new 

With custom sizing:

ssh srv new  --cpus  --ram  --rootfs-size 

--ram and --rootfs-size accept units like 2G, 512M, or plain MiB integers. --cpus must be 1 or an even number, up to 32.

The control plane:

1. Clones the base rootfs as a reflink
2. Allocates a /30 network subnet and TAP device
3. Mints a one-off Tailscale auth key via MMDS
4. Boots the VM through the jailer with cgroup v2 enforcement
5. Polls until the guest reports state: ready

17.2 Inspect

ssh srv inspect 

Shows instance state, vCPU count, memory, rootfs size, network addresses, Tailscale name and IP, and event history.

Machine-readable:

ssh srv -- --json inspect 

17.3 List and status

ssh srv listssh srv status

list shows the VMs visible to the caller: all VMs for admins, or only the caller's own VMs for regular users. status is admin-only and reports host capacity — instance counts plus CPU, memory, and disk headroom.

17.4 Logs

ssh srv logs ssh srv logs  serialssh srv logs  firecrackerssh srv logs -f  serialssh srv logs -f  firecracker

Both log sources are append-only. Always check the newest lines first.

17.5 Stop

ssh srv stop 

Performs a graceful shutdown through Firecracker. The rootfs is preserved on disk.

17.6 Start

ssh srv start 

Boots a previously stopped VM. Stopped guests pick up the currently configured SRV_BASE_KERNEL and optional SRV_BASE_INITRD on their next start.

17.7 Restart

ssh srv restart 

Stops and starts the VM in one command. Also picks up the current kernel and initrd.

17.8 Delete

ssh srv delete 

Removes the VM's rootfs, runtime directory, TAP device, cgroup, and jailer workspace. This is irreversible.

17.9 Warm boot behavior

When a VM that already has tailscaled state reboots (via start or restart), it reuses its existing Tailscale identity instead of minting a new auth key. This is called a warm boot.

On cold boot (first new), the guest bootstrap service reads the Tailscale auth key from MMDS and runs tailscale up --auth-key=... --ssh exactly once.

18. Resize a VM

Resize lets you change the vCPU count, memory, and rootfs size of a stopped VM.

18.1 Prerequisites

• The VM must be stopped — resize is rejected for running VMs
• vCPUs and memory can be increased or decreased as long as the requested values stay within the supported limits
• Rootfs size is grow-only — shrink requests are rejected

18.2 Resize

ssh srv stop demossh srv resize demo --cpus 4 --ram 8G --rootfs-size 20Gssh srv start demo

You can specify any combination of flags. Omitted flags keep the current value:

# Only increase RAMssh srv stop demossh srv resize demo --ram 16Gssh srv start demo

18.3 How it works

• vCPU count: stored in the instance record and applied on the next boot
• Memory: stored in the instance record and applied on the next boot
• Rootfs size: uses resize2fs to grow the ext4 filesystem. The underlying file is expanded first, then the filesystem is grown. This operation only increases the filesystem — it never shrinks

18.4 Limits

vCPU count must be 1 or an even number. CPU and memory changes are applied on the next boot.

19. View logs

srv provides two log sources for each VM: serial output and Firecracker VMM logs.

19.1 Serial log

Shows kernel boot, srv-bootstrap.service, tailscaled, and general guest console output:

ssh srv logs demossh srv logs demo serialssh srv logs -f demo serial

19.2 Firecracker log

Shows VMM lifecycle events, API requests, and microVM-level errors:

ssh srv logs demo firecrackerssh srv logs -f demo firecracker

19.3 Both logs at once (default)

Without a log source argument, ssh srv logs <name> shows the serial log by default.

19.4 Where logs live on the host

Both are append-only. Always check the newest lines first when debugging multiple attempts against the same instance name.

19.5 Systemd logs

The VM runner and network helper also log to journald:

sudo journalctl -u srv-vm-runner -fsudo journalctl -u srv-net-helper -fsudo journalctl -u srv -f

19.6 Debugging a failed VM

When a VM is stuck or has failed:

1. ssh srv inspect <name> — control-plane view, state, and recorded events
2. ssh srv logs <name> serial — guest boot and bootstrap errors
3. ssh srv logs <name> firecracker — VMM errors
4. journalctl -u srv-vm-runner --no-pager — jailer and stop-time cleanup failures
5. Check cgroup limits: cat /sys/fs/cgroup/firecracker-vms/<name>/memory.max

20. Sandboxed AI coding agent

srv makes it straightforward to run AI coding agents in isolated microVMs. Each VM gets its own cgroup limits, per-instance Tailscale identity, and an optional Zen API proxy that injects the host's API key without exposing it inside the guest.

20.1 Create a VM for an agent

ssh srv new agent-1 --cpus 4 --ram 8G --rootfs-size 30G

Wait for it to report ready:

ssh srv inspect agent-1

Look for state: ready and a tailscale-ip.

20.2 Zen API proxy

When SRV_ZEN_API_KEY is configured on the host, srv binds a per-instance HTTP proxy on the guest's gateway IP and port 11434 (configurable via SRV_ZEN_GATEWAY_PORT). The proxy:

• Only accepts requests from that VM's guest IP
• Forwards /v1/... requests to the upstream OpenCode Zen API with the host key injected
• The guest bootstrap writes /root/.config/opencode/opencode.json and Pi config under /root/.pi/agent/ pointing at this gateway

This means the agent inside the VM can use opencode, pi, or any OpenAI-compatible client against http://<gateway-ip>:11434/v1 without ever seeing the real API key.

20.3 Connect the agent

ssh root@agent-1

The preinstalled opencode and pi CLIs are already configured to target the per-VM gateway. If you are using a different agent framework, point its API client at:

http://:11434/v1

The gateway IP is the default route inside the VM. You can read it from the inspect output under host-addr.

20.4 Resource limits

Each VM runs in its own cgroup v2 leaf with:

• cpu.max — capped at the vCPU count
• memory.max — capped at the requested RAM
• memory.swap.max — set to 0 (no swap)
• pids.max — default 512, configurable via SRV_VM_PIDS_MAX

This prevents a misbehaving agent from consuming the entire host.

20.5 Clean up

ssh srv delete agent-1

20.6 Multiple agents

Create as many agent VMs as the host can hold. Each gets independent networking, identity, and resource limits:

ssh srv new agent-2 --cpus 2 --ram 4Gssh srv new agent-3 --cpus 2 --ram 4G

Use ssh srv status to check remaining host capacity before creating more.

21. Dev/test environment

srv's reflink-based cloning makes it fast to spin up a VM from a base image, modify it, and then use backup/restore to reset it to a known-good state.

21.1 Create a dev VM

ssh srv new dev --cpus 4 --ram 8G --rootfs-size 30G

21.2 Install your toolchain

ssh root@dev# Inside the VMpacman -Syupacman -S nodejs npm# ... set up your project ...

21.3 Back up the clean state

Once the VM is configured the way you like, take a checkpoint before making risky changes:

ssh srv stop devssh srv backup create devssh srv start dev

21.4 Wipe and reset

After a bad experiment:

ssh srv stop devssh srv backup list devssh srv restore dev ssh srv start dev

The restore rolls the rootfs back to the exact state captured at backup time. This is fast because it replaces the writable disk image rather than copying file by file.

21.5 Repeat

You can create multiple backups at different points:

ssh srv stop devssh srv backup create dev    # backup 1: cleanssh srv start dev# ... make changes ...ssh srv stop devssh srv backup create dev    # backup 2: with toolchainssh srv start dev

Then restore whichever snapshot you need:

ssh srv stop devssh srv restore dev ssh srv start dev

21.6 Key constraints

• Backups and restores only work on stopped VMs
• Backups are tied to the original VM record — they cannot be restored onto a differently created VM even if the name is reused
• Restore is in-place: it overwrites the VM's current rootfs with the backup's rootfs

22. Isolated service

Each srv VM runs in its own cgroup v2 leaf with its own /30 network subnet, TAP device, and Tailscale identity. This makes it straightforward to run services that need full isolation — container runtimes, network services, databases — without sharing the host's PID, network, or filesystem namespace.

22.1 Run a database

ssh srv new db --cpus 2 --ram 4G --rootfs-size 20G

Once the VM is ready:

ssh root@dbpacman -S postgresql# ... configure and start postgresql ...

The database is now reachable at its Tailscale IP from any other machine on the tailnet.

22.2 Run Docker workloads

The guest image includes Docker and docker-compose:

ssh srv new builder --cpus 4 --ram 8G --rootfs-size 40G

ssh root@builderdocker run ...docker compose up -d

The overlay and br_netfilter kernel modules are preloaded, and nftables supports both IPv4 and IPv6 families.

22.3 Per-VM resource isolation

Each VM is enforced by cgroup v2:

You can verify the live cgroup limits:

cat /sys/fs/cgroup/firecracker-vms//cpu.maxcat /sys/fs/cgroup/firecracker-vms//memory.maxcat /sys/fs/cgroup/firecracker-vms//memory.swap.maxcat /sys/fs/cgroup/firecracker-vms//pids.max

22.4 Per-VM networking

Each VM gets its own /30 subnet, TAP device, and NAT rules. VMs cannot reach each other's private networks. They can reach the host and the internet through MASQUERADE rules.

From any tailnet machine, you can SSH directly to the VM's Tailscale name or IP — no port forwarding needed.

22.5 Clean separation

When the service is no longer needed:

ssh srv delete db

The rootfs, TAP device, cgroup, jailer workspace, and iptables rules are all cleaned up.

23. Throwaway debug VM

One of the most immediate uses for srv is spinning up an isolated Linux environment where you can install packages, run risky commands, or reproduce a bug — then delete it without any trace on the host.

23.1 Create

ssh srv new debug-vm

By default the VM gets 1 vCPU, 1 GiB of RAM, and a 10 GiB rootfs. Adjust if you need more:

ssh srv new debug-vm --cpus 2 --ram 4G --rootfs-size 30G

23.2 Watch it boot

ssh srv logs -f debug-vm serial

You will see the kernel boot, srv-bootstrap.service set up networking and Tailscale, and tailscale up --ssh complete. Once inspect reports state: ready:

ssh srv inspect debug-vm

Look for the tailscale-name and tailscale-ip fields.

23.3 Connect and use

ssh root@debug-vm

The guest image comes with Docker, Go, Neovim, Git, perf, valgrind, and common development tools preinstalled. Install anything else with pacman -S.

23.4 Clean up

When you are done:

ssh srv delete debug-vm

This removes the VM's rootfs, runtime directory, TAP device, cgroup, and jailer workspace. No trace remains on the host.

23.5 Tips

• Use -- --json inspect <name> to get machine-readable output for scripting
• The serial log under ssh srv logs <name> serial is append-only — always check the newest lines
• If a VM gets stuck in provisioning or failed, check ssh srv inspect <name> for the last_error field and ssh srv logs <name> firecracker for VMM errors

24. Networking overview

Every srv VM gets its own isolated network stack: a dedicated /30 subnet, TAP device, and NAT rules. Guest egress is routed through the host's outbound interface after MASQUERADE.

24.1 How it works

Internet    │    │ (host outbound interface)    │┌───┴─────────────────────┐│      Linux host         ││  ┌──────────────────┐   ││  │ iptables MASQ    │   ││  │ + FORWARD rules  │   ││  └────────┬─────────┘   ││           │             ││  ┌────────┴─────────┐   ││  │ TAP device       │   ││  │ (per-VM /30)     │   ││  └────────┬─────────┘   ││           │             ││  ┌────────┴─────────┐   ││  │ Firecracker VM   │   ││  │ gateway = .1     │   ││  │ guest   = .2     │   ││  └──────────────────┘   │└─────────────────────────┘

When ssh srv new demo runs, the network helper:

1. Allocates the next free /30 from SRV_VM_NETWORK_CIDR (default 172.28.0.0/16)
2. Creates a TAP device for the VM
3. Installs MASQUERADE and FORWARD rules for guest egress
4. Configures the gateway address on the host side

The VM's bootstrap configures the guest interface with:

• Gateway: SRV_DATA_DIR-derived host address (first usable IP in the /30)
• Guest IP: second usable IP in the /30
• DNS: configurable via SRV_VM_DNS (default 1.1.1.1, 1.0.0.1)

24.2 Tailscale integration

Each VM gets its own Tailscale identity. The control plane mints a one-off auth key and injects it through Firecracker MMDS metadata. The guest bootstrap service:

1. Reads the MMDS payload
2. Starts tailscaled
3. Runs tailscale up --auth-key=... --ssh on the first authenticated boot
4. Persists tailscaled state for warm reboots

This means any machine on the tailnet can reach the VM by its Tailscale name or IP — no port forwarding needed.

24.2.1 SSH access

Guests expose SSH through Tailscale's --ssh flag, so ssh root@<tailscale-name> works from any tailnet machine. Per-user OpenSSH keys are not injected — Tailscale SSH handles authentication based on tailnet identity.

24.3 Configuration

24.4 IPv4 forwarding

Guest NAT depends on IP forwarding:

sudo tee /etc/sysctl.d/90-srv-ip-forward.conf >/dev/null <<'EOF'net.ipv4.ip_forward = 1EOFsudo sysctl --system

This must stay enabled. Disabling it breaks guest egress.

24.5 Network cleanup

When a VM is deleted, the network helper removes:

• The TAP device
• The MASQUERADE rule
• The FORWARD rule
• The gateway address from the host interface

24.6 Host-side API gateways

srv can also expose host-side HTTP gateways on each VM's gateway IP:

• The Zen gateway on SRV_ZEN_GATEWAY_PORT proxies /v1/... to the configured OpenCode Zen upstream with the host API key injected.
• The generic integration gateway on SRV_INTEGRATION_GATEWAY_PORT proxies /integrations/<name>/... to operator-defined HTTP integrations with host-managed auth or headers injected.

Both gateway types only accept requests from the owning guest IP. See Zen gateway and HTTP integrations.

25. Zen gateway

When SRV_ZEN_API_KEY is configured on the host, srv sets up a per-instance HTTP proxy that allows guest VMs to reach the OpenCode Zen API without storing the real API key inside the guest.

25.1 How it works

┌───────────────────────────────────────────────┐│                     Host                      ││                                               ││  ┌───────────────┐    ┌────────────────────┐  ││  │ srv control   │    │ Zen gateway        │  ││  │ plane         │    │ :11434 on          │  ││  │               │    │ gateway IP         │  ││  └───────────────┘    │                    │  ││                       │ injects            │  ││                       │ SRV_ZEN_API_KEY    │  ││                       │ into Authorization │  ││                       │ header             │  ││                       └─────────┬──────────┘  ││                                 │             ││                    ┌────────────┴──────────┐  ││                    │ upstream:             │  ││                    │ opencode.ai/zen/v1    │  ││                    └───────────────────────┘  │└───────────────────────────────────────────────┘                 │                 │ /30 network                 │       ┌─────────┴─────────┐       │ Guest VM          │       │                   │       │ opencode ->       │       │ http://gateway    │       │ :11434/v1         │       └───────────────────┘

For each VM, srv binds an HTTP proxy on the VM's host/gateway IP address and the configured SRV_ZEN_GATEWAY_PORT (default 11434). The proxy:

• Only accepts requests from that VM's guest IP
• Forwards /v1/... requests to the upstream Zen API
• Injects the host's SRV_ZEN_API_KEY into the Authorization header
• The real key never leaves the host

25.2 Guest bootstrap

When the Zen gateway is enabled, the guest srv-bootstrap.service writes /root/.config/opencode/opencode.json targeting the per-VM gateway:

{  "provider": "opencode",  "apiKey": "local-placeholder",  "baseURL": "http://:11434/v1"}

The apiKey is a local placeholder only so OpenCode keeps Zen's paid model catalog visible — the real credential still lives only on the host and is injected by the proxy.

Bootstrap also writes Pi config under /root/.pi/agent/ so the preinstalled pi CLI uses the same gateway by default:

{  "providers": {    "opencode": {      "baseUrl": "http://:11434/v1",      "apiKey": "srv-zen-gateway"    }  }}

When the gateway is disabled (no SRV_ZEN_API_KEY), bootstrap removes those managed default config files.

25.3 Configuration

25.4 Using the gateway from the guest

The preinstalled opencode and pi CLIs work out of the box. For other agents or HTTP clients:

# Inside the VMcurl http://$(ip route show default | awk '{print $3}'):11434/v1/models

25.5 Disabling the gateway

Remove or leave SRV_ZEN_API_KEY unset. After the next guest boot, the bootstrap service will remove the managed OpenCode and Pi config files.

26. srv Cheatsheet

Quick reference for the srv control plane.

26.1 Commands (via SSH)

ssh srv -- [--json]  [args]

Use --json with the non-streaming instance and backup commands when you need machine-readable output.

26.2 Integrations (admin only)

26.3 Quick Examples

# Create VMssh srv new demossh srv statusssh srv -- --json inspect demo# With sizingssh srv new demo --cpus 4 --ram 8G --rootfs-size 20G# Integrationsssh srv integration add http openai --target https://api.openai.com/v1 --bearer-env SRV_SECRET_OPENAI_PRODssh srv integration enable demo openaissh srv inspect demo# Resize (must be stopped)ssh srv stop demossh srv resize demo --cpus 4 --ram 8Gssh srv start demo# Backup and restore (VM must be stopped)ssh srv stop demossh srv backup create demossh srv backup list demossh srv restore demo # View logsssh srv logs demossh srv logs demo serialssh srv logs demo firecrackerssh srv logs -f demo serialssh srv logs -f demo firecracker# Live VM usagessh -t srv topssh -t srv top --interval 2s

26.4 Systemd Management

# Statussudo systemctl status srv srv-net-helper srv-vm-runner# Restartsudo systemctl stop srv srv-net-helper srv-vm-runnersleep 5sudo systemctl start srv-vm-runner srv-net-helper srv# View logssudo journalctl -u srv -fsudo journalctl -u srv-vm-runner -f

26.5 Smoke Test

sudo ./contrib/smoke/host-smoke.sh

26.6 Build Artifacts

# Binariesgo build ./cmd/srvgo build ./cmd/srv-net-helpergo build ./cmd/srv-vm-runner# Guest imagesudo OUTPUT_DIR=/var/lib/srv/images/arch-base ./images/arch-base/build.sh# Installsudo ./contrib/systemd/install.shsudoedit /etc/srv/srv.envsudo ./contrib/systemd/install.sh --enable-now

26.7 Key Env Variables

26.8 Backup & Restore

# Backupsudo systemctl stop srv srv-net-helper srv-vm-runnersudo tar -czf backup.tar.gz /etc/srv /var/lib/srv# Restoresudo ./contrib/systemd/install.sh# restore /etc/srv/srv.env and /var/lib/srvsudo systemctl daemon-reloadsudo systemctl enable --now srv-vm-runner srv-net-helper srvsudo ./contrib/smoke/host-smoke.sh

26.9 Debug Commands

# VM inspectionssh srv inspect # Recent logsssh srv logs  serialssh srv logs  firecracker# System logsjournalctl -u srv-vm-runner | tail# Check cgroupscat /sys/fs/cgroup/firecracker-vms//cpu.maxcat /sys/fs/cgroup/firecracker-vms//memory.max

26.10 Notes

• VM disks at: SRV_DATA_DIR/instances/<name>/rootfs.img
• VM backups live at: SRV_DATA_DIR/backups/<name>/<backup-id>/
• Resize requires a stopped VM; CPU and RAM can go up or down within limits, but rootfs shrink is rejected
• Resize only works on stopped VMs
• Backup and restore only work on stopped VMs and only restore onto the original VM record, not a newly recreated VM with the same name
• Creators manage their own VMs; admins manage all VMs
• Integration commands are admin-only, and integration secrets are referenced by host env name rather than passed as raw SSH arguments
• Warm start/restart reuses tailscaled state
• Host reboot auto-restarts active VMs