Setup

Four steps: a server, a private network, the CLI on whatever submits work, and the machines. Only the server needs a public address.

1. The server

A small VPS is enough, the server schedules and does no computing itself. 4 GB of memory runs Postgres, dstack and Caddy with room to spare. The repository's deploy/ directory holds the whole stack as one compose file.

Once, on a fresh server
scp deploy/bootstrap.sh root@<vps>:/tmp/
ssh root@<vps> "bash /tmp/bootstrap.sh"

The bootstrap installs Docker, adds swap, and writes /opt/falllow/.env with a generated database password and admin token. Point DNS at the server, then publish a release, or run the Deploy workflow by hand: it builds the site and the manager, syncs the stack and waits until the server answers.

The admin token in the .env is the server's own, and what you sign in to the manager with the first time. It is only read when the server first starts against an empty database; after that the database holds it. Everything else, each device, agent and CI repository, gets a consumer of its own under Admin, Access: a token that can be replaced or deactivated without touching the rest, and runs listed under its name.

2. The network

The server reaches machines over SSH, so it has to be able to reach them. A rented box has a public address, a workstation at home does not. So every machine opens a WireGuard tunnel to the server and keeps it alive, and the server reaches the machine through it. Nothing forwards a port, nothing needs an account, and there is no third party in the path: WireGuard is in the Linux kernel.

Once, on the server
bash deploy/wg/server.sh

That is the hub, 10.7.0.1. Machines get 10.7.0.2 and up as they join, and keep their address wherever they are.

3. The CLI

On every machine that submits work: your laptop, a CI runner, an agent's sandbox. It needs Python 3.10 or newer and brings dstack's client with it.

Install and log in
uv tool install "git+https://github.com/milanofthe/falllow#subdirectory=cli"
falllow login --server https://app.falllow.com --token <token>

The login is checked before it is stored, in ~/.dstack/config.yml, so the dstack command that comes with it talks to the same server. Where nothing should be written to disk, as in CI, FALLLOW_SERVER and FALLLOW_TOKEN in the environment take the place of the login. That is all a machine that submits work needs: the working directory is shipped from it, nothing is cloned.

4. The machines

A machine needs Linux and an account with sudo. A dual boot workstation joins while it is booted into Linux. It joins by invitation: an admin asks for one from anywhere the CLI is logged in with an admin token, and gets one line to run on the machine.

Anywhere, as admin
$ falllow pool invite workstation --data /mnt/data
falllow: run on workstation as a sudo user within 24h:
curl -fsSL -X POST https://app.falllow.com/join/<token> | sudo bash
falllow: sets up network, Docker and tools, then joins the fleet

Run the line on the machine. It installs WireGuard and Docker, brings up the tunnel with an address of its own, adds the server's fleet key and prepares the machine for jobs. Its last step tells the server, which writes the machine into the pool file and applies the fleet. Nothing is edited by hand, and nobody needs a shell on the server.

The invitation is good for a day and until the machine has joined. The script is fetched with POST, so a chat client that previews a pasted link gets a sentence rather than the machine's network key. --user names the account the server logs in as, with passwordless sudo, for machines where root has no login (default root). --data names a directory on the disk jobs should use: images, container layers and the pool's own directories move there. --blocks sets how finely the machine is shared, one block per core unless told otherwise.

Check it
falllow devices
falllow run --no-workspace -- nproc

falllow pool remove <name> takes a machine out of the pool file, the fleet and the network. Both inviting and removing need an admin token.

A Windows PC

A Windows PC joins inside a WSL distribution of its own, through deploy/windows/falllow-windows.ps1 (falllow-windows below), with a budget its owner sets. The distribution gets only what it is given: a number of cores, an amount of memory, a disk limit, and the GPU if its owner shares it. It mounts no Windows drive and cannot start Windows programs.

On the PC, in PowerShell
falllow-windows setup                                   # once: WSL, Docker, limits, autostart
falllow pool invite home-pc --user falllow              # anywhere the CLI is logged in as admin
falllow-windows join "<the line it printed>"
falllow-windows budget -Cores 8 -MemoryGB 24 -Gpu off   # any time
falllow-windows status                                  # what the pool gets, what runs now
falllow-windows pause                                  # out of the pool until resume
falllow-windows resume
falllow-windows always-on                               # in the pool from boot, nobody logged in

Cores and memory are limits of WSL as a whole; the disk is capped at 150 GB unless -DiskGB says otherwise. A new budget restarts WSL, is refused while a pool job runs there unless -Force ends it, and the server offers the PC at its new size within a few minutes. Without always-on the PC is in the pool while somebody is logged in; with it, the PC never sleeps on mains power and rejoins after a restart without anyone logging in. Sharing the GPU makes the PC a single slot: dstack splits a machine by its GPUs, so with one GPU one job at a time runs there, a CPU-only test included.

The pool file

The machines are written down once, in the server's pool file, /opt/falllow/pool.yml. dstack's fleet is generated from it and the power supervisor reads the same file. Joining and removing change it; for anything else, such as a machine that can be switched off, edit it on the server and apply it there. deploy/pool.yml in the repository only seeds a new server.

/opt/falllow/pool.yml
# /opt/falllow/pool.yml, on the server
fleet: pool

ssh:
  user: root
  identity_file: /opt/falllow/pool_fleet

machines:
  - name: workstation          # always on: nothing more to say
    address: 10.7.0.2
    user: root
    blocks: auto

  - name: rented               # switched on when a job needs it, off when idle
    address: 10.7.0.3
    user: root
    blocks: auto
    power:
      provider: command
      status: hcloud server describe rented -o format='{{.Status}}'
      running_when: running
      start: hcloud server poweron rented
      stop: hcloud server poweroff rented
      stop_after_idle: 2h

On the server, deploy/falllow.sh is the CLI signed in as admin and pointed at this file, the same one the server's timers run.

Apply it, on the server
bash /opt/falllow/deploy/falllow.sh pool apply   # the fleet now matches the file
bash /opt/falllow/deploy/falllow.sh pool show    # power, reachability, blocks in use, hours left

blocks is how finely a machine is shared, not a limit on one job: auto is one block per core, and a job takes as many blocks as what it asked for needs. The SSH key is the fleet's own, /opt/falllow/pool_fleet on the server, never anybody's personal one. Changes dstack cannot make in place, a changed address or block count, need falllow pool apply --replace, which rebuilds the fleet while the machines keep running and is refused while any run is still going.

A machine with a power section is switched on when a waiting job needs it and off once it has idled for stop_after_idle; budget_hours caps its running time, for metered or sponsored machines: once it is spent the machine is switched off, jobs on it included, which end with 170. The command provider runs shell commands for status, start and stop, which covers any host with a CLI, an IPMI or a smart plug; gcp drives a Compute Engine instance with a service account key. A machine that stops answering is shown as unreachable and gets no new work until it is back.

Updates

A release updates the server and then every machine that answers and has nothing running: the pool's tools, the cleanup that keeps its disk in bounds, and the commands the server uses to collect artifacts. A machine that is off or busy keeps what it has, which still works, until the server sees it idle, which it checks every quarter of an hour.

On the server
bash /opt/falllow/deploy/machine/update.sh            # every machine not yet on this release
bash /opt/falllow/deploy/machine/update.sh rented     # only this one

What a machine may keep is set in /etc/falllow/machine.conf on the machine: 50 GB for named caches, 30 GB for what jobs download into /pool, and artifacts for 14 days, unless the file says otherwise. An update keeps what is there.

Backups

The server dumps its database every night into /var/backups/falllow and keeps 14 days. Beside each dump it writes the configuration a new server could not be rebuilt without: the .env, dstack's encryption key, the pool file, the fleet's and the server's other keys, and the WireGuard configuration. Both stay on the server's own disk, so an off-site copy is what protects against losing the machine.