Setup
Four steps: a server, a private network, the CLI on whatever submits work, and the machines. Only the server needs a public address.
1. The server
A small VPS is enough, the server schedules and does no computing itself. 4 GB of memory runs
Postgres, dstack and Caddy with room to spare. The repository's deploy/ directory
holds the whole stack as one compose file.
scp deploy/bootstrap.sh root@<vps>:/tmp/
ssh root@<vps> "bash /tmp/bootstrap.sh"The bootstrap installs Docker, adds swap, and writes /opt/falllow/.env with a
generated database password and admin token. Point DNS at the server, then publish a release, or
run the Deploy workflow by hand: it builds the site and the manager, syncs the stack and waits
until the server answers.
The admin token in the .env is the server's own, and what you sign in to the manager
with the first time. It is only read when the server first starts against an empty database;
after that the database holds it. Everything else, each device, agent and CI repository, gets a
consumer of its own under Admin, Access: a token that can be replaced or deactivated without
touching the rest, and runs listed under its name.
2. The network
The server reaches machines over SSH, so it has to be able to reach them. A rented box has a public address, a workstation at home does not. So every machine opens a WireGuard tunnel to the server and keeps it alive, and the server reaches the machine through it. Nothing forwards a port, nothing needs an account, and there is no third party in the path: WireGuard is in the Linux kernel.
bash deploy/wg/server.shThat is the hub, 10.7.0.1. Machines get 10.7.0.2 and up as they
join, and keep their address wherever they are.
3. The CLI
On every machine that submits work: your laptop, a CI runner, an agent's sandbox. It needs Python 3.10 or newer and brings dstack's client with it.
uv tool install "git+https://github.com/milanofthe/falllow#subdirectory=cli"
falllow login --server https://app.falllow.com --token <token>The login is checked before it is stored, in ~/.dstack/config.yml, so the dstack command that comes with it talks to the same server. Where nothing should
be written to disk, as in CI, FALLLOW_SERVER and FALLLOW_TOKEN in the
environment take the place of the login. That is all a machine that submits work needs: the
working directory is shipped from it, nothing is cloned.
4. The machines
A machine needs Linux and an account with sudo. A dual boot workstation joins while it is booted into Linux. It joins by invitation: an admin asks for one from anywhere the CLI is logged in with an admin token, and gets one line to run on the machine.
$ falllow pool invite workstation --data /mnt/data
falllow: run on workstation as a sudo user within 24h:
curl -fsSL -X POST https://app.falllow.com/join/<token> | sudo bash
falllow: sets up network, Docker and tools, then joins the fleetRun the line on the machine. It installs WireGuard and Docker, brings up the tunnel with an address of its own, adds the server's fleet key and prepares the machine for jobs. Its last step tells the server, which writes the machine into the pool file and applies the fleet. Nothing is edited by hand, and nobody needs a shell on the server.
The invitation is good for a day and until the machine has joined. The script is fetched with
POST, so a chat client that previews a pasted link gets a sentence rather than the machine's
network key. --user names the account the server logs in as, with passwordless
sudo, for machines where root has no login (default root). --data names a directory on the disk jobs should use: images, container layers and the pool's own
directories move there. --blocks sets how finely the machine is shared, one block
per core unless told otherwise.
falllow devices
falllow run --no-workspace -- nprocfalllow pool remove <name> takes a machine out of the pool file, the fleet
and the network. Both inviting and removing need an admin token.
A Windows PC
A Windows PC joins inside a WSL distribution of its own, through deploy/windows/falllow-windows.ps1 (falllow-windows below), with a
budget its owner sets. The distribution gets only what it is given: a number of cores, an amount
of memory, a disk limit, and the GPU if its owner shares it. It mounts no Windows drive and
cannot start Windows programs.
falllow-windows setup # once: WSL, Docker, limits, autostart
falllow pool invite home-pc --user falllow # anywhere the CLI is logged in as admin
falllow-windows join "<the line it printed>"
falllow-windows budget -Cores 8 -MemoryGB 24 -Gpu off # any time
falllow-windows status # what the pool gets, what runs now
falllow-windows pause # out of the pool until resume
falllow-windows resume
falllow-windows always-on # in the pool from boot, nobody logged inCores and memory are limits of WSL as a whole; the disk is capped at 150 GB unless -DiskGB says otherwise. A new budget restarts WSL, is refused while a pool job runs
there unless -Force ends it, and the server offers the PC at its new size within a
few minutes. Without always-on the PC is in the pool while somebody is logged in;
with it, the PC never sleeps on mains power and rejoins after a restart without anyone logging
in. Sharing the GPU makes the PC a single slot: dstack splits a machine by its GPUs, so with one
GPU one job at a time runs there, a CPU-only test included.
The pool file
The machines are written down once, in the server's pool file, /opt/falllow/pool.yml. dstack's fleet is generated from it and the power
supervisor reads the same file. Joining and removing change it; for anything else, such as a
machine that can be switched off, edit it on the server and apply it there. deploy/pool.yml in the repository only seeds a new server.
# /opt/falllow/pool.yml, on the server
fleet: pool
ssh:
user: root
identity_file: /opt/falllow/pool_fleet
machines:
- name: workstation # always on: nothing more to say
address: 10.7.0.2
user: root
blocks: auto
- name: rented # switched on when a job needs it, off when idle
address: 10.7.0.3
user: root
blocks: auto
power:
provider: command
status: hcloud server describe rented -o format='{{.Status}}'
running_when: running
start: hcloud server poweron rented
stop: hcloud server poweroff rented
stop_after_idle: 2hOn the server, deploy/falllow.sh is the CLI signed in as admin and pointed at this
file, the same one the server's timers run.
bash /opt/falllow/deploy/falllow.sh pool apply # the fleet now matches the file
bash /opt/falllow/deploy/falllow.sh pool show # power, reachability, blocks in use, hours leftblocks is how finely a machine is shared, not a limit on one job: auto is one block per core, and a job takes as many blocks as what it asked for
needs. The SSH key is the fleet's own, /opt/falllow/pool_fleet on the server,
never anybody's personal one. Changes dstack cannot make in place, a changed address or block
count, need falllow pool apply --replace, which rebuilds the fleet while the
machines keep running and is refused while any run is still going.
A machine with a power section is switched on when a waiting job needs it and off
once it has idled for stop_after_idle; budget_hours caps its running
time, for metered or sponsored machines: once it is spent the machine is switched off, jobs
on it included, which end with 170. The command provider runs shell commands
for status, start and stop, which covers any host with a CLI, an IPMI or a smart plug; gcp drives a Compute Engine instance with a service account key. A machine that
stops answering is shown as unreachable and gets no new work until it is back.
Updates
A release updates the server and then every machine that answers and has nothing running: the pool's tools, the cleanup that keeps its disk in bounds, and the commands the server uses to collect artifacts. A machine that is off or busy keeps what it has, which still works, until the server sees it idle, which it checks every quarter of an hour.
bash /opt/falllow/deploy/machine/update.sh # every machine not yet on this release
bash /opt/falllow/deploy/machine/update.sh rented # only this oneWhat a machine may keep is set in /etc/falllow/machine.conf on the machine: 50 GB
for named caches, 30 GB for what jobs download into /pool, and artifacts for 14
days, unless the file says otherwise. An update keeps what is there.
Backups
The server dumps its database every night into /var/backups/falllow and keeps 14
days. Beside each dump it writes the configuration a new server could not be rebuilt without:
the .env, dstack's encryption key, the pool file, the fleet's and the server's other
keys, and the WireGuard configuration. Both stay on the server's own disk, so an off-site copy
is what protects against losing the machine.