CLI
falllow is how work is submitted and followed. Progress goes to stderr and the
command's output to stdout, so it pipes the same as the local command. Every command takes --json, and every command explains itself with examples in falllow <command> --help.
Commands
| Command | What it does |
|---|---|
login | Store the server and a token for this machine, checked before it replaces a login that works. |
login --consumer [NAME] | Make a consumer with the current admin login and log in as it, named after the host by default. --gh-secret OWNER/REPO puts its token into that repository's FALLLOW_TOKEN secret instead. |
run | Submit a job and stream it. Everything after -- is the command. |
ps | Recent runs with status, exit code and machine, everyone's unless narrowed with --mine or --tag. |
logs | Print a run's output, or -f to follow it and exit with its code. |
wait | Block until a run settles, then exit with its code. --notify URL posts the outcome. |
show | One run in detail, including why it did not complete. |
fetch | Bring back what a run wrote to /artifacts, and its log. --list only lists, -o DIR picks where. |
cancel | Stop runs, by name or id, or every active one picked by --mine or --tag. --abort kills instead of stopping gracefully. |
queue | What is waiting for a machine, in the server's order, when each is tried next and how much of a machine it needs. |
bump | Move a waiting run to the front, at --priority 90 unless told otherwise. |
devices | The machines: up, asleep or unreachable, cores, memory, blocks in use, disk free where jobs write, and budget. |
cache ls | The named caches on every machine that is on, with their size. |
cache rm NAME | Delete a cache on every machine, or on one with --machine. |
rm | Delete a run's artifacts, the server's copy and the machine's. |
pool invite NAME | Print the line that takes a machine into the pool. --user, --data, --blocks. Admin token. |
pool remove NAME | Take a machine out of the pool, the fleet and the network. Admin token. |
pool prune | Have every machine that is on drop the images nothing uses, within ten minutes and once nothing runs there. |
pool show, pool apply | The pool file's machines with power and budget, and making the fleet match it. Run on the server. |
Running things
# stream it to completion, exit with its code
falllow run -- pytest -q
# ask for more of a machine
falllow run --cpu 8 --memory 16G -- cargo test --release
# a GPU and an image with the toolkit in it
falllow run --gpu 24GB --image nvidia/cuda:12.4.1-devel-ubuntu22.04 -- ./train.sh
# a cache that outlives the run, on that machine
falllow run --cache cargo-target -- cargo build --release
# no sources needed
falllow run --no-workspace -- nproc
# hand back the run name and carry on
falllow run --detach -- ./long-solve.sh| Flag | Meaning |
|---|---|
--cpu | Cores the job needs at least, or a range like 4..16: as many as one machine has free, within it. The job sees what it got as FALLLOW_CORES. Default 1. |
--memory | Memory it needs at least, e.g. 8G. Default 1G. |
--disk | Disk it needs at least. Default 10G. Said when no machine has that much free where jobs write. |
--gpu | A GPU, by memory or name: 24GB, A100:2. |
--image | Container image to run in. Without it, Ubuntu with build tools, dstack's base image. |
--env, -e | KEY=VALUE, or a bare KEY to pass on its value here. Repeatable. |
--cache | Mount the machine's cache NAME at /cache/NAME, kept between runs. Lower case letters, digits and dashes. Repeatable. |
--timeout | Wall clock limit once running, e.g. 30m or 2h. Default 7d. |
--queue-timeout | How long to wait for a free machine. Default 1d. |
--priority | 0 to 100, higher is placed first when machines are busy. |
--machine | Run only on this machine, named as falllow devices shows it. Waits while it is busy. Repeatable, for any of several. |
--shard | Run this many copies, each told its FALLLOW_SHARD_INDEX. |
--name | Run name. Generated from the command, plus a random suffix, if left out. |
--tag | KEY=VALUE, a label for ps --tag and cancel --tag. host, repo and session are added. Repeatable. |
--notify | POST the outcome as JSON to this URL when the run settles, from the server. Also on wait, from there. |
--detach, -d | Print the run name and return at once. With --json, a list of each run's name and id. |
--include | Ship this path even when an ignore rule excludes it. Repeatable. |
--no-vcs-ignore | Ship the directory as it lies, without git's ignore rules. |
--no-workspace | Ship no sources, for commands that need none. |
A job that no machine in the pool could ever take, more cores or memory than any machine has, a
GPU where there is none, an image with no build for the pool's architecture, is refused before
anything is sent. When the memory asked for, not the cores, decides how much of a machine a job
takes, run says so, since that is what makes a job wait while cores are free.
The working directory
What is on disk is what runs. The enclosing git repository, or the directory itself if there is
none, is packed on your machine and unpacked on the pool's, committed or not. Nothing is cloned
from anywhere, so the pool needs no way into GitHub and no token travels with the job. A home
directory or a filesystem root is refused rather than packed whole; run from the project's
directory, or pass --no-workspace.
.git always stays home, so git describe does not work inside a job,
and so does everything .dstackignore names. .gitignore decides inside
a git repository, where it means build output, caches and virtualenvs. Outside a repository it
does not: a directory assembled by hand is meant to travel as it lies, and a .gitignore copied in with the sources would otherwise keep excluding files where
there is no git at all.
What a rule leaves out is named when the job is sent, not discovered as a missing file in a
container. --include PATH ships a path however it was ignored, repeatable, and --no-vcs-ignore ships the whole directory without git's rules: for the wheel built
for the pool, the model, the data package.
$ falllow run -- ./build.sh
falllow: shipping /home/me/project
falllow: not shipped: remote/_wheel/, target/; override with --include PATHFrom 50 MB on, run says how much travels and which top level entries make up most
of it. The server accepts 256 MB per upload unless configured otherwise; a tree bigger than that
wants a .dstackignore, or the large part can stay on the machine instead of
travelling with every job. Packing stops at 1 GB in any case.
The directory lands at /workspace, and the command runs from the subdirectory you
typed it in: falllow run -- make in crates/solver runs in /workspace/crates/solver.
Sharding
--shard submits several independent jobs rather than splitting the work itself, only
your command knows how to divide it. Each shard is told its slice and decides what that means.
The working directory is packed once and the same snapshot goes to every shard.
Each shard is a run of its own, so --cpu and --memory are what one
shard gets, not what they share: eight shards at --memory 4G ask the pool for
32 GB in total. They do not have to be on the same machine, or run at the same time; whatever
does not fit waits its turn. A shard that exceeds its memory is killed on its own, and the
others carry on.
falllow run --shard 8 -- pytest -qimport os
index = int(os.environ.get("FALLLOW_SHARD_INDEX", 0))
count = int(os.environ.get("FALLLOW_SHARD_COUNT", 1))
mine = [t for i, t in enumerate(tests) if i % count == index]The shards queue like any other jobs and run wherever there is room. Their output is printed one shard after another as they finish, since interleaved lines from eight jobs read as none. The exit code is the first genuine failure among them; an infrastructure failure only surfaces when no shard actually failed. If sending the group fails halfway, the shards already sent are stopped.
Waiting for a machine
Before anything is sent, falllow run says whether the job starts now and where, or
what it waits for: how many blocks of the machine closest to taking it, how many are free, its
place in the queue, and the flags that would make it start now. While it waits it says the same
again whenever that changes.
$ falllow run --cpu 16 --memory 40G -- ./solve.sh
falllow: waiting: needs 18/20 blocks on milan-pc (memory-bound, 2.4 GB/block), 6 free; queue position 2, behind sweep-sh-9c1e4b7a
falllow: starts now with --cpu 12 --memory 23G (strato-1)--cpu 4..16 asks for at least four cores and as many up to sixteen as one machine
has free when it is sent, and the job is told what it got in FALLLOW_CORES. A job
that scales with its cores starts now with what there is rather than waiting for all of it.
Memory has no range: a job given less than it needs is killed, one given fewer cores only takes
longer.
A job that finds every fitting machine busy waits rather than failing, for up to --queue-timeout. It is tried again within seconds of a machine it may use having
room for it, in queue order. Besides that, dstack tries a waiting run after a delay that grows:
15s, 30s, 1m, 2m, 5m, then every 10 minutes. falllow queue shows what is waiting,
in queue order, when each one is due to be tried again at the latest, and how many blocks it
needs. A machine that is being
switched on for it, or cannot be, is said while run waits.
$ falllow queue
# RUN PRIO WAITING TRIES NEXT TRY NEEDS
1 sweep-sh-9c1e4b7a 90 17m 7 12s 16 cores, 32 GB: 16 blocks
2 pytest-77c2e05b 0 12m 6 4m 2 cores, 16 GB: 4 blocks (memory-bound)
falllow: move to front: falllow bump <run>Order is priority first, then whoever has waited longest. --priority 90 on run asks to be placed before the rest, and falllow bump <run> does the same for something already waiting: it queues
the same job again at a higher priority and stops the copy that was waiting. Nothing is
uploaded twice, the archive already on the server is reused, and the run gets a new id.
Free blocks go to the waiting runs in that order, and a run that does not fit leaves them to the
next one that does. A new run is placed the moment it arrives, so priority is a strong
preference rather than a guarantee.
A large run would still lose to small ones: they are placed the moment they arrive, and it needs
many blocks free at once. So once the head of the queue has
waited 15 minutes, falllow run holds back a new run that would take blocks the head
needs, says so, and sends it once the head has started. Nothing is sent while it is held, so
Ctrl-C gives up cleanly, and a run still held after --queue-timeout exits
170 with nothing sent. A run with a higher --priority than the head
is not held, nor is one for machines the head cannot run on, its --machine included.
With -d nothing is held: the run is sent at once, with a line naming the run it
would have waited for. Until the 15 minutes are up the pool is shared as before, so a quick test
does not queue behind a large run that has only just arrived.
$ falllow run --cpu 2 -- pytest -q
falllow: held for sweep-sh-9c1e4b7a (waiting 17m, needs these blocks); nothing sent yet; skip with --priority 91
falllow: releasedImages
--image runs the job in an image of your own. Left to itself, the server would read
that image's configuration from its registry on every submission, and registries limit how often
anonymous callers may do that. So falllow run looks up the two things the server
would ask for, the user the image runs as and the platforms it is built for, once a week on your
machine (~/.cache/falllow/images.json), and puts them in the run. Several agents on
one machine share that file. Should a registry still answer 429, run waits and
tries again for about eight minutes before exiting 170.
What a job leaves behind
Every run gets a directory of its own on the machine it runs on, mounted at /artifacts and named in FALLLOW_ARTIFACTS. It outlives the container,
so what a job wrote is still there when the run is over, including when the run failed. Nothing
is collected by pattern: what the job put there is what it kept.
falllow fetch <run> brings it back, together with the run's log, into artifacts/<run>/. It reads through the pool's server, so it works from a
laptop or from CI without being on the pool's network. A transfer that breaks off writes nothing
and exits 170; fetching again is the remedy.
Each run writes into a directory of its own, its name plus a token, and the server files
what it collects under the run's id. Names repeat: --name nightly is the same
name every night, and dstack frees a name once its run has ended, so a directory keyed by
the name alone would let one night's results overwrite the last. falllow fetch <run> takes a name, meaning the most recent run that had it,
or an id from --json, meaning one particular run for ever.
Once a run settles the server collects its artifacts onto its own disk, so fetching keeps
working after the machine is switched off, replaced or gone. The machine keeps its own copy for
14 days. A run too large to collect stays where it was made and says so; the collection as a
whole stays inside a budget, oldest dropped first, and falllow devices prints what
it takes up. falllow rm <run> deletes both copies.
$ falllow fetch sweep-sh-9c1e4b7a
falllow: 4 files in artifacts/sweep-sh-9c1e4b7a
$ ls artifacts/sweep-sh-9c1e4b7a
falllow.log result.json plots/sweep.png plots/error.pngThe environment a job sees
| Path | What it is |
|---|---|
/workspace | The shipped directory. The command starts in the subdirectory it was typed in. |
/artifacts | The run's own directory on the machine, which outlives the container. falllow fetch brings it back. |
/scratch | Room to work, on the machine's data disk where it has one, the run's own and removed once it ends. Large intermediate files belong here rather than in the container's own filesystem. |
/cache/<name> | For each --cache NAME: kept on that machine across runs and shared by every
run that names it, for what is slow to build and the same each time. Per machine, so a
run elsewhere finds it empty. Runs that write it at once should write to a temporary name
and rename. Each machine keeps 50 GB of caches, dropping the least recently changed. |
/pool | The machine's tools: Rust, uv and sccache, on the PATH, with cargo's, uv's,
pip's and sccache's caches pointed here, so a second build on the same machine reuses
what the first downloaded and compiled. Only where the image does not set its own, so an
image with a toolchain keeps it. What a job installs here with cargo install or uv tool install stays for the next run on that machine. |
| Variable | Value |
|---|---|
FALLLOW_CORES | The cores asked for. Pass it on: make -j$FALLLOW_CORES. |
FALLLOW_MEMORY_MIB | The memory asked for, in MiB. free reports the whole machine, so size by this. |
FALLLOW_ARTIFACTS | /artifacts, where to put what should outlive the job. |
FALLLOW_SCRATCH | /scratch, room to work. |
FALLLOW_CACHE | /cache, when the run asked for a cache. |
FALLLOW_NOTIFY | The --notify URL, when one was given. |
MAKEFLAGS, CARGO_BUILD_JOBS, OMP_NUM_THREADS, and friends | Set to the same number, so tools that read them keep to the share the job was
given. A job gets a CPU quota, not a smaller machine, so nproc still
answers with the whole machine. Anything passed with -e wins. |
FALLLOW_SHARD_INDEX | This shard's index, zero based. Only when sharding. |
FALLLOW_SHARD_COUNT | How many shards there are. |
FALLLOW_NO_POOL | Not set by falllow. Pass -e FALLLOW_NO_POOL=1 to leave /pool's tools and caches out of the job's environment. |
A job asks for 1 core, 1 GB of memory and 10 GB of disk unless told otherwise. dstack's own defaults are 2 cores, 8 GB and 100 GB, which would keep small jobs off small machines.
For scripts and agents
The contract is the exit code and --json; nothing needs parsing from the human
output. The simplest use is the blocking one: falllow run in place of the command,
then branch on the exit code.
falllow run --cpu 4 -- pytest -q
case $? in
0) echo "green" ;;
170) echo "the pool failed, not the tests, safe to run again" ;;
171) echo "failed with its status lost, check the disk first" ;;
172) echo "falllow refused it, read the message" ;;
137) echo "killed, most likely out of memory: ask for more" ;;
130) echo "cancelled, or this process was interrupted" ;;
*) echo "the tests failed" ;;
esacWhat a job takes, it takes from everyone else: blocks asked for and not used are blocks
another job waits for. So after a run, falllow run says what it used of what it asked
for, and what would have done; before one, it says when the same command from the same
repository used far less the last times. The server keeps every run's peaks for 90 days,
since dstack drops a running job's samples after an hour. --json has the same under usage, falllow show prints it, and the manager shows it with the job.
falllow: last 4 runs: max 2.6 cores, 1.6 GB; enough: --cpu 3 --memory 2G
...
falllow: usage: cpu 1.9 avg, 2.6 max of 16; memory 1.6 max of 32 GB
falllow: enough: --cpu 3 --memory 2GFor anything long, start it detached and come back. The run name is the handle; every command
takes it, or the run's id, and --json gives the same object everywhere.
name=$(falllow run -d -- ./long-solve.sh) # prints the run name at once
falllow show "$name" --json # status, exit_code (null until settled), machine
falllow wait "$name" --json # blocks, prints the outcome, exits with its code
falllow logs "$name" -f # the output, following
falllow fetch "$name" # what it wrote to /artifacts, and its log
falllow run -d --json -- ./long-solve.sh # [{"name": "long-solve-sh-...", "id": "..."}]$ falllow wait --json cargo-test-3f9a1c2d
{
"name": "cargo-test-3f9a1c2d",
"id": "8f0e2c4a-6d1b-4f3e-9a57-2b8c1d0e4f6a",
"status": "failed",
"exit_code": 1,
"exit_status": 1,
"termination_reason": "container_exited_with_error",
"submitted_at": "2026-09-16T09:38:50.980859+00:00",
"finished_at": "2026-09-16T09:39:32.780373+00:00",
"instance": "10.7.0.3",
"attempts": 1,
"tags": {"host": "laptop", "repo": "solver", "session": "agent-7"}
}exit_code is what falllow run would have exited with, null while the
run is unsettled. exit_status is the command's raw status if it got to run. termination_reason says why a run ended when it was not the command's doing.
Several agents share the pool, and falllow ps shows everyone's runs. Each run is
tagged with the machine it came from and the directory it shipped. An agent that sets FALLLOW_SESSION once gets its runs tagged with that too, and --mine on ps and cancel then means this machine and this session. --tag KEY=VALUE adds a label of your own and selects by it.
export FALLLOW_SESSION=agent-7 # once, per agent or terminal
falllow run -d --tag study=gallery -- ./sweep.sh
falllow ps --mine # this machine and this session
falllow ps --tag study=gallery --json
falllow cancel --mine # everything this session still has going
falllow cancel --tag study=galleryA line on stderr that starts with falllow: pool problem: means a part of the pool
itself is broken: a full disk, a service that is down, a health check that has stopped. The
same lines appear on run, ps, queue and devices and above every page of the manager. They do not change the exit code; pass
them on to whoever runs the pool.
To be told rather than to poll, --notify on run posts the outcome to a
URL when the run settles. The URL is carried on the run and the pool's server does the posting,
so a job started with -d is announced long after the terminal that started it is
closed; a server that is not sending says so when the run is submitted. The body is the same
JSON --json prints, plus a one line text for webhooks that show a
message and nothing else. On wait the flag posts from the waiting process instead,
since a run already under way cannot be told where to report.
falllow run -d --notify https://hooks.example.com/falllow -- ./sweep.sh
# the server POSTs the same JSON that --json prints, once the run settles,
# whether or not anything is still attached$ falllow ps --mine
NAME STATUS EXIT INSTANCE SUBMITTED
sweep-sh-9c1e4b7a-1 running 10.7.0.2 2026-09-16T09:41:55
cargo-test-3f9a1c2d failed 1 10.7.0.3 2026-09-16T09:38:50
pytest-77c2e05b done 0 10.7.0.3 2026-09-16T09:38:26Exit codes
| Code | Meaning |
|---|---|
0 | The command succeeded. |
| anything else | The command's own exit code, passed through unchanged. |
130 | Somebody cancelled the run, or this command was interrupted: Ctrl-C or a SIGTERM from a caller's timeout detaches and the run goes on, and the message says how to pick it up. |
137 | The kernel killed the container, nearly always because it used more memory than it asked for. |
170 | Reserved: the pool failed, not your command. No machine free in time, a machine lost, the timeout passed, the server out of reach, a registry still rate limiting. Running it again may help. |
171 | Reserved: your command failed, but its exit status was lost with the container. So far always a full disk, so running it again as it is fails the same way. |
172 | Reserved: falllow turned the request down and nothing ran. Not logged in, no machine that could ever fit it, a name in use, no such run, a value it cannot read, a directory too large to ship. The message says which; the same request gets the same answer. |
Ctrl-C while following detaches and leaves the job running; falllow logs -f picks it
up again. A job waits for a free machine for up to a day unless --queue-timeout says
otherwise. An unknown command or option, or a missing argument, is a usage error and exits 2
with the usage line, as any command line tool does.