CLI

falllow is how work is submitted and followed. Progress goes to stderr and the command's output to stdout, so it pipes the same as the local command. Every command takes --json, and every command explains itself with examples in falllow <command> --help.

Commands

CommandWhat it does
loginStore the server and a token for this machine, checked before it replaces a login that works.
login --consumer [NAME]Make a consumer with the current admin login and log in as it, named after the host by default. --gh-secret OWNER/REPO puts its token into that repository's FALLLOW_TOKEN secret instead.
runSubmit a job and stream it. Everything after -- is the command.
psRecent runs with status, exit code and machine, everyone's unless narrowed with --mine or --tag.
logsPrint a run's output, or -f to follow it and exit with its code.
waitBlock until a run settles, then exit with its code. --notify URL posts the outcome.
showOne run in detail, including why it did not complete.
fetchBring back what a run wrote to /artifacts, and its log. --list only lists, -o DIR picks where.
cancelStop runs, by name or id, or every active one picked by --mine or --tag. --abort kills instead of stopping gracefully.
queueWhat is waiting for a machine, in the server's order, when each is tried next and how much of a machine it needs.
bumpMove a waiting run to the front, at --priority 90 unless told otherwise.
devicesThe machines: up, asleep or unreachable, cores, memory, blocks in use, disk free where jobs write, and budget.
cache lsThe named caches on every machine that is on, with their size.
cache rm NAMEDelete a cache on every machine, or on one with --machine.
rmDelete a run's artifacts, the server's copy and the machine's.
pool invite NAMEPrint the line that takes a machine into the pool. --user, --data, --blocks. Admin token.
pool remove NAMETake a machine out of the pool, the fleet and the network. Admin token.
pool pruneHave every machine that is on drop the images nothing uses, within ten minutes and once nothing runs there.
pool show, pool applyThe pool file's machines with power and budget, and making the fleet match it. Run on the server.

Running things

falllow run
# stream it to completion, exit with its code
falllow run -- pytest -q

# ask for more of a machine
falllow run --cpu 8 --memory 16G -- cargo test --release

# a GPU and an image with the toolkit in it
falllow run --gpu 24GB --image nvidia/cuda:12.4.1-devel-ubuntu22.04 -- ./train.sh

# a cache that outlives the run, on that machine
falllow run --cache cargo-target -- cargo build --release

# no sources needed
falllow run --no-workspace -- nproc

# hand back the run name and carry on
falllow run --detach -- ./long-solve.sh
FlagMeaning
--cpuCores the job needs at least, or a range like 4..16: as many as one machine has free, within it. The job sees what it got as FALLLOW_CORES. Default 1.
--memoryMemory it needs at least, e.g. 8G. Default 1G.
--diskDisk it needs at least. Default 10G. Said when no machine has that much free where jobs write.
--gpuA GPU, by memory or name: 24GB, A100:2.
--imageContainer image to run in. Without it, Ubuntu with build tools, dstack's base image.
--env, -eKEY=VALUE, or a bare KEY to pass on its value here. Repeatable.
--cacheMount the machine's cache NAME at /cache/NAME, kept between runs. Lower case letters, digits and dashes. Repeatable.
--timeoutWall clock limit once running, e.g. 30m or 2h. Default 7d.
--queue-timeoutHow long to wait for a free machine. Default 1d.
--priority0 to 100, higher is placed first when machines are busy.
--machineRun only on this machine, named as falllow devices shows it. Waits while it is busy. Repeatable, for any of several.
--shardRun this many copies, each told its FALLLOW_SHARD_INDEX.
--nameRun name. Generated from the command, plus a random suffix, if left out.
--tagKEY=VALUE, a label for ps --tag and cancel --tag. host, repo and session are added. Repeatable.
--notifyPOST the outcome as JSON to this URL when the run settles, from the server. Also on wait, from there.
--detach, -dPrint the run name and return at once. With --json, a list of each run's name and id.
--includeShip this path even when an ignore rule excludes it. Repeatable.
--no-vcs-ignoreShip the directory as it lies, without git's ignore rules.
--no-workspaceShip no sources, for commands that need none.

A job that no machine in the pool could ever take, more cores or memory than any machine has, a GPU where there is none, an image with no build for the pool's architecture, is refused before anything is sent. When the memory asked for, not the cores, decides how much of a machine a job takes, run says so, since that is what makes a job wait while cores are free.

The working directory

What is on disk is what runs. The enclosing git repository, or the directory itself if there is none, is packed on your machine and unpacked on the pool's, committed or not. Nothing is cloned from anywhere, so the pool needs no way into GitHub and no token travels with the job. A home directory or a filesystem root is refused rather than packed whole; run from the project's directory, or pass --no-workspace.

.git always stays home, so git describe does not work inside a job, and so does everything .dstackignore names. .gitignore decides inside a git repository, where it means build output, caches and virtualenvs. Outside a repository it does not: a directory assembled by hand is meant to travel as it lies, and a .gitignore copied in with the sources would otherwise keep excluding files where there is no git at all.

What a rule leaves out is named when the job is sent, not discovered as a missing file in a container. --include PATH ships a path however it was ignored, repeatable, and --no-vcs-ignore ships the whole directory without git's rules: for the wheel built for the pool, the model, the data package.

What stays home
$ falllow run -- ./build.sh
falllow: shipping /home/me/project
falllow: not shipped: remote/_wheel/, target/; override with --include PATH

From 50 MB on, run says how much travels and which top level entries make up most of it. The server accepts 256 MB per upload unless configured otherwise; a tree bigger than that wants a .dstackignore, or the large part can stay on the machine instead of travelling with every job. Packing stops at 1 GB in any case.

The directory lands at /workspace, and the command runs from the subdirectory you typed it in: falllow run -- make in crates/solver runs in /workspace/crates/solver.

Sharding

--shard submits several independent jobs rather than splitting the work itself, only your command knows how to divide it. Each shard is told its slice and decides what that means. The working directory is packed once and the same snapshot goes to every shard.

Each shard is a run of its own, so --cpu and --memory are what one shard gets, not what they share: eight shards at --memory 4G ask the pool for 32 GB in total. They do not have to be on the same machine, or run at the same time; whatever does not fit waits its turn. A shard that exceeds its memory is killed on its own, and the others carry on.

Eight ways at once
falllow run --shard 8 -- pytest -q
What the command does with it
import os

index = int(os.environ.get("FALLLOW_SHARD_INDEX", 0))
count = int(os.environ.get("FALLLOW_SHARD_COUNT", 1))

mine = [t for i, t in enumerate(tests) if i % count == index]

The shards queue like any other jobs and run wherever there is room. Their output is printed one shard after another as they finish, since interleaved lines from eight jobs read as none. The exit code is the first genuine failure among them; an infrastructure failure only surfaces when no shard actually failed. If sending the group fails halfway, the shards already sent are stopped.

Waiting for a machine

Before anything is sent, falllow run says whether the job starts now and where, or what it waits for: how many blocks of the machine closest to taking it, how many are free, its place in the queue, and the flags that would make it start now. While it waits it says the same again whenever that changes.

Before it is sent
$ falllow run --cpu 16 --memory 40G -- ./solve.sh
falllow: waiting: needs 18/20 blocks on milan-pc (memory-bound, 2.4 GB/block), 6 free; queue position 2, behind sweep-sh-9c1e4b7a
falllow: starts now with --cpu 12 --memory 23G (strato-1)

--cpu 4..16 asks for at least four cores and as many up to sixteen as one machine has free when it is sent, and the job is told what it got in FALLLOW_CORES. A job that scales with its cores starts now with what there is rather than waiting for all of it. Memory has no range: a job given less than it needs is killed, one given fewer cores only takes longer.

A job that finds every fitting machine busy waits rather than failing, for up to --queue-timeout. It is tried again within seconds of a machine it may use having room for it, in queue order. Besides that, dstack tries a waiting run after a delay that grows: 15s, 30s, 1m, 2m, 5m, then every 10 minutes. falllow queue shows what is waiting, in queue order, when each one is due to be tried again at the latest, and how many blocks it needs. A machine that is being switched on for it, or cannot be, is said while run waits.

falllow queue
$ falllow queue
 #  RUN               PRIO  WAITING TRIES  NEXT TRY  NEEDS
 1  sweep-sh-9c1e4b7a   90      17m     7       12s  16 cores, 32 GB: 16 blocks
 2  pytest-77c2e05b      0      12m     6        4m  2 cores, 16 GB: 4 blocks (memory-bound)
falllow: move to front: falllow bump <run>

Order is priority first, then whoever has waited longest. --priority 90 on run asks to be placed before the rest, and falllow bump <run> does the same for something already waiting: it queues the same job again at a higher priority and stops the copy that was waiting. Nothing is uploaded twice, the archive already on the server is reused, and the run gets a new id. Free blocks go to the waiting runs in that order, and a run that does not fit leaves them to the next one that does. A new run is placed the moment it arrives, so priority is a strong preference rather than a guarantee.

A large run would still lose to small ones: they are placed the moment they arrive, and it needs many blocks free at once. So once the head of the queue has waited 15 minutes, falllow run holds back a new run that would take blocks the head needs, says so, and sends it once the head has started. Nothing is sent while it is held, so Ctrl-C gives up cleanly, and a run still held after --queue-timeout exits 170 with nothing sent. A run with a higher --priority than the head is not held, nor is one for machines the head cannot run on, its --machine included. With -d nothing is held: the run is sent at once, with a line naming the run it would have waited for. Until the 15 minutes are up the pool is shared as before, so a quick test does not queue behind a large run that has only just arrived.

Held for a run that has waited
$ falllow run --cpu 2 -- pytest -q
falllow: held for sweep-sh-9c1e4b7a (waiting 17m, needs these blocks); nothing sent yet; skip with --priority 91
falllow: released

Images

--image runs the job in an image of your own. Left to itself, the server would read that image's configuration from its registry on every submission, and registries limit how often anonymous callers may do that. So falllow run looks up the two things the server would ask for, the user the image runs as and the platforms it is built for, once a week on your machine (~/.cache/falllow/images.json), and puts them in the run. Several agents on one machine share that file. Should a registry still answer 429, run waits and tries again for about eight minutes before exiting 170.

What a job leaves behind

Every run gets a directory of its own on the machine it runs on, mounted at /artifacts and named in FALLLOW_ARTIFACTS. It outlives the container, so what a job wrote is still there when the run is over, including when the run failed. Nothing is collected by pattern: what the job put there is what it kept.

falllow fetch <run> brings it back, together with the run's log, into artifacts/<run>/. It reads through the pool's server, so it works from a laptop or from CI without being on the pool's network. A transfer that breaks off writes nothing and exits 170; fetching again is the remedy.

Each run writes into a directory of its own, its name plus a token, and the server files what it collects under the run's id. Names repeat: --name nightly is the same name every night, and dstack frees a name once its run has ended, so a directory keyed by the name alone would let one night's results overwrite the last. falllow fetch <run> takes a name, meaning the most recent run that had it, or an id from --json, meaning one particular run for ever.

Once a run settles the server collects its artifacts onto its own disk, so fetching keeps working after the machine is switched off, replaced or gone. The machine keeps its own copy for 14 days. A run too large to collect stays where it was made and says so; the collection as a whole stays inside a budget, oldest dropped first, and falllow devices prints what it takes up. falllow rm <run> deletes both copies.

falllow fetch
$ falllow fetch sweep-sh-9c1e4b7a
falllow: 4 files in artifacts/sweep-sh-9c1e4b7a

$ ls artifacts/sweep-sh-9c1e4b7a
falllow.log   result.json   plots/sweep.png   plots/error.png

The environment a job sees

PathWhat it is
/workspaceThe shipped directory. The command starts in the subdirectory it was typed in.
/artifactsThe run's own directory on the machine, which outlives the container. falllow fetch brings it back.
/scratchRoom to work, on the machine's data disk where it has one, the run's own and removed once it ends. Large intermediate files belong here rather than in the container's own filesystem.
/cache/<name>For each --cache NAME: kept on that machine across runs and shared by every run that names it, for what is slow to build and the same each time. Per machine, so a run elsewhere finds it empty. Runs that write it at once should write to a temporary name and rename. Each machine keeps 50 GB of caches, dropping the least recently changed.
/poolThe machine's tools: Rust, uv and sccache, on the PATH, with cargo's, uv's, pip's and sccache's caches pointed here, so a second build on the same machine reuses what the first downloaded and compiled. Only where the image does not set its own, so an image with a toolchain keeps it. What a job installs here with cargo install or uv tool install stays for the next run on that machine.
VariableValue
FALLLOW_CORESThe cores asked for. Pass it on: make -j$FALLLOW_CORES.
FALLLOW_MEMORY_MIBThe memory asked for, in MiB. free reports the whole machine, so size by this.
FALLLOW_ARTIFACTS/artifacts, where to put what should outlive the job.
FALLLOW_SCRATCH/scratch, room to work.
FALLLOW_CACHE/cache, when the run asked for a cache.
FALLLOW_NOTIFYThe --notify URL, when one was given.
MAKEFLAGS, CARGO_BUILD_JOBS, OMP_NUM_THREADS, and friendsSet to the same number, so tools that read them keep to the share the job was given. A job gets a CPU quota, not a smaller machine, so nproc still answers with the whole machine. Anything passed with -e wins.
FALLLOW_SHARD_INDEXThis shard's index, zero based. Only when sharding.
FALLLOW_SHARD_COUNTHow many shards there are.
FALLLOW_NO_POOLNot set by falllow. Pass -e FALLLOW_NO_POOL=1 to leave /pool's tools and caches out of the job's environment.

A job asks for 1 core, 1 GB of memory and 10 GB of disk unless told otherwise. dstack's own defaults are 2 cores, 8 GB and 100 GB, which would keep small jobs off small machines.

For scripts and agents

The contract is the exit code and --json; nothing needs parsing from the human output. The simplest use is the blocking one: falllow run in place of the command, then branch on the exit code.

Branching on the outcome
falllow run --cpu 4 -- pytest -q
case $? in
  0)   echo "green" ;;
  170) echo "the pool failed, not the tests, safe to run again" ;;
  171) echo "failed with its status lost, check the disk first" ;;
  172) echo "falllow refused it, read the message" ;;
  137) echo "killed, most likely out of memory: ask for more" ;;
  130) echo "cancelled, or this process was interrupted" ;;
  *)   echo "the tests failed" ;;
esac

What a job takes, it takes from everyone else: blocks asked for and not used are blocks another job waits for. So after a run, falllow run says what it used of what it asked for, and what would have done; before one, it says when the same command from the same repository used far less the last times. The server keeps every run's peaks for 90 days, since dstack drops a running job's samples after an hour. --json has the same under usage, falllow show prints it, and the manager shows it with the job.

Asked for, and used
falllow: last 4 runs: max 2.6 cores, 1.6 GB; enough: --cpu 3 --memory 2G
...
falllow: usage: cpu 1.9 avg, 2.6 max of 16; memory 1.6 max of 32 GB
falllow: enough: --cpu 3 --memory 2G

For anything long, start it detached and come back. The run name is the handle; every command takes it, or the run's id, and --json gives the same object everywhere.

Start, check, wait
name=$(falllow run -d -- ./long-solve.sh)   # prints the run name at once
falllow show "$name" --json                 # status, exit_code (null until settled), machine
falllow wait "$name" --json                 # blocks, prints the outcome, exits with its code
falllow logs "$name" -f                     # the output, following
falllow fetch "$name"                       # what it wrote to /artifacts, and its log

falllow run -d --json -- ./long-solve.sh    # [{"name": "long-solve-sh-...", "id": "..."}]
falllow wait --json
$ falllow wait --json cargo-test-3f9a1c2d
{
  "name": "cargo-test-3f9a1c2d",
  "id": "8f0e2c4a-6d1b-4f3e-9a57-2b8c1d0e4f6a",
  "status": "failed",
  "exit_code": 1,
  "exit_status": 1,
  "termination_reason": "container_exited_with_error",
  "submitted_at": "2026-09-16T09:38:50.980859+00:00",
  "finished_at": "2026-09-16T09:39:32.780373+00:00",
  "instance": "10.7.0.3",
  "attempts": 1,
  "tags": {"host": "laptop", "repo": "solver", "session": "agent-7"}
}

exit_code is what falllow run would have exited with, null while the run is unsettled. exit_status is the command's raw status if it got to run. termination_reason says why a run ended when it was not the command's doing.

Several agents share the pool, and falllow ps shows everyone's runs. Each run is tagged with the machine it came from and the directory it shipped. An agent that sets FALLLOW_SESSION once gets its runs tagged with that too, and --mine on ps and cancel then means this machine and this session. --tag KEY=VALUE adds a label of your own and selects by it.

Telling runs apart
export FALLLOW_SESSION=agent-7               # once, per agent or terminal
falllow run -d --tag study=gallery -- ./sweep.sh
falllow ps --mine                             # this machine and this session
falllow ps --tag study=gallery --json
falllow cancel --mine                         # everything this session still has going
falllow cancel --tag study=gallery

A line on stderr that starts with falllow: pool problem: means a part of the pool itself is broken: a full disk, a service that is down, a health check that has stopped. The same lines appear on run, ps, queue and devices and above every page of the manager. They do not change the exit code; pass them on to whoever runs the pool.

To be told rather than to poll, --notify on run posts the outcome to a URL when the run settles. The URL is carried on the run and the pool's server does the posting, so a job started with -d is announced long after the terminal that started it is closed; a server that is not sending says so when the run is submitted. The body is the same JSON --json prints, plus a one line text for webhooks that show a message and nothing else. On wait the flag posts from the waiting process instead, since a run already under way cannot be told where to report.

Getting told
falllow run -d --notify https://hooks.example.com/falllow -- ./sweep.sh
# the server POSTs the same JSON that --json prints, once the run settles,
# whether or not anything is still attached
falllow ps
$ falllow ps --mine
NAME                             STATUS         EXIT  INSTANCE           SUBMITTED
sweep-sh-9c1e4b7a-1              running              10.7.0.2           2026-09-16T09:41:55
cargo-test-3f9a1c2d              failed            1  10.7.0.3           2026-09-16T09:38:50
pytest-77c2e05b                  done              0  10.7.0.3           2026-09-16T09:38:26

Exit codes

CodeMeaning
0The command succeeded.
anything elseThe command's own exit code, passed through unchanged.
130Somebody cancelled the run, or this command was interrupted: Ctrl-C or a SIGTERM from a caller's timeout detaches and the run goes on, and the message says how to pick it up.
137The kernel killed the container, nearly always because it used more memory than it asked for.
170Reserved: the pool failed, not your command. No machine free in time, a machine lost, the timeout passed, the server out of reach, a registry still rate limiting. Running it again may help.
171Reserved: your command failed, but its exit status was lost with the container. So far always a full disk, so running it again as it is fails the same way.
172Reserved: falllow turned the request down and nothing ran. Not logged in, no machine that could ever fit it, a name in use, no such run, a value it cannot read, a directory too large to ship. The message says which; the same request gets the same answer.

Ctrl-C while following detaches and leaves the job running; falllow logs -f picks it up again. A job waits for a free machine for up to a day unless --queue-timeout says otherwise. An unknown command or option, or a missing argument, is a usage error and exits 2 with the usage line, as any command line tool does.