How the job system works

Understand what a job is, how the queue matches work to workers, and how Programs, pipelines and files fit together.

A job is a self-contained run of Python code together with its exact inputs, dependency lock, logs, and output files. You can run a job on your laptop in seconds or submit the exact same job to a queue of remote CPU, GPU, or Ray workers.

1. What a job looks like

Every job runs inside a fresh, isolated working directory (attempt directory) and can be written in one of two ways:

A standalone Python script

For quick experiments or one-off scripts, write a standard .py file that reads command-line arguments or staged input files and writes any files you want to keep into results/:

python

"""A tiny ordinary script for trying the local compute CLI."""
from __future__ import annotations
import sysfrom pathlib import Path
Path("results/hello.txt").write_text(" ".join(sys.argv[1:]) or "hello")

A typed Python function (`Program`)

When you want a calculation to be reusable across your team or chained into larger workflows, wrap a Python function with strict Pydantic input and output models and publish it as a versioned Program (such as example.double@1):

python

from pathlib import Path
from pydantic import BaseModel, ConfigDict, Field

class Number(BaseModel):    model_config = ConfigDict(extra="forbid", strict=True, frozen=True)    value: int

def double(request: Number) -> Number:    result = Number(value=request.value * 2)    Path("results/value.txt").write_text(str(result.value))    return result

Whether you submit a script or a published Program, every job captures and produces the same four things:

Part	What is inside	Where it lives
Code & environment	Your frozen script or `Program` (`name@version`) plus its locked Python environment (`uv.lock`, shared scientific runtime, or container).	Frozen at submission time so editing your local checkout later never alters a queued or running job.
Parameters	Small structured settings (numbers, flags, method names) passed via `--params` or a Pydantic model.	Validated before execution starts; typos and unknown keys are rejected immediately.
Input files	Explicitly staged files such as structures or configs (`--input geometry.xyz=./structure.xyz`).	Uploaded once to private storage and placed in the job's working directory before your code runs.
Outputs & logs	A compact typed return value (JSON/Pydantic), `stdout`/`stderr` logs, and any files written under `results/**` (trajectories, `.extxyz` structures, CSVs).	Automatically compressed and uploaded to private object storage when the job finishes.

2. How the queuing system works

When you run ct-cli run, your script executes immediately on your machine with no server involved. When you run ct-cli submit (or client.submit(...) in Python), your job goes through four steps:

text

Your laptop / script  │  1. Freeze code & upload input files  ▼Backend API & Queue (Hatchet)  │  2. Hold in queue until a matching worker is free  │     (pool + platform + CPUs / GPUs / memory)  ▼Registered Worker (laptop, lab server, HPC node, or cloud VM)  │  3. Prepare isolated scratch workspace & download inputs  │  4. Run Python code & stream logs  ▼Private Object Storage (S3) & Project Database     5. Upload results/** files, record typed output & clean up workspace

Freeze and admit: The CLI or Python SDK freezes your code and dependency lock, uploads your declared --input files directly to private S3-compatible storage using short-lived presigned URLs, and registers the job with the backend. You immediately receive a JOB_UUID—you can close your laptop and the job will still run.
Queue and match capacity: The backend hands the job to the scheduler (Hatchet). Your job waits in the queue until an enabled worker matching your requested pool (for example cpu or gpu), platform (such as linux-x86_64-py3.13-cpu), and hardware resources (--cpus, --gpus, --memory) has free capacity.
Execute in a clean workspace: The worker claims the job from the queue, creates a fresh temporary directory for that attempt, prepares the Python environment (see Choose code and dependencies), stages your input files, and starts your Python process.
Collect outputs and clean up: When the Python process exits, the worker compresses and uploads every file matching results/** along with execution logs, reports the validated return value back to the server, and deletes the temporary scratch directory. Temporary files outside results/ are not uploaded.

3. Job states and lifecycle

You can check any job from any terminal using its UUID:

A job moves through these states:

State	What is happening	What you should do
`queued`	Admitted by the server and waiting in the scheduler queue for either parent jobs (`depends_on`) to finish or a matching worker in your target `--pool` to become free.	Nothing—or run `ct-cli worker list` if a job stays queued because no worker is registered for that pool.
`running`	Claimed by a worker; staging inputs, running your Python code, or uploading `results/**` artifacts.	Stream progress with `ct-cli logs JOB_UUID --follow`, or detach safely with `Ctrl-C`.
`succeeded`	The Python process exited cleanly, its typed output passed schema validation, and all `results/**` files are durably stored.	Download files with `ct-cli fetch JOB_UUID --output ./job-output`.
`failed`	The Python process raised an exception, timed out, or returned an invalid output schema. Any diagnostic files written to `results/` before failure are still uploaded.	Inspect `ct-cli logs JOB_UUID` and see Inspect and recover a run.
`cancelled`	Stopped explicitly via `ct-cli cancel JOB_UUID` (or because a required sibling job in a Flow failed).	Worker terminates the process tree and cleans up the scratch directory.

Important: A succeeded job state means the computation finished without crashing. In scientific workflows, process completion alone never establishes that a transition state or reaction pathway is physically valid—scientific acceptance happens in a dedicated verification step.

4. How jobs compose into pipelines and Flows

You never need a separate workflow engine or custom queue for multi-step science. Complex workflows are built by connecting ordinary jobs in the same queue:

Building block	How it works	When to use it
Script (`ct-cli run` / `submit`)	Runs a `.py` file once with optional CLI arguments and `--input` files.	Run your first job or test an idea quickly.
Program (`name@version`)	Publishes a Python function with Pydantic input/output models so anyone can call it by name.	Make a function reusable across scripts, CLI calls, and pipelines.
Dependency chain (`depends_on`)	Submits `Run → Check → Save` jobs up front. The queue holds `Check` until `Run` succeeds, then automatically stages `Run`'s output values and files into `Check`.	Build a scientific pipeline where failed verification or database saving can be retried without repeating the expensive calculation.
Flow (`prepare → map → reduce → then`)	Fans out a Program across many inputs in parallel (with configurable `--concurrency`) and collects all outputs into a reducer job.	Run parameter sweeps, batch DFT evaluations, or multi-structure optimizations.
Benchmark (`ct-cli benchmark`)	Runs a candidate Program across a frozen dataset, scores each prediction with an evaluator Program, and reduces the scores into a ranked summary.	Create an algorithm benchmark to compare parsers, models, or TS search strategies fairly.

Ready to try it hands-on? Continue to Run your first job to execute a script locally and submit it to a worker.