# How the job system works

Understand what a job is, how the queue matches work to workers, and how Programs, pipelines and files fit together.

A **job** is a self-contained run of Python code together with its exact inputs, dependency lock, logs, and output files. You can run a job on your laptop in seconds or submit the exact same job to a queue of remote CPU, GPU, or Ray workers.

## 1. What a job looks like

Every job runs inside a fresh, isolated working directory (`attempt directory`) and can be written in one of two ways:

### A standalone Python script

For quick experiments or one-off scripts, write a standard `.py` file that reads command-line arguments or staged input files and writes any files you want to keep into `results/`:


```python
"""A tiny ordinary script for trying the local compute CLI."""

from __future__ import annotations

import sys
from pathlib import Path

Path("results/hello.txt").write_text(" ".join(sys.argv[1:]) or "hello")
```


### A typed Python function (`Program`)

When you want a calculation to be reusable across your team or chained into larger workflows, wrap a Python function with strict Pydantic input and output models and publish it as a versioned **Program** (such as `example.double@1`):


```python
from pathlib import Path

from pydantic import BaseModel, ConfigDict, Field


class Number(BaseModel):
    model_config = ConfigDict(extra="forbid", strict=True, frozen=True)
    value: int


def double(request: Number) -> Number:
    result = Number(value=request.value * 2)
    Path("results/value.txt").write_text(str(result.value))
    return result
```


Whether you submit a script or a published Program, every job captures and produces the same four things:

| Part                   | What is inside                                                                                                                                             | Where it lives                                                                                       |
| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------- |
| **Code & environment** | Your frozen script or `Program` (`name@version`) plus its locked Python environment (`uv.lock`, shared scientific runtime, or container).                  | Frozen at submission time so editing your local checkout later never alters a queued or running job. |
| **Parameters**         | Small structured settings (numbers, flags, method names) passed via `--params` or a Pydantic model.                                                        | Validated before execution starts; typos and unknown keys are rejected immediately.                  |
| **Input files**        | Explicitly staged files such as structures or configs (`--input geometry.xyz=./structure.xyz`).                                                            | Uploaded once to private storage and placed in the job's working directory before your code runs.    |
| **Outputs & logs**     | A compact typed return value (JSON/Pydantic), `stdout`/`stderr` logs, and any files written under `results/**` (trajectories, `.extxyz` structures, CSVs). | Automatically compressed and uploaded to private object storage when the job finishes.               |

## 2. How the queuing system works

When you run `ct-cli run`, your script executes immediately on your machine with no server involved. When you run `ct-cli submit` (or `client.submit(...)` in Python), your job goes through four steps:

```text
Your laptop / script
  │  1. Freeze code & upload input files
  ▼
Backend API & Queue (Hatchet)
  │  2. Hold in queue until a matching worker is free
  │     (pool + platform + CPUs / GPUs / memory)
  ▼
Registered Worker (laptop, lab server, HPC node, or cloud VM)
  │  3. Prepare isolated scratch workspace & download inputs
  │  4. Run Python code & stream logs
  ▼
Private Object Storage (S3) & Project Database
     5. Upload results/** files, record typed output & clean up workspace
```

1. **Freeze and admit:** The CLI or Python SDK freezes your code and dependency lock, uploads your declared `--input` files directly to private S3-compatible storage using short-lived presigned URLs, and registers the job with the backend. You immediately receive a `JOB_UUID`—you can close your laptop and the job will still run.
2. **Queue and match capacity:** The backend hands the job to the scheduler (**Hatchet**). Your job waits in the queue until an enabled [worker](/operate/workers/) matching your requested **pool** (for example `cpu` or `gpu`), **platform** (such as `linux-x86_64-py3.13-cpu`), and hardware resources (`--cpus`, `--gpus`, `--memory`) has free capacity.
3. **Execute in a clean workspace:** The worker claims the job from the queue, creates a fresh temporary directory for that attempt, prepares the Python environment (see [Choose code and dependencies](/operate/environments/)), stages your input files, and starts your Python process.
4. **Collect outputs and clean up:** When the Python process exits, the worker compresses and uploads every file matching `results/**` along with execution logs, reports the validated return value back to the server, and deletes the temporary scratch directory. Temporary files outside `results/` are not uploaded.

## 3. Job states and lifecycle

You can check any job from any terminal using its UUID:

```bash
ct-cli status JOB_UUID
ct-cli logs JOB_UUID
ct-cli fetch JOB_UUID --output ./job-output
```

A job moves through these states:

| State           | What is happening                                                                                                                                                      | What you should do                                                                                       |
| --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
| **`queued`**    | Admitted by the server and waiting in the scheduler queue for either parent jobs (`depends_on`) to finish or a matching worker in your target `--pool` to become free. | Nothing—or run `ct-cli worker list` if a job stays queued because no worker is registered for that pool. |
| **`running`**   | Claimed by a worker; staging inputs, running your Python code, or uploading `results/**` artifacts.                                                                    | Stream progress with `ct-cli logs JOB_UUID --follow`, or detach safely with `Ctrl-C`.                    |
| **`succeeded`** | The Python process exited cleanly, its typed output passed schema validation, and all `results/**` files are durably stored.                                           | Download files with `ct-cli fetch JOB_UUID --output ./job-output`.                                       |
| **`failed`**    | The Python process raised an exception, timed out, or returned an invalid output schema. Any diagnostic files written to `results/` before failure are still uploaded. | Inspect `ct-cli logs JOB_UUID` and see [Inspect and recover a run](/operate/recovery/).                  |
| **`cancelled`** | Stopped explicitly via `ct-cli cancel JOB_UUID` (or because a required sibling job in a Flow failed).                                                                  | Worker terminates the process tree and cleans up the scratch directory.                                  |

> **Important:** A **`succeeded`** job state means the computation finished without crashing. In scientific workflows, process completion alone never establishes that a transition state or reaction pathway is physically valid—scientific acceptance happens in a dedicated verification step.

## 4. How jobs compose into pipelines and Flows

You never need a separate workflow engine or custom queue for multi-step science. Complex workflows are built by connecting ordinary jobs in the same queue:

| Building block                             | How it works                                                                                                                                                      | When to use it                                                                                                                                            |
| ------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Script** (`ct-cli run` / `submit`)       | Runs a `.py` file once with optional CLI arguments and `--input` files.                                                                                           | [Run your first job](/start/first-job/) or test an idea quickly.                                                                                          |
| **Program** (`name@version`)               | Publishes a Python function with Pydantic input/output models so anyone can call it by name.                                                                      | [Make a function reusable](/start/reusable-programs/) across scripts, CLI calls, and pipelines.                                                           |
| **Dependency chain** (`depends_on`)        | Submits `Run → Check → Save` jobs up front. The queue holds `Check` until `Run` succeeds, then automatically stages `Run`'s output values and files into `Check`. | [Build a scientific pipeline](/start/pipelines/) where failed verification or database saving can be retried without repeating the expensive calculation. |
| **Flow** (`prepare → map → reduce → then`) | Fans out a Program across many inputs in parallel (with configurable `--concurrency`) and collects all outputs into a reducer job.                                | Run parameter sweeps, batch DFT evaluations, or multi-structure optimizations.                                                                            |
| **Benchmark** (`ct-cli benchmark`)         | Runs a candidate Program across a frozen dataset, scores each prediction with an evaluator Program, and reduces the scores into a ranked summary.                 | [Create an algorithm benchmark](/test/benchmarks/) to compare parsers, models, or TS search strategies fairly.                                            |

Ready to try it hands-on? Continue to [Run your first job](/start/first-job/) to execute a script locally and submit it to a worker.