# Pipelines and pipes

A **pipeline** is one run. Each YAML file in `.acahti/pipelines/` is a **job** in that run, and each job has named **steps**. A **pipe** is an official, reusable step: `pipe: <name>@v1` with inputs under `with:`.

Before the Runner starts a job, the gateway expands it: `pipe:` becomes a call to `acahti-pipe <name>`, mapped secrets become step environment, and publishing pipes receive the identity of the person who triggered the run (`ACAHTI_USER`, `ACAHTI_TOKEN`). The Runner only ever sees expanded YAML.

## Writing a job

A step is either `pipe:` with `with:`, or plain `commands:` for one-offs. `when`, `depends_on`, and `labels` pass through unchanged. Do not vendor helper scripts into the repository and do not write `uses:`.

```yaml
steps:
  test:
    image: node:22
    commands:
      - npm ci
      - npm test
```

Jobs order by `depends_on` between files, parents first. If a job declares `concurrency:`, its `group` is scoped to the repository, so `group: deploy` in two repositories does not serialize them. Leave `concurrency` out to run freely.

In `commands:`, write `$VAR` for step environment. `${VAR}` is interpolated before the shell starts and is empty unless it is job context, such as `${CI_COMMIT_SHA}` or `${CI_COMMIT_TAG}`.

Trigger runs with `git push`, a tag, or a pull request. Git tags are `vX.Y.Z`; image tags drop the `v` (`${CI_COMMIT_TAG#v}`).

## Secrets

Pipeline secrets are credentials for systems outside Acahti: registries, clusters, object storage. Admins add them in the web app (Admin → Org secrets, or a repository's Secrets page) or with `secret_put`. Names are lowercase `snake_case`. Values are never listed or returned.

A step opts in. List a secret when its name is the variable the pipe wants; map it when the names differ:

```yaml
secrets:
  KUBECONFIG: kubeconfig_prod
  DOCKER_USERNAME: registry_username
  DOCKER_PASSWORD: registry_password
```

Do not put file paths to credentials in `with:`. Acahti's own npm and PyPI registries use the triggering person's identity, so never create a token secret for them.

## Official pipes

### docker-build

Build images with BuildKit and push them, or build cache-only for CI.

| Input | Required | Description |
|---|---|---|
| `images` | yes | One image per line. First field is the primary tag. Optional KEY=VAL: also extra tags (comma-separated), context (default .), file (Dockerfile), anything else is --build-arg. Job identity is written as BuildKit secrets id=npmrc and id=netrc (HTTP Basic from ACAHTI_USER / ACAHTI_TOKEN). Dockerfiles mount id=npmrc at /root/.npmrc and id=netrc at /root/.netrc. Default --network=host on buildx. CODEUP_NETRC, when set, is passed as id=codeup_netrc. |
| `push` |  | true (default) is buildx --push; false is --output=type=cacheonly (CI) |
| `network` |  | docker buildx --network (default host) |
| `builder` |  | buildx builder name (default acahti, docker-container + host network) |

### docker-gc

Reclaim Runner disk. Runs from an hourly timer on the Runner, not from pipelines.

### docker-login

Log in to a container registry for later steps in the job. Secret DOCKER_PASSWORD.

| Input | Required | Description |
|---|---|---|
| `registry` | yes | Registry hostname. An http:// prefix is the same as http: true. |
| `http` |  | true if the registry is HTTP/insecure (BuildKit). Default false (HTTPS). |
| `username` |  | Registry user; password is DOCKER_PASSWORD (or DOCKER_USERNAME + DOCKER_PASSWORD) |

### helm

Install or upgrade a Helm release. Secret KUBECONFIG holds the kubeconfig document.

| Input | Required | Description |
|---|---|---|
| `release` | yes |  |
| `namespace` | yes |  |
| `chart` |  | Chart path relative to the workspace (default chart) |
| `files` |  | Extra -f values files, one per line, relative to the workspace |
| `set` |  | Extra --set lines (foo=bar or --set foo=bar) |
| `timeout` |  | Helm --timeout (default 5m) |
| `take_ownership` |  | true to pass --take-ownership |
| `jump` |  | SSH host, reachable from the Runner, that runs helm when the cluster API is private |

### npm-publish-acahti

Build an npm package and publish it to this org's Acahti registry.

| Input | Required | Description |
|---|---|---|
| `path` | yes | Package directory (package.json). Installs, builds, packs, and PUTs to this Acahti org's npm. |
| `image` |  | Node image (default node:22-alpine). Private registry images need a prior docker-login. |

### oci-gc

Prune old tags from OCI repositories, keeping latest and the newest few.

| Input | Required | Description |
|---|---|---|
| `repos` | yes | OCI repositories to prune (no tag), one per line |
| `keep` |  | Non-latest tags to keep (default 2). latest is always kept. |

### oss-put

Upload files to Alibaba Cloud OSS. Secrets OSS_ACCESS_KEY_ID and OSS_ACCESS_KEY_SECRET.

| Input | Required | Description |
|---|---|---|
| `endpoint` | yes |  |
| `bucket` | yes |  |
| `prefix` | yes | Object key prefix (no oss://) |
| `files` | yes | Local files to upload, one per line (workspace-relative) |

### pypi-publish-acahti

Build a Python package and publish it to this org's Acahti registry.

| Input | Required | Description |
|---|---|---|
| `path` |  | Directory with pyproject.toml (default .). Builds and PUTs to this Acahti org's PyPI. |
| `image` |  | Python image (default python:3.12-slim-bookworm). uv is installed if missing. Private registry images need a prior docker-login. |

### uv

Run commands in a job-local Python environment with uv.

| Input | Required | Description |
|---|---|---|
| `project` |  | Directory with pyproject.toml (default .) |
| `run` | yes | One command per line; each is uv run --project <dir> -- <line> |

### wait-http

Wait until URLs stop answering 502, 503, 504, or nothing.

| Input | Required | Description |
|---|---|---|
| `urls` | yes | URLs to poll, newline or semicolon separated |
| `timeout` |  | Seconds to wait (default 180) |
| `user_agent` |  | User-Agent header for the probes |

## Examples

CI that builds an image without pushing it:

```yaml
steps:
  build:
    pipe: docker-build@v1
    with:
      push: false
      images: |
        app:${CI_COMMIT_SHA}
```

CD that builds, deploys, and prunes old tags, one deploy at a time:

```yaml
concurrency:
  limit: 1
  group: deploy
steps:
  login:
    pipe: docker-login@v1
    with:
      registry: registry.example.com
    secrets:
      DOCKER_USERNAME: registry_username
      DOCKER_PASSWORD: registry_password
  build:
    depends_on: [login]
    pipe: docker-build@v1
    with:
      images: |
        registry.example.com/app:${CI_COMMIT_SHA} also=registry.example.com/app:latest
  deploy:
    depends_on: [build]
    pipe: helm@v1
    with:
      release: app
      namespace: default
      files: |
        chart/values.yaml
      set: |
        image.tag=${CI_COMMIT_SHA}
    secrets:
      KUBECONFIG: kubeconfig_prod
  gc:
    depends_on: [deploy]
    pipe: oci-gc@v1
    with:
      repos: |
        registry.example.com/app
```

Publish a package to the org's registry on a tag:

```yaml
when:
  event: tag
steps:
  publish:
    pipe: pypi-publish-acahti@v1
```

End-to-end tests after a deploy job named `cd`:

```yaml
depends_on: [cd]
steps:
  e2e:
    pipe: uv@v1
    secrets:
      ARGOS_DASH_URL: argos_dash_url
      ARGOS_TOKEN: argos_token
    with:
      run: |
        argos run all --dash
```

When a job is named `e2e…` or has a step named `e2e`, the pipeline page links the Argos run it reports.