# The TestFinch CLI

> Run your tests from a terminal, an agent or CI, on cloud browsers or against localhost, and fix bugs from the shell.

The `testfinch` command runs your tests from a terminal, a coding agent or a CI job.
It runs them on cloud browsers or on your own computer, against `localhost` or any other address.
It also maps a site, and it gives you the bugs you are fixing, with their evidence, as files you can open.

It needs Node.js 20.11 or newer.

## Install

Run it without installing:

```sh
npx @testfinch/cli@0.1 status
```

Or install it once:

```sh
npm install -g @testfinch/cli
testfinch --version
```

## Sign in

```sh
testfinch login
```

Your browser opens on TestFinch.
Approve the sign-in and the command finishes.

The approval page asks you to allow **Run tests on this computer**.
If you allow it, the CLI receives each project's test values, secret ones included, to type them into your site.
Anything that can run commands as you on this computer, a coding agent included, can read them too.
If you do not allow it, everything works except runs on this computer.

Over SSH, in a dev container, or on Linux without a display, the CLI prints an address and a code instead:

```text
Open https://app.testfinch.com/device and type the code ABCD-EFGH.
```

Open the address on any device and type the code.
Nothing is filled in for you, so you can check that you started it.
`testfinch login --device` asks for this anywhere.
Set `TESTFINCH_NO_BROWSER=1` and the CLI never opens a browser: it prints the address to open instead.
With `--ndjson`, the address and the code arrive as a `progress` event with `verification_uri` and `user_code`.

To sign in with a personal access token, create one in TestFinch under Settings, API tokens, and pipe it in:

```sh
printf '%s' "$TOKEN" | testfinch login --with-token
```

A personal access token cannot run tests on this computer.

`testfinch logout` ends the sign-in on TestFinch and forgets it here.
If TestFinch cannot be reached, the sign-in is forgotten here and ends by itself within 30 days.

A browser sign-in lasts 90 days at most, however often you use it.
After that, a command stops with exit code 3: run `testfinch login` again.
To end a sign-in from anywhere, for example on a lost laptop, use Signed-in apps in your TestFinch account settings.

### Where the sign-in is kept

The CLI keeps it in your computer's credential store: the macOS Keychain, the Windows Credential Manager, or the Linux Secret Service.
Where there is none, it keeps it in a file only you can read, and says so:

| System | The CLI's folder |
| --- | --- |
| macOS | `~/Library/Application Support/TestFinch CLI` |
| Windows | `%APPDATA%\TestFinch\CLI` |
| Linux | `$XDG_CONFIG_HOME/testfinch`, or `~/.config/testfinch` |

Set `TESTFINCH_CREDENTIAL_STORE=keychain` to refuse the file, or `TESTFINCH_CREDENTIAL_STORE=file` to use the file even where a credential store exists.

Do not share the CLI's folder between computers or containers.
Each one signs in on its own.

On macOS, any program you run with the same `node`, such as a dev server the CLI starts, can read the sign-in from the Keychain without asking.
If you think another program read it, end the sign-in under Signed-in apps in TestFinch.

### Which sign-in a command uses

The CLI uses the first of these it finds:

1. `TESTFINCH_TRIGGER_TOKEN`: a CI trigger's token.
   It starts that trigger's runs and nothing else.
2. `TESTFINCH_TOKEN`: a personal access token, for CI and scripts.
3. The sign-in stored by `testfinch login`.

### Where your sign-in is sent

A sign-in or token goes only to the API address from `--api`, then the `TESTFINCH_API_URL` environment variable, then `https://api.testfinch.com`.
The address in `testfinch.json` never chooses it.
If `testfinch.json` names a different API than the one the command uses, the command stops with exit 2 and tells you to pass `--api` with that address.
An address that starts with `http://` works only for this computer (`localhost` or `127.0.0.1`).
Everything else must be `https`.

## Link a repository

```sh
testfinch init
```

`init` asks which project and which default environment this repository uses, and writes `testfinch.json` at the repository root.
Commit that file.
Every command run anywhere below it finds it, so it works from a package inside a monorepo too.
`init --env <id>` chooses the environment without asking, and `init --force` replaces a file that is already there.

```json
{
  "$schema": "https://testfinch.com/schemas/cli-config-1.json",
  "version": 1,
  "project": "prj_...",
  "environment": "staging",
  "local": { "start": "pnpm dev", "waitFor": "http://localhost:3000", "url": "http://localhost:3000" }
}
```

| Key | What it does |
| --- | --- |
| `version` | Always `1`. |
| `project` | The project the commands act on. |
| `environment` | The environment `test` uses when you do not name one. |
| `api` | For a self-hosted TestFinch. It must be the same address the command uses (see above); it never moves where your sign-in goes. |
| `local.start` | The command that starts your dev server for runs on this computer. |
| `local.waitFor` | The address that answers once the server is ready. |
| `local.url` | The address `testfinch test --local` tests when you give none. |

A key that is not in this table is refused.
Never put a secret in `testfinch.json`.
Test values live in the project's variables on TestFinch.

These override the file:

| Setting | Overrides |
| --- | --- |
| `--project <id>` or `TESTFINCH_PROJECT` | `project` |
| `--env <id>` (on `init` and `test`) or `TESTFINCH_ENVIRONMENT` | `environment` |

The order is the flag, then the environment variable, then the file.

What the commands write goes in a `.testfinch/` folder beside `testfinch.json`.
It keeps itself out of git.

## Commands

Every command takes these:

| Flag | What it does |
| --- | --- |
| `--json` | Prints one JSON document when the command finishes. |
| `--ndjson` | Prints one JSON event per line while `map` or `test` runs. |
| `--quiet` | No progress on stderr. |
| `--project <id>` | The project to act on. |
| `--api <url>` | The TestFinch API. |
| `-v`, `--version` | Prints the version. |

### `testfinch status`

Shows who you are, the linked project, its environments, whether cloud browsers are available, the mapping in progress and the latest runs.
It also says whether this version of the CLI is still supported.

### `testfinch bugs`

Lists open bugs.

| Flag | What it does |
| --- | --- |
| `--status <list>` | Comma-separated: `inbox`, `triaged`, `in_progress`, `fixed`, `verified`, `closed`, `rejected`. |
| `--severity <list>` | Comma-separated: `blocker`, `major`, `minor`, `trivial`. |
| `--mine` | Only bugs assigned to you. |
| `--query <text>` | Search text. |
| `--release <id>` | Only bugs in this release. |
| `--limit <n>` | At most this many, 1 to 50. The default is 20. |

### `testfinch bug <ref>`

Prints one bug: what the reporter saw and did, the page, the console errors and failed requests, the tests linked to it and the comments.
Its screenshots and other attachments are saved to `.testfinch/bugs/<ref>/`, and their paths are printed.

### `testfinch resolve <ref> --note <text>`

Marks the bug fixed and adds your note as a comment.
`--commit <sha>` adds the commit.
The person who reported the bug checks the fix and verifies it.
The CLI never verifies or closes a bug.

### `testfinch test [url]`

Runs the project's tests and gives a verdict.

| Flag | What it does |
| --- | --- |
| `--cloud` | Run on cloud browsers. |
| `--local` | Run on this computer. |
| `--env <id>` | The environment to run. |
| `--test <id>` | One test, by flow id or `qat_` id. Repeat for more. |
| `--module <path>` | The tests in a module. |
| `--suite <id>` | The tests in a suite. |
| `--bug <ref>` | The tests linked to a bug. |
| `--workers <n>` | Tests at once, 1 to 16. The default is 1. |
| `--label <text>` | The run's label. The default is `cli <branch>@<sha>`. |
| `--start <command>` | Start your dev server first and stop it after. Runs on this computer only. |
| `--wait-for <url>` | The address that answers once the server is ready. The default is the address you test. |
| `--start-timeout <seconds>` | How long the server may take to be ready. The default is 120. |
| `--headed` | Show the browser. Runs on this computer only. |
| `--browser chrome` | Use the Google Chrome installed on this computer instead of Chromium. |
| `--timeout <minutes>` | Stop and cancel the runs after this long. |

Choose one of `--test`, `--module`, `--suite` and `--bug`.
With none of them, every recorded test runs.

Give an address to test that address.
If it is one of the project's environments, that environment runs.
If it is any other address, the chosen environment runs against it, as a deploy hook does.
An address on this computer or a private network runs on this computer.
A public address runs in the cloud.
`--local` and `--cloud` choose for you, and `--cloud` with an address a cloud browser cannot reach stops with exit 2.

After a run, read `.testfinch/runs/<run-id>/results.json`.
It names each test's verdict and, for a failure, the step that failed.
A run on this computer saves the screenshots of failing steps in the same folder.

### `testfinch runs show <run-id>`

Prints a run's verdict.
`--artifacts` saves the failing tests' screenshots and `results.json` under `.testfinch/runs/<run-id>/`.
Screenshots are kept for your plan's retention, and the command says so when they are gone.

If the run is still going, `runs show` prints `"ok": false` and exits 0, because nothing has failed yet.
Check `status` in the output, and ask again later.

### `testfinch map <url>`

Sends the AI agent through the site in a cloud browser, and it proposes tests for the journeys it finds.
See [Map a site](#map-a-site).

### `testfinch questions` and `testfinch answer <question-id>`

List and answer what the AI agent asks while it maps.
See [Map a site](#map-a-site).

### `testfinch browsers install`

Downloads Chromium for runs on this computer.
`--with-deps` also installs the system libraries Chromium needs on Linux.

### `testfinch guide`

Prints the guide for coding agents.

## Run on this computer

```sh
testfinch browsers install
testfinch test http://localhost:3000
```

The first command downloads Chromium once.
If you skip it, `test` offers to install it at a terminal, and stops with exit 2 and the command to run in a script.
Runs on this computer use Chromium only.
`--browser chrome` uses the Google Chrome installed here instead.
If the project runs on other browsers too, those run in the cloud only, and the CLI says which it left out.

Your sign-in has to allow **Run tests on this computer**.
If it does not, the command stops with exit 3 and tells you to run `testfinch login` again and approve it.

Start your server for the run:

```sh
testfinch test http://localhost:3000 --start "pnpm dev"
```

The CLI starts the server, waits until the address answers, runs the tests and stops the server, also when you press Ctrl-C.
Your server does not receive `TESTFINCH_TOKEN` or `TESTFINCH_TRIGGER_TOKEN`.
Put `local.start` and `local.waitFor` in `testfinch.json` to do the same without the flag.
`--wait-for` is the address that answers when it is not the one you test.
`--start-timeout` changes how long the server may take.

### What stays on your computer

TestFinch receives each test's result and the failing step.
The screenshots, and the run's folder `.testfinch/runs/<run-id>/`, stay on your computer.

### Site passwords and sign-ins

Site passwords and sign-ins are sent only to the environment's own address.
If you test another address, such as `localhost` or a preview, the CLI tells you before the run, and a test that needs one stops and says so.

### The hourly limit

You can start up to 600 runs on this computer an hour, to protect against a loop gone wrong.
Past that, the command stops with exit 4 and says when to try again.
Runs on this computer never count against your plan.

## Map a site

```sh
testfinch map https://staging.example.com --goal "the checkout"
```

The address has to be on one of the project's environments.
`--add-environment <id>` adds it as a new one, for project admins.
Mapping runs on cloud browsers, so an address on this computer or a private network is refused, with a pointer to FlowQA Desktop.

| Flag | What it does |
| --- | --- |
| `--goal <text>` | What to focus on. |
| `--journeys <n>` | How many tests to propose, 1 to 60. |
| `--budget <pages>` | The most pages to map, 5 to 200. The default is 60. |
| `--opening <id>` | An opening to run first, such as `auth/sign-in`. |
| `--context-file <path>` | A brief or PRD for the AI agent to read. |
| `--until <stage>` | `review` (the default), `done` or `started`. |
| `--yes` | Start even when it uses credits. |
| `--timeout <minutes>` | Stop following after this long. The mapping goes on. |

By default the command returns when the plan is ready for review.
`--until done` keeps going until the tests are published.
`--until started` returns at once with the mapping's id, and `testfinch map show <id>` follows it later.

When the mapping would use credits, the CLI asks first.
Without a terminal it needs `--yes`, or it exits 5 without starting.

### Plans need a person

The CLI never approves a plan.
What gets tested is your call, so the command prints the plan and a link, and you review it in TestFinch.
With `--until done` and nobody at the terminal, `map` exits 5 at the review.

### Questions

The AI agent may ask a question while it maps: which of two buttons is right, or a value to type.
`testfinch questions` lists the open ones.
`--screenshots` saves the page each one is about to `.testfinch/questions/<question-id>.jpg`.

Answer with `testfinch answer <question-id>` and one of:

| Flag | For |
| --- | --- |
| `--choice <id>` | Which element to use. |
| `--value-stdin` | A value, piped in. |
| `--allow` or `--deny` | Whether it may follow a link. |
| `--yes` or `--no` | Whether the goal is reached. |
| `--give-up` | Giving this journey up. |

Add `--note <text>` for a sentence to the AI agent, `--exploration <id>` to find the question faster, and `--no-remember` to not keep the answer for next time.

Pipe a value in, so it never appears on a command line or in your shell history:

```sh
printf '%s' "$PROMO_CODE" | testfinch answer qqs_... --exploration qxp_... --value-stdin --no-secret
```

A value is kept as a secret unless you add `--no-secret`.
A question about a test account, the site's password or what opens the site can only be answered by you in TestFinch.
`answer` exits 5 with the link.

## Output and exit codes

Data goes to stdout.
Progress and notices go to stderr.
So stdout is always data you can parse.

### Exit codes

The exit codes are the contract.
Scripts and agents branch on them.

| Exit | Meaning |
| --- | --- |
| 0 | Done. For `test`, every run finished with no failed or blocked test. |
| 1 | Tests failed or were blocked. |
| 2 | Usage or configuration: a bad flag, no linked project, a URL that matches no environment, no browser installed. |
| 3 | Not signed in, the sign-in or token no longer works, or you are not allowed to do this. |
| 4 | The plan's limit or credits ran out. |
| 5 | A person needs to act: approve a plan, confirm spending credits, or answer a question only they can answer. |
| 6 | Did not finish: cancelled, no browser became free in time, the browser was lost before a test could finish, or `--timeout` was reached. |
| 7 | TestFinch could not be reached, or kept failing after retries. Try again later. |
| 8 | This version of the CLI is no longer supported. Upgrade with `npm install -g @testfinch/cli@latest`. |
| 70 | A bug in the CLI. Run again with `TESTFINCH_DEBUG=1` and report what it prints. |
| 130 | Interrupted. The run was cancelled first. |

### `--json`

One JSON document, with keys in `snake_case`.
`testfinch test --json` prints:

```json
{
  "ok": false,
  "target": "cloud",
  "environment": "staging",
  "base_url": null,
  "executions": [
    {
      "execution_id": "qex_...",
      "project_id": "prj_...",
      "status": "completed",
      "target": "cloud",
      "environment": "staging",
      "base_url": null,
      "combination": null,
      "summary": { "total": 2, "passed": 1, "failed": 1, "blocked": 0, "flaky": 0, "skipped": 0 },
      "failures": [
        { "test": "checkout/total", "status": "fail", "step": 3, "step_text": "Check that the page shows Total 20", "error": "...", "screenshot": null }
      ],
      "error": null,
      "run_dir": "/path/to/repo/.testfinch/runs/qex_...",
      "web_link": "https://app.testfinch.com/p/prj_.../runs?run=qex_...",
      "screenshots_removed": null
    }
  ],
  "skipped": []
}
```

`ok` is true only when every run completed with no failed or blocked test.
Flaky and skipped tests do not fail the job.

### `--ndjson`

One JSON object per line while `map` or `test` runs.
Each has a `type` of `progress`, `question` or `result`, and the last line of a failed command has the type `error`.

```json
{"type":"progress","message":"Run qex_... is running","execution_id":"qex_...","status":"running"}
{"type":"progress","message":"pass  home/open","execution_id":"qex_...","test":"home/open","status":"pass"}
{"type":"result","ok":true,"target":"cloud","environment":"staging","base_url":null,"executions":[],"skipped":[]}
```

A `question` event carries `question_id`, `exploration_id`, `kind`, `person_only`, `text`, `options`, `context`, `person_link` and `web_link`.

### Errors

When a command fails, `--json` prints this and exits with `exit_code`:

```json
{
  "error": {
    "code": "signed_out",
    "message": "You are not signed in. Run testfinch login, or set TESTFINCH_TOKEN.",
    "exit_code": 3
  }
}
```

`details` is added when the error has any.
The message says what to do next.
With `--ndjson`, the same object is the last line, as `{"type":"error","error":{...}}`.

## Run in CI

On cloud browsers, against an address your pipeline deploys, use a personal access token:

```yaml
      - run: npx @testfinch/cli@0.1 test "${{ github.event.deployment_status.environment_url }}" --cloud --json > testfinch-result.json
        env:
          TESTFINCH_TOKEN: ${{ secrets.TESTFINCH_TOKEN }}
          TESTFINCH_PROJECT: prj_...
```

On the CI machine, against a server the job starts, use a CI trigger's token:

```yaml
name: TestFinch
on: [pull_request]
jobs:
  testfinch:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22 }
      - run: npm ci
      - run: npx @testfinch/cli@0.1 browsers install --with-deps
      - run: npx @testfinch/cli@0.1 test http://localhost:3000 --local --start "npm run dev" --label "pr ${{ github.event.number }}"
        env:
          TESTFINCH_TRIGGER_TOKEN: ${{ secrets.TESTFINCH_TRIGGER_TOKEN }}
      - if: failure()
        uses: actions/upload-artifact@v4
        with: { name: testfinch-runs, path: .testfinch/runs }
```

A personal access token cannot run tests on this computer, so this recipe needs a trigger.
Create one in the project's Settings, then Schedules & CI, under CI triggers, and turn on **Run tests on the CI machine**.
It is off by default.
With it on, the CI machine receives each test value, secret ones included, to type them into your site, and so can anyone who reads your CI logs or changes your pipeline.
Without it, the trigger runs on cloud browsers only.
A trigger created earlier shows a **Run on the CI machine** button to turn it on later.

A trigger's token starts that trigger's tests only.
`--test <flow-id>` narrows them; `--module`, `--suite` and `--bug` are not available.
Without an address, the trigger runs in the cloud.

The job fails on any exit other than 0.
Exit 1 means a test failed, so read `.testfinch/runs`; exit 6 means the run did not finish.

### Which addresses a run may use

When you give an address that is not the environment's own, it has to be one the environment allows.
The rule is the same for the CLI and for CI:

- An address on this computer or a private network runs on the machine that runs the CLI, and never in the cloud.
- Any other address has to be the environment's address or one of its allowed hosts, which you set in the project's settings. Otherwise the run is refused and says which host to add.
- A cloud run needs `https`.

## Upgrades

Once a day, the CLI asks the API whether a newer version is out, and says so on stderr.
It stays quiet under `--json`, `--ndjson` and `--quiet`, when stderr is not a terminal, and in CI.
Set `TESTFINCH_NO_UPDATE_NOTICE=1` to turn it off.
The question carries no sign-in.

When a version is no longer supported, the API refuses it and the command exits 8.
Upgrade with `npm install -g @testfinch/cli@latest`.

## Settings you can set in the environment

| Variable | What it does |
| --- | --- |
| `TESTFINCH_TOKEN` | A personal access token. |
| `TESTFINCH_TRIGGER_TOKEN` | A CI trigger's token. |
| `TESTFINCH_API_URL` | The TestFinch API. The default is `https://api.testfinch.com`. |
| `TESTFINCH_PROJECT` | The project to act on. |
| `TESTFINCH_ENVIRONMENT` | The environment `test` runs. |
| `TESTFINCH_CREDENTIAL_STORE` | `keychain` or `file`. |
| `TESTFINCH_NO_UPDATE_NOTICE` | `1` turns off the daily check for a newer version. |
| `TESTFINCH_NO_BROWSER` | `1`: never open a browser; the address to open is printed instead. |
| `TESTFINCH_APP_URL` | The web app the links point to, for a self-hosted TestFinch. |
| `TESTFINCH_AGENT` | Names the agent driving the CLI, shown with what it does in TestFinch. |
| `TESTFINCH_DEBUG` | `1` adds details to a bug report and to local runs. |

## What the CLI sends

The CLI sends no usage statistics.
It talks to:

- the TestFinch API, for everything it does, and for the daily version check;
- the site you test, from a run on this computer;
- Playwright's download servers, only while `testfinch browsers install` runs;
- your own browser, which it opens on TestFinch to sign in.

Each request to the API says it comes from the CLI, with its version, and which coding agent is driving it when one is.
