TestFinch

DocsFlowQA

The TestFinch CLI

The testfinch command runs your tests from a terminal, a coding agent or a CI job. It runs them on cloud browsers or on your own computer, against localhost or any other address. It also maps a site, and it gives you the bugs you are fixing, with their evidence, as files you can open.

It needs Node.js 20.11 or newer.

Install#

Run it without installing:

npx @testfinch/cli@0.1 status

Or install it once:

npm install -g @testfinch/cli
testfinch --version

Sign in#

testfinch login

Your browser opens on TestFinch. Approve the sign-in and the command finishes.

The approval page asks you to allow Run tests on this computer. If you allow it, the CLI receives each project's test values, secret ones included, to type them into your site. Anything that can run commands as you on this computer, a coding agent included, can read them too. If you do not allow it, everything works except runs on this computer.

Over SSH, in a dev container, or on Linux without a display, the CLI prints an address and a code instead:

Open https://app.testfinch.com/device and type the code ABCD-EFGH.

Open the address on any device and type the code. Nothing is filled in for you, so you can check that you started it. testfinch login --device asks for this anywhere. Set TESTFINCH_NO_BROWSER=1 and the CLI never opens a browser: it prints the address to open instead. With --ndjson, the address and the code arrive as a progress event with verification_uri and user_code.

To sign in with a personal access token, create one in TestFinch under Settings, API tokens, and pipe it in:

printf '%s' "$TOKEN" | testfinch login --with-token

A personal access token cannot run tests on this computer.

testfinch logout ends the sign-in on TestFinch and forgets it here. If TestFinch cannot be reached, the sign-in is forgotten here and ends by itself within 30 days.

A browser sign-in lasts 90 days at most, however often you use it. After that, a command stops with exit code 3: run testfinch login again. To end a sign-in from anywhere, for example on a lost laptop, use Signed-in apps in your TestFinch account settings.

Where the sign-in is kept#

The CLI keeps it in your computer's credential store: the macOS Keychain, the Windows Credential Manager, or the Linux Secret Service. Where there is none, it keeps it in a file only you can read, and says so:

System The CLI's folder
macOS ~/Library/Application Support/TestFinch CLI
Windows %APPDATA%\TestFinch\CLI
Linux $XDG_CONFIG_HOME/testfinch, or ~/.config/testfinch

Set TESTFINCH_CREDENTIAL_STORE=keychain to refuse the file, or TESTFINCH_CREDENTIAL_STORE=file to use the file even where a credential store exists.

Do not share the CLI's folder between computers or containers. Each one signs in on its own.

On macOS, any program you run with the same node, such as a dev server the CLI starts, can read the sign-in from the Keychain without asking. If you think another program read it, end the sign-in under Signed-in apps in TestFinch.

Which sign-in a command uses#

The CLI uses the first of these it finds:

  1. TESTFINCH_TRIGGER_TOKEN: a CI trigger's token. It starts that trigger's runs and nothing else.
  2. TESTFINCH_TOKEN: a personal access token, for CI and scripts.
  3. The sign-in stored by testfinch login.

Where your sign-in is sent#

A sign-in or token goes only to the API address from --api, then the TESTFINCH_API_URL environment variable, then https://api.testfinch.com. The address in testfinch.json never chooses it. If testfinch.json names a different API than the one the command uses, the command stops with exit 2 and tells you to pass --api with that address. An address that starts with http:// works only for this computer (localhost or 127.0.0.1). Everything else must be https.

testfinch init

init asks which project and which default environment this repository uses, and writes testfinch.json at the repository root. Commit that file. Every command run anywhere below it finds it, so it works from a package inside a monorepo too. init --env <id> chooses the environment without asking, and init --force replaces a file that is already there.

{
  "$schema": "https://testfinch.com/schemas/cli-config-1.json",
  "version": 1,
  "project": "prj_...",
  "environment": "staging",
  "local": { "start": "pnpm dev", "waitFor": "http://localhost:3000", "url": "http://localhost:3000" }
}
Key What it does
version Always 1.
project The project the commands act on.
environment The environment test uses when you do not name one.
api For a self-hosted TestFinch. It must be the same address the command uses (see above); it never moves where your sign-in goes.
local.start The command that starts your dev server for runs on this computer.
local.waitFor The address that answers once the server is ready.
local.url The address testfinch test --local tests when you give none.

A key that is not in this table is refused. Never put a secret in testfinch.json. Test values live in the project's variables on TestFinch.

These override the file:

Setting Overrides
--project <id> or TESTFINCH_PROJECT project
--env <id> (on init and test) or TESTFINCH_ENVIRONMENT environment

The order is the flag, then the environment variable, then the file.

What the commands write goes in a .testfinch/ folder beside testfinch.json. It keeps itself out of git.

Commands#

Every command takes these:

Flag What it does
--json Prints one JSON document when the command finishes.
--ndjson Prints one JSON event per line while map or test runs.
--quiet No progress on stderr.
--project <id> The project to act on.
--api <url> The TestFinch API.
-v, --version Prints the version.

testfinch status#

Shows who you are, the linked project, its environments, whether cloud browsers are available, the mapping in progress and the latest runs. It also says whether this version of the CLI is still supported.

testfinch bugs#

Lists open bugs.

Flag What it does
--status <list> Comma-separated: inbox, triaged, in_progress, fixed, verified, closed, rejected.
--severity <list> Comma-separated: blocker, major, minor, trivial.
--mine Only bugs assigned to you.
--query <text> Search text.
--release <id> Only bugs in this release.
--limit <n> At most this many, 1 to 50. The default is 20.

testfinch bug <ref>#

Prints one bug: what the reporter saw and did, the page, the console errors and failed requests, the tests linked to it and the comments. Its screenshots and other attachments are saved to .testfinch/bugs/<ref>/, and their paths are printed.

testfinch resolve <ref> --note <text>#

Marks the bug fixed and adds your note as a comment. --commit <sha> adds the commit. The person who reported the bug checks the fix and verifies it. The CLI never verifies or closes a bug.

testfinch test [url]#

Runs the project's tests and gives a verdict.

Flag What it does
--cloud Run on cloud browsers.
--local Run on this computer.
--env <id> The environment to run.
--test <id> One test, by flow id or qat_ id. Repeat for more.
--module <path> The tests in a module.
--suite <id> The tests in a suite.
--bug <ref> The tests linked to a bug.
--workers <n> Tests at once, 1 to 16. The default is 1.
--label <text> The run's label. The default is cli <branch>@<sha>.
--start <command> Start your dev server first and stop it after. Runs on this computer only.
--wait-for <url> The address that answers once the server is ready. The default is the address you test.
--start-timeout <seconds> How long the server may take to be ready. The default is 120.
--headed Show the browser. Runs on this computer only.
--browser chrome Use the Google Chrome installed on this computer instead of Chromium.
--timeout <minutes> Stop and cancel the runs after this long.

Choose one of --test, --module, --suite and --bug. With none of them, every recorded test runs.

Give an address to test that address. If it is one of the project's environments, that environment runs. If it is any other address, the chosen environment runs against it, as a deploy hook does. An address on this computer or a private network runs on this computer. A public address runs in the cloud. --local and --cloud choose for you, and --cloud with an address a cloud browser cannot reach stops with exit 2.

After a run, read .testfinch/runs/<run-id>/results.json. It names each test's verdict and, for a failure, the step that failed. A run on this computer saves the screenshots of failing steps in the same folder.

testfinch runs show <run-id>#

Prints a run's verdict. --artifacts saves the failing tests' screenshots and results.json under .testfinch/runs/<run-id>/. Screenshots are kept for your plan's retention, and the command says so when they are gone.

If the run is still going, runs show prints "ok": false and exits 0, because nothing has failed yet. Check status in the output, and ask again later.

testfinch map <url>#

Sends the AI agent through the site in a cloud browser, and it proposes tests for the journeys it finds. See Map a site.

testfinch questions and testfinch answer <question-id>#

List and answer what the AI agent asks while it maps. See Map a site.

testfinch browsers install#

Downloads Chromium for runs on this computer. --with-deps also installs the system libraries Chromium needs on Linux.

testfinch guide#

Prints the guide for coding agents.

Run on this computer#

testfinch browsers install
testfinch test http://localhost:3000

The first command downloads Chromium once. If you skip it, test offers to install it at a terminal, and stops with exit 2 and the command to run in a script. Runs on this computer use Chromium only. --browser chrome uses the Google Chrome installed here instead. If the project runs on other browsers too, those run in the cloud only, and the CLI says which it left out.

Your sign-in has to allow Run tests on this computer. If it does not, the command stops with exit 3 and tells you to run testfinch login again and approve it.

Start your server for the run:

testfinch test http://localhost:3000 --start "pnpm dev"

The CLI starts the server, waits until the address answers, runs the tests and stops the server, also when you press Ctrl-C. Your server does not receive TESTFINCH_TOKEN or TESTFINCH_TRIGGER_TOKEN. Put local.start and local.waitFor in testfinch.json to do the same without the flag. --wait-for is the address that answers when it is not the one you test. --start-timeout changes how long the server may take.

What stays on your computer#

TestFinch receives each test's result and the failing step. The screenshots, and the run's folder .testfinch/runs/<run-id>/, stay on your computer.

Site passwords and sign-ins#

Site passwords and sign-ins are sent only to the environment's own address. If you test another address, such as localhost or a preview, the CLI tells you before the run, and a test that needs one stops and says so.

The hourly limit#

You can start up to 600 runs on this computer an hour, to protect against a loop gone wrong. Past that, the command stops with exit 4 and says when to try again. Runs on this computer never count against your plan.

Map a site#

testfinch map https://staging.example.com --goal "the checkout"

The address has to be on one of the project's environments. --add-environment <id> adds it as a new one, for project admins. Mapping runs on cloud browsers, so an address on this computer or a private network is refused, with a pointer to FlowQA Desktop.

Flag What it does
--goal <text> What to focus on.
--journeys <n> How many tests to propose, 1 to 60.
--budget <pages> The most pages to map, 5 to 200. The default is 60.
--opening <id> An opening to run first, such as auth/sign-in.
--context-file <path> A brief or PRD for the AI agent to read.
--until <stage> review (the default), done or started.
--yes Start even when it uses credits.
--timeout <minutes> Stop following after this long. The mapping goes on.

By default the command returns when the plan is ready for review. --until done keeps going until the tests are published. --until started returns at once with the mapping's id, and testfinch map show <id> follows it later.

When the mapping would use credits, the CLI asks first. Without a terminal it needs --yes, or it exits 5 without starting.

Plans need a person#

The CLI never approves a plan. What gets tested is your call, so the command prints the plan and a link, and you review it in TestFinch. With --until done and nobody at the terminal, map exits 5 at the review.

Questions#

The AI agent may ask a question while it maps: which of two buttons is right, or a value to type. testfinch questions lists the open ones. --screenshots saves the page each one is about to .testfinch/questions/<question-id>.jpg.

Answer with testfinch answer <question-id> and one of:

Flag For
--choice <id> Which element to use.
--value-stdin A value, piped in.
--allow or --deny Whether it may follow a link.
--yes or --no Whether the goal is reached.
--give-up Giving this journey up.

Add --note <text> for a sentence to the AI agent, --exploration <id> to find the question faster, and --no-remember to not keep the answer for next time.

Pipe a value in, so it never appears on a command line or in your shell history:

printf '%s' "$PROMO_CODE" | testfinch answer qqs_... --exploration qxp_... --value-stdin --no-secret

A value is kept as a secret unless you add --no-secret. A question about a test account, the site's password or what opens the site can only be answered by you in TestFinch. answer exits 5 with the link.

Output and exit codes#

Data goes to stdout. Progress and notices go to stderr. So stdout is always data you can parse.

Exit codes#

The exit codes are the contract. Scripts and agents branch on them.

Exit Meaning
0 Done. For test, every run finished with no failed or blocked test.
1 Tests failed or were blocked.
2 Usage or configuration: a bad flag, no linked project, a URL that matches no environment, no browser installed.
3 Not signed in, the sign-in or token no longer works, or you are not allowed to do this.
4 The plan's limit or credits ran out.
5 A person needs to act: approve a plan, confirm spending credits, or answer a question only they can answer.
6 Did not finish: cancelled, no browser became free in time, the browser was lost before a test could finish, or --timeout was reached.
7 TestFinch could not be reached, or kept failing after retries. Try again later.
8 This version of the CLI is no longer supported. Upgrade with npm install -g @testfinch/cli@latest.
70 A bug in the CLI. Run again with TESTFINCH_DEBUG=1 and report what it prints.
130 Interrupted. The run was cancelled first.

--json#

One JSON document, with keys in snake_case. testfinch test --json prints:

{
  "ok": false,
  "target": "cloud",
  "environment": "staging",
  "base_url": null,
  "executions": [
    {
      "execution_id": "qex_...",
      "project_id": "prj_...",
      "status": "completed",
      "target": "cloud",
      "environment": "staging",
      "base_url": null,
      "combination": null,
      "summary": { "total": 2, "passed": 1, "failed": 1, "blocked": 0, "flaky": 0, "skipped": 0 },
      "failures": [
        { "test": "checkout/total", "status": "fail", "step": 3, "step_text": "Check that the page shows Total 20", "error": "...", "screenshot": null }
      ],
      "error": null,
      "run_dir": "/path/to/repo/.testfinch/runs/qex_...",
      "web_link": "https://app.testfinch.com/p/prj_.../runs?run=qex_...",
      "screenshots_removed": null
    }
  ],
  "skipped": []
}

ok is true only when every run completed with no failed or blocked test. Flaky and skipped tests do not fail the job.

--ndjson#

One JSON object per line while map or test runs. Each has a type of progress, question or result, and the last line of a failed command has the type error.

{"type":"progress","message":"Run qex_... is running","execution_id":"qex_...","status":"running"}
{"type":"progress","message":"pass  home/open","execution_id":"qex_...","test":"home/open","status":"pass"}
{"type":"result","ok":true,"target":"cloud","environment":"staging","base_url":null,"executions":[],"skipped":[]}

A question event carries question_id, exploration_id, kind, person_only, text, options, context, person_link and web_link.

Errors#

When a command fails, --json prints this and exits with exit_code:

{
  "error": {
    "code": "signed_out",
    "message": "You are not signed in. Run testfinch login, or set TESTFINCH_TOKEN.",
    "exit_code": 3
  }
}

details is added when the error has any. The message says what to do next. With --ndjson, the same object is the last line, as {"type":"error","error":{...}}.

Run in CI#

On cloud browsers, against an address your pipeline deploys, use a personal access token:

      - run: npx @testfinch/cli@0.1 test "${{ github.event.deployment_status.environment_url }}" --cloud --json > testfinch-result.json
        env:
          TESTFINCH_TOKEN: ${{ secrets.TESTFINCH_TOKEN }}
          TESTFINCH_PROJECT: prj_...

On the CI machine, against a server the job starts, use a CI trigger's token:

name: TestFinch
on: [pull_request]
jobs:
  testfinch:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22 }
      - run: npm ci
      - run: npx @testfinch/cli@0.1 browsers install --with-deps
      - run: npx @testfinch/cli@0.1 test http://localhost:3000 --local --start "npm run dev" --label "pr ${{ github.event.number }}"
        env:
          TESTFINCH_TRIGGER_TOKEN: ${{ secrets.TESTFINCH_TRIGGER_TOKEN }}
      - if: failure()
        uses: actions/upload-artifact@v4
        with: { name: testfinch-runs, path: .testfinch/runs }

A personal access token cannot run tests on this computer, so this recipe needs a trigger. Create one in the project's Settings, then Schedules & CI, under CI triggers, and turn on Run tests on the CI machine. It is off by default. With it on, the CI machine receives each test value, secret ones included, to type them into your site, and so can anyone who reads your CI logs or changes your pipeline. Without it, the trigger runs on cloud browsers only. A trigger created earlier shows a Run on the CI machine button to turn it on later.

A trigger's token starts that trigger's tests only. --test <flow-id> narrows them; --module, --suite and --bug are not available. Without an address, the trigger runs in the cloud.

The job fails on any exit other than 0. Exit 1 means a test failed, so read .testfinch/runs; exit 6 means the run did not finish.

Which addresses a run may use#

When you give an address that is not the environment's own, it has to be one the environment allows. The rule is the same for the CLI and for CI:

  • An address on this computer or a private network runs on the machine that runs the CLI, and never in the cloud.
  • Any other address has to be the environment's address or one of its allowed hosts, which you set in the project's settings. Otherwise the run is refused and says which host to add.
  • A cloud run needs https.

Upgrades#

Once a day, the CLI asks the API whether a newer version is out, and says so on stderr. It stays quiet under --json, --ndjson and --quiet, when stderr is not a terminal, and in CI. Set TESTFINCH_NO_UPDATE_NOTICE=1 to turn it off. The question carries no sign-in.

When a version is no longer supported, the API refuses it and the command exits 8. Upgrade with npm install -g @testfinch/cli@latest.

Settings you can set in the environment#

Variable What it does
TESTFINCH_TOKEN A personal access token.
TESTFINCH_TRIGGER_TOKEN A CI trigger's token.
TESTFINCH_API_URL The TestFinch API. The default is https://api.testfinch.com.
TESTFINCH_PROJECT The project to act on.
TESTFINCH_ENVIRONMENT The environment test runs.
TESTFINCH_CREDENTIAL_STORE keychain or file.
TESTFINCH_NO_UPDATE_NOTICE 1 turns off the daily check for a newer version.
TESTFINCH_NO_BROWSER 1: never open a browser; the address to open is printed instead.
TESTFINCH_APP_URL The web app the links point to, for a self-hosted TestFinch.
TESTFINCH_AGENT Names the agent driving the CLI, shown with what it does in TestFinch.
TESTFINCH_DEBUG 1 adds details to a bug report and to local runs.

What the CLI sends#

The CLI sends no usage statistics. It talks to:

  • the TestFinch API, for everything it does, and for the daily version check;
  • the site you test, from a run on this computer;
  • Playwright's download servers, only while testfinch browsers install runs;
  • your own browser, which it opens on TestFinch to sign in.

Each request to the API says it comes from the CLI, with its version, and which coding agent is driving it when one is.