TestFinch

DocsAgents and API

The TestFinch CLI guide for agents

You are working in a repository whose product has tests on TestFinch. The testfinch command maps the site, runs its tests in the cloud or on this computer, and gives you what you need to fix a bug. Read this guide once per session.

1. The CLI or the MCP connector#

Use the CLI when you have a terminal in the repository. It runs tests against the app on this computer (http://localhost:3000 and the like), writes results and screenshots to files you can open, and its exit codes tell you what happened. Use the MCP connector when you have no terminal, or to write and save tests: the connector has the authoring tools and their guide (flowqa_guide). Both work on the same projects, tests, runs and bugs, so you can switch between them.

2. Signing in#

The CLI uses the first of these it finds:

  1. TESTFINCH_TRIGGER_TOKEN: a CI trigger's token. It starts runs only, with the trigger's own choice of tests. It runs tests on the CI machine only when the person turned on Run tests on the CI machine for the trigger, in Schedules & CI under CI triggers.
  2. TESTFINCH_TOKEN: a personal access token, for CI and scripts. It can do everything except run tests on this computer.
  3. The sign-in stored by testfinch login, for a person and the agents working beside them.

If nothing is signed in, ask the person to run testfinch login. It opens their browser to sign in. With TESTFINCH_NO_BROWSER=1 (CI, SSH, an agent's sandbox) it never opens a browser and prints the address to open instead. Over SSH or without a display it prints an address and a code instead, and the person opens the address and types the code; nothing is filled in for them, and --device asks for this anywhere. With --ndjson, the link and the code arrive as a progress event with verification_uri and user_code. testfinch login --with-token stores a personal access token read from stdin. testfinch logout ends the stored sign-in on TestFinch and forgets it here. Never ask the person to paste a token into the conversation.

3. Linking the repository: testfinch init and testfinch.json#

testfinch init asks which project and default environment this repository uses and writes testfinch.json at the repository root. Commit that file. Every command run anywhere below it finds it, so it works from a package inside a monorepo too.

{
  "$schema": "https://testfinch.com/schemas/cli-config-1.json",
  "version": 1,
  "project": "prj_...",
  "environment": "staging",
  "local": { "start": "pnpm dev", "waitFor": "http://localhost:3000", "url": "http://localhost:3000" }
}

api is optional, for a self-hosted TestFinch. It never chooses where a sign-in or token goes: that is --api, then TESTFINCH_API_URL, then https://api.testfinch.com. If api names a different address than the command uses, the command stops with exit 2 until you pass --api with the same address. An API address over plain http works only for this computer (localhost or 127.0.0.1); anywhere else it must be https. Never put a secret in testfinch.json: test values live in the project's variables on TestFinch. --project <id> and TESTFINCH_PROJECT choose a project without the file, and win over it, as does TESTFINCH_ENVIRONMENT. What commands write goes in .testfinch/ beside testfinch.json, which keeps itself out of git.

4. Reading the output#

Pass --json and read stdout: you get exactly one JSON document, with keys in snake_case. --ndjson prints one JSON object per line while map or test runs, each with a type of progress, question or result. If the command fails, the last line has the type error and the same error object as --json. Progress and notices go to stderr, so stdout is always data. An error in --json is { "error": { "code", "message", "details", "exit_code" } }; the message says what to do next.

Branch on the exit code:

Exit Meaning
0 Done. For test, every run finished with no failed or blocked test.
1 Tests failed or were blocked.
2 Usage or configuration: a bad flag, no linked project, a URL that matches no environment, no browser installed.
3 Not signed in, the sign-in or token no longer works, or you are not allowed to do this.
4 The plan's limit or credits ran out.
5 A person needs to act: approve a plan, confirm spending credits, or answer a question only they can answer.
6 Did not finish: cancelled, no browser became free in time, the browser was lost before a test could finish, or --timeout was reached.
7 TestFinch could not be reached, or kept failing after retries. Try again later.
8 This version of the CLI is no longer supported. Upgrade with npm install -g @testfinch/cli@latest.
70 A bug in the CLI. Run again with TESTFINCH_DEBUG=1 and report what it prints.
130 Interrupted. The run was cancelled first.

5. Running tests: testfinch test#

testfinch test runs every recorded test of the linked project. Narrow it with --test <id> (repeatable), --module <path>, --suite <id> or --bug <ref>. Give a URL to test that site: testfinch test http://localhost:3000. A localhost or private address runs on this computer; a public one runs in the cloud. --local and --cloud choose for you. Runs on this computer use Chromium only; --browser chrome uses the Google Chrome installed there instead. Install Chromium once before the first run on this computer: testfinch browsers install. The person's sign-in must allow Run tests on this computer, or the command exits 3 and says so. Site passwords and sign-ins are sent only to the environment's own address. When you test another address, such as localhost, a test that needs one stops and says so. --start "<command>" starts your dev server, waits until the URL answers, runs the tests and stops the server. local.start and local.waitFor in testfinch.json do the same without the flag.

After a run, read .testfinch/runs/<run-id>/results.json. It names each test's verdict and, for a failure, the step that failed. The screenshots of failing steps are in the same folder; open them to see what the page showed. testfinch runs show <run-id> --artifacts fetches the same files for a run that happened elsewhere. runs show on a run that is still going prints "ok": false and exits 0, because nothing has failed yet: check status in the output and ask again later.

6. Mapping a site: testfinch map#

testfinch map <url> sends the AI agent through the site in a cloud browser and proposes tests for the journeys it finds. By default it returns when the plan is ready for review (--until review). --until done keeps going until the tests are published; --until started returns at once with the map's id.

The CLI never approves a plan without a person. A plan is the person's call on what gets tested, so print the plan and its link, and ask them to review it in TestFinch. With --until done and no person at the terminal, map exits 5 at the review. When a map would use credits, it asks first; without a terminal it needs --yes, or it exits 5 without starting.

The AI agent may ask a question while it maps, such as a value to type or which of two buttons is right. testfinch questions lists the open ones; --screenshots saves each page it was looking at to .testfinch/questions/<question-id>.jpg. Answer with testfinch answer <question-id> --exploration <exploration-id> and one of --choice <id>, --value-stdin, --allow, --deny, --yes, --no or --give-up, with --note <text> if it helps. Pipe a value into --value-stdin, so it never appears on a command line or in your transcript. It is kept as a secret unless you add --no-secret. A question about a test account, the site's password or what opens the site can only be answered by the person in TestFinch. answer exits 5 with its link: give it to them.

7. Fixing a bug#

  1. testfinch bugs lists the open bugs; --mine, --severity, --query and --release narrow it.
  2. testfinch bug <ref> prints the bug: what the reporter saw and did, the page, the console errors and failed requests, the tests linked to it and the comments. Its screenshots and other attachments are saved to .testfinch/bugs/<ref>/, and their paths printed, so open them.
  3. Fix the code.
  4. testfinch test --bug <ref> runs the tests linked to the bug against your fix.
  5. testfinch resolve <ref> --note "<what you changed>" --commit <sha> marks the bug fixed and comments with your note and commit.

The person who reported the bug checks the fix and verifies it. Do not mark a bug verified or closed yourself.

8. Secrets#

Nothing the CLI prints or stores contains a secret value: test values marked secret appear as their names only. Its own sign-in lives in this computer's credential store, or in a file only you can read where there is none, and never in the repository. Do not share the CLI's config folder between computers or containers: each one signs in on its own.