TestFinch

BlogFlowQA

Autonomous test generation: how it works, where it fails, what to review

An agent can map a site, propose journeys, write the tests and check that each one can fail. It still needs a person at three points. Here is what happens at each and what to look at.

Autonomous test generation works in four stages: an agent maps the site, proposes a plan of journeys, generates a test for each, and checks every test by replaying it and then breaking the page to watch it fail. It fails in predictable places: pages it cannot reach, values it cannot know, and outcomes it cannot judge. The review it needs is small and specific, and it comes at three points: the plan, the questions, and the check results.

Stage one: map#

The agent opens the site the way a person would, follows the navigation, and records what each page is for, what it contains, and where it leads. It stays inside the hosts you allow and stops at the depth you set. The output is a map: pages, their purposes, the forms and actions on them. A good tool keeps this map as notes the next run reads, so the second exploration starts where the first ended.

Where it fails. Pages behind a login it does not have, flows that need a real email or a payment, and anything that only appears after a background job. It should say so rather than guess.

Stage two: plan#

From the map the agent proposes journeys: "sign up with email and reach the dashboard", "create a project and see it in the list". Each has a start page, the steps it expects, and the outcome that would show it worked.

What to review. This is the first point where a person's time is well spent. A journey crossed off here is a test nobody has to generate, check or maintain, and an outcome corrected here is cheaper than a wrong assertion found later. Cross off journeys that do not matter, add the ones the agent missed because it could not see them, and correct outcomes that are wrong: the agent may think "a toast appears" shows a save worked when you know the toast appears even when the save fails.

Stage three: generate#

For each journey the agent drives the browser, fills forms with values it invents or you provide, and writes a test from what it did, with assertions on the outcome.

Where it asks. It cannot know your test account's password, which of two similar buttons you meant, or whether it may follow a link to another host. A good tool asks, with a screenshot, and waits. The answers are kept, so it asks once. On a desktop it can also hand you the browser for the part it cannot do, and take the recording back when you are done.

Stage four: check#

A generated test that passes once tells you little; it might pass on a blank page, or only in the session that wrote it. So the agent replays each test twice, each time on a fresh session, because assertions written while logged in and warmed up often depend on state a new browser will not have. Then it breaks the page the way the test's own assertions describe, hiding the text or the element each assertion looks for, and runs the test once more. Only a test that passes both replays and fails on the broken page is kept.

That shows two things: the test is repeatable, and its assertions can fail. It does not show that the assertions check the right thing. A test that only checks that a confirmation message appears will not notice a wrong total. And a test whose only assertion is a URL cannot be broken from inside the page, so it skips the broken-page run and is marked as unchecked.

What to review. The check results, and the assertions themselves. A journey that could not be kept is the agent telling you it does not have a reliable assertion for that outcome, and that is where a person adds one. A kept test is worth a quick read too: is it asserting the outcome you care about, or only one that happens beside it?

A worked example#

A composed example: the team, the timings and the counts show the shape of the work and are not measurements from one customer.

A team points the agent at their staging site with one test account. The map finds twenty-three pages. The plan proposes eleven journeys; the reviewer removes two (an admin-only report nobody uses) and corrects one outcome. During generation the agent asks three questions: a verification code from the inbox, which of two "New" buttons to use, and whether it may follow the link to the billing provider's hosted page (no). Nine tests are generated; eight pass both replays and fail on the broken page. The ninth asserted a date that changes daily, and the agent flagged it rather than keeping a test that would fail tomorrow. The reviewer fixes that assertion by hand. The person's work was the plan review, three answers and one assertion, for nine tests that run on every deploy from then on.

The honest limits#

The agent is as good as the site is navigable. A product that hides everything behind hover states and custom widgets with no labels will need more questions and more review. That is also a finding about the product, and often a cheaper one to fix than the tests.

FlowQA explores your site, writes the tests, keeps each one only once it fails on a page broken the way its checks describe, and runs them after every change.

See FlowQA