Runs and results

A run is one execution of a QA scenario. It moves through a set of statuses, ends with a verdict or a "couldn't run" reason, and carries the evidence Matcha collected along the way. This page is how you read one.

Run status

A run reports one of these statuses:

• queued: admitted, waiting for its turn or a free device.
• running: driving the app.
• stopping: a stop was requested and the run is winding down.
• completed: the run finished and carries a verdict.
• failed: the run ended without a verdict. A "couldn't run" run lands here.
• canceled: someone stopped the run.

While a run is running it can also be waiting for a slot: either for a free device or browser, or for another run signed into the same test user to finish. The status stays running while it waits. The agent can also pause the run on purpose to wait outside the browser for a longer external process, such as a deployment, a build, or a webhook, then resume in a later turn. The status stays running throughout.

Verdict versus "couldn't run"

A completed run carries a verdict:

• pass: the flow did what you asked. A flow that correctly refuses still passes.
• fail: Matcha observed a real defect in the app.
• warning: the flow worked, but something is worth your attention.

A run that never got a fair shot at the app carries no verdict. Coldtea marks it "Couldn't run" and shows the reason, and keeps it out of your pass and fail counts. The reasons are:

• Browser, device, or provider infrastructure failed.
• The QA budget is exhausted.
• No device or browser slot came free.
• A bot wall or access barrier hid the app under test.
• The test's credentials were refused before the flow began.
• Matcha could not drive the flow itself.

Read a fail as a claim about your app, and a "couldn't run" as a claim about the run. Fix the setup and run it again before treating a "couldn't run" as a product bug.

Steps

A run is a list of ordered steps. Each step has a title, a status, and an observation of what Matcha saw. A step's status is queued, running, passed, warning, or broken. A broken step can also show that Matcha is retrying around it, or that a later attempt recovered the flow, so a broken attempt that Matcha's own oversight caused and then fixed does not read as your app failing.

Matcha's internal scaffolding steps (observation, reflection, health, session lifecycle) stay in the run data for diagnostics but are hidden from the step list you read.

Evidence

A run collects evidence you can open:

• Screenshots, tied to the step they came from.
• A frame replay: a recorded playback of the run assembled from captured frames.
• A video recording, when one is available.

Activity

Alongside the steps, a run carries an Activity panel with two tabs:

• Tools: every tool call the agent made, each as a row with how long it took and an expandable input and output, or the error when a call failed. This is where you see the agent create a test inbox, sign in, or pause for an external process.
• Console: the browser console output from the app under test, filterable by level (error, warning, info, log, debug) and searchable by text. Use it to tie a failure to what the app logged.

Live view and replay

While a run is live, you can watch it, and while a step runs you can follow the agent's thinking. Web runs expose a live browser view. Mobile runs expose a read-only device frame stream. Once the run completes, the live view gives way to the recording, so you can replay what happened.

Ratings

You can rate a completed run helpful or not helpful. Coldtea keeps the rating with the run and uses it as a quality signal: a "not helpful" on a passed run is a candidate false pass worth a second look.

Reading runs through tools

An agent reads runs with qa_run_list (recent runs, metadata only) and qa_run_get (one run with its steps and summary). An agent can stop a run with qa_run_stop. See MCP and local API.

Related

• Share a run: hand a finished run to someone outside the app.
• PR reporting: how runs report back on a pull request.