Agentic testing

Test users

A test user is a saved sign-in identity that Matcha runs as inside your app. When a run signs in, it does so as one of these users, so you never paste credentials into a prompt. Test users belong to a TeaHouse, so a whole team shares them, and Matcha reuses them across runs, tests, and groups.

A run that carries no test user runs signed out (anonymous). Site protection is a separate concern: it opens the deployment before your app loads, while a test user signs in once your app is open. See site protection.

What a test user holds

Each test user lives in a named environment (for example "Staging") and carries the facts Matcha needs to sign in:

  • A display name, so a person can pick it.
  • A login identifier: an email, a username, a phone number, or a custom label and value.
  • The secret: a password, and where relevant a TOTP secret.
  • An optional description telling an agent when to pick this user over the others.

Coldtea stores the password and TOTP secret as secrets and never returns them to the UI.

Auth methods

A web test user signs in one of three ways:

  • password: the app's own password form.
  • password_totp: a password plus a time-based one-time code.
  • sso: sign in through an SSO provider instead of a password form. The providers are Google, Microsoft, GitHub, Okta, and Auth0.

A mobile test user signs in with password or password_totp. Mobile does not use SSO.

Matcha-owned accounts

Some apps have no shared credential to hand over: they email a login code on every sign-in. For those, create the test user with matchaAccount set. Matcha creates and owns the account on an inbox it can read, so you supply no login identifier and no password. The created user echoes back the address Matcha minted. On each run, Matcha reads the emailed code and signs in.

Create and list test users

You create and list test users from the Matcha QA screen, or through the QA MCP tools.

  • qa_test_user_create creates a web test user. You give it a name, an environment name, the web app origin it signs into, the login identifier (and its kind, defaulting to email), and the password. Add a totpSecret to make it a password plus TOTP user. Add an ssoProvider to sign in through SSO instead of the password form. Set matchaAccount for an app that emails a login code, and Matcha owns the account instead: send no login identifier or password.
  • qa_test_user_list returns the test users in a TeaHouse, each with the id pair a run needs to run as it.

See MCP and local API for the tool inputs.

Run as a test user

To run as a test user, a run references the user by an id pair: the test user id and the auth target id. That pair is what qa_test_user_list hands back, and what you pass to qa_run_start as runAs.

  • A web run must run as a web test user or run signed out.
  • A mobile run must run as a mobile test user whose platform matches the run, or run signed out.

When you run under a group, the run inherits the group's identity, so you do not repeat the id pair on every run. A saved test can override its group's identity when a specific flow needs a different account.

Temporary inboxes for a run

Beyond saved test users, an agent can mint disposable email inboxes during a run and read them, for flows that need a fresh address: a sign-up, an invite, a magic link, or a one-time code. Each inbox is isolated to that run and expires after a day, so a sign-up gets a clean mailbox without touching a real one. You can watch this happen in the run's Tools tab as the agent creates an inbox and reads a message.

On this page