How the AI agent runs tests
Every execution is carried out by Qalion's AI web agent in a real browser. It works like a careful tester: it looks at the page, acts, and checks what it was told to expect.
What happens in an execution
1
Qalion starts a browser that matches the browser profile: engine, window size, device emulation, user agent, locale and time zone.
2
It applies the environment's Access entries (cookies, storage items, headers, basic auth) before the first page loads.
3
The agent reads the test case. Gherkin and free text are turned into steps. Dataset values replace $column references.
4
For each step, the agent looks at a screenshot of the page in which the interactive elements are marked, picks the element it needs and acts: click, type, scroll, navigate.
5
It then compares the page with the expected result and records a verdict for the step, with a screenshot.
6
The execution ends Passed, Failed or Error. The video, console log, network log and the agent's action log are kept.
Note
The agent runs on a Qalion model tier (Qalion Core unless your organization chose another). If the tier's model is busy or unavailable, Qalion moves the run to the tier's next model by itself. There is no model setting per run, and the browser runs without a visible window on Qalion's servers.
What the agent sees
- Screenshots of the page as it works.
- The test case: title, description, steps or scenario, and the images attached to steps as reference.
- Dataset values, except the values of secret columns.
- The names of the environment's Access entries, never their values.
Secrets
Values of Access entries are set by code and never sent to the AI. Values of secret dataset columns are typed by the agent into the page but replaced by a placeholder for the model, and never shown in its reasoning or in results.
Warning
The agent works from screenshots, so anything your app shows on screen in plain text is visible to the model. Test with dedicated accounts and data.
Failed or Error?
Failed means the agent reached a verdict: the app did not do what the step expected. Error means it never reached one: it timed out, got stuck on an element, couldn't reach the site, was cancelled, or hit a platform issue. Errors are left out of pass rates and are usually worth a re-run.
Limits
- Each execution must finish within the project's execution time limit (2 to 30 minutes, 15 by default).
- Executions use browser sessions. Your organization has a limit on sessions in use at once; extra executions wait in a queue.
- Executions use credits, based on the AI model calls they make. A run can't start on an empty balance.
- Without a tunnel, the agent can't reach localhost or private network addresses. See Private apps.
- Failed executions are not retried automatically. Re-run them from the run page.
Writing tests the agent understands
- Name things the way the page does: "Click Add to cart", not "click the second button".
- Make each expected result observable: a message, a heading, a value, a page that opens.
- Keep one behavior per case. Long chains are harder to judge and to debug.
- Attach an image to a step when the expected look matters.
- Put test data in datasets and credentials in secret columns.
Was this page helpful?