Insights
Insights turns your results into trends: how reliable your suite is, how fast it runs and what it costs.
What Insights are
Every number on Insights is computed from your results with fixed rules. No AI is involved, so the same results always give the same figures. Charts appear as soon as the range holds at least one execution, and an open page refreshes by itself shortly after a run ends.
Insights is available to Owners and Admins.
Tabs and filters
| Tab | Cards |
|---|---|
| Overview | KPIs, Pass rate and what broke, cost summary, Flaky leaderboard, Top root causes, Slowest tests |
| Reliability | KPIs, Pass rate and what broke, Failures by profile, Flaky leaderboard, Always green, Top root causes, Time to fix |
| Performance | KPIs, Runs per day by result, Slowest tests, Run duration |
| Coverage | Never run, Suite growth and volume, Requirements proven over time |
| Cost | Cost KPIs and trend, cost by plan, the most expensive test cases and runs |
- Range: 7, 14, 30 (default) or 90 days.
- Filters: Environment (one) and Target (one or more browser profiles). Clicking a bar in Failures by profile filters by that profile.
- Always green lists tests with no failure in their last 6 or more verdicts: candidates to run less often.
- Never run lists cases that have never been executed, by folder.
KPI definitions
| KPI | Definition |
|---|---|
| Pass rate | Passed ÷ (passed + failed). Errors are excluded. |
| Infra error rate | Errors ÷ all executions |
| Flaky tests | Tests judged unreliable (see below), with the number of wasted re-runs |
| Median run | The median run duration, with the 95th percentile |
| Mean time to fix | From a problem's first sighting to its fix (approximate) |
| Executions | How many executions ran in the range |
Each KPI shows a 15-day trend and the change against the previous period.
How flaky tests are found
A "test" here is one test case on one browser profile with one dataset row. Qalion looks at its last 10 executions within 90 days:
- Error results are not verdicts. They show in the history but are not counted.
- With fewer than 6 verdicts, there is too little data to judge.
- A flip is a change from pass to fail or from fail to pass. Two flips or more make the test unreliable (flaky). One flip means it broke or recovered; none means it is stable.
- Flip rate = flips ÷ (verdicts − 1). Fail rate = fails ÷ verdicts.
- A wasted run is a failure that the very next execution contradicted with a pass.
The Flaky leaderboard sorts tests by flip rate, then fail rate, and offers an action per row:
| Flip rate | Action |
|---|---|
| Over 40% | Quarantine |
| 15% or more | Investigate (opens the last run) |
| Under 15% | Watch |
| Already isolated | Quarantined (click to take it out of quarantine) |
Quarantine
A quarantined test keeps running, but its results no longer count in pass rates. Use it for a test you know is unreliable while someone fixes it. Quarantine a test from the Flaky leaderboard, from Needs attention on the Dashboard, from a run ("Mark flaky · quarantine") or through Qalion AI. Owners, Admins and Testers can quarantine. Take a test out of quarantine and it counts again.
Export and Qalion AI
- Export downloads Flaky tests (CSV), with flip and fail rates, or Slowest tests (CSV), with median and 95th percentile against the previous period.
- Explain this trend opens Qalion AI to explain a change, with links to its sources.
Was this page helpful?