Test results and checks¶
How every run is graded, and what each result means. Only a checked pass is green and only a checked failure is red; uncertainty stays neutral.
Success checks¶
The agent never sees the check. It works the task, gives an answer, and an independent grader decides.
| Check | Passes when | Use it for |
|---|---|---|
| Answer mentions… | The agent's final answer contains the expected value. Numbers the task itself states aren't required again. | Facts: prices, dates, counts, statuses. Required for comparisons. |
| Reaches page… | The run ends on the given page. | Navigation: "get to the signup form". |
| Page shows text… | The final page shows the given text. | Outcomes visible on screen: "Note added". |
| None (agent self-reports) | — | Watching behaviour. Results show as Self-reported. |
Directory test prompts use their own checks: a positive prompt passes when the expected tool is called successfully (and, if set, the answer matches); a negative prompt passes when no tool is called.
Run results¶
| Result | Meaning |
|---|---|
| Passed | The check confirmed success. |
| Failed | The check confirmed failure, or the run stopped (step or time limit, blocked at sign-in, a tool error the agent gave up on). The report says why. |
| Not completed (agent-reported) | No check was set and the agent said it couldn't finish. |
| Self-reported | No check was set; the agent says it succeeded. |
| Inconclusive | The run finished but the check couldn't be applied (for example, the page needed for a page check never loaded). |
| Not run | The run never started, for example because the test was stopped or credits ran out. Unstarted runs are refunded. |
| Queued / Running | Still waiting or in progress. |
Task results¶
Each task sums up its runs: Passed all runs, Failed, Mixed results (some passed, some failed), Partially verified (some passed, none failed, some unconfirmed), Inconclusive or Self-reported.
Comparison results¶
- Pass rate per side: passed runs out of runs that ran.
- Speed-up: reported only when both sides passed the same check on the same task and trial at least 3 times. Times are medians of those matched passing runs, excluding setup and verification. Otherwise the result is Not enough data.
- Result confirmed means the speed-up meets that rule.
Limits per run¶
| Limit | |
|---|---|
| AI browser test or comparison run | 20 steps, 4 minutes |
| Explore agent | 12 steps, 2 minutes; up to 20 runs and 5 minutes per explore |
| Session setup (signing in) | 2.5 minutes; app sign-in 2 minutes, with the stuck step named |
| Tasks per test | 3 |
| Runs per task | 1, 3 or 5 (comparisons: 3–10 trials per side) |
| AI models per comparison | 3 |