An assurance suite replays a maintained set of checks against a live deploy and produces a pinned, dated report. This page covers running one and reading what comes back. For what the two suites are and how they differ, see Assurance suites.
Open the suite
From your project, select Assure, then Assurance. (Assure opens on Governance first — switch to the Assurance leaf.)
A pill toggle at the top of the page switches between the two suites — Regression and Security Regression. (For what each one does and how they differ, see Assurance suites.)
Pick the suite you want before you run it. The heading, the run list, and the results below all follow the suite you’ve selected.
Run the suite
Select Run at the top-right. A dialog opens — its title reads Run Regression Suite or Run Security Regression, depending on the suite.
Fill in three things:
- Environment (pipeline stage). Choose the stage whose deploy you want to test. The suite runs against that stage’s live deploy target(s), and the report is pinned to the exact version that was live at run time. A stage with nothing to test against shows (no targets) beside its name.
- Targets to run. Tick the deploy targets for the chosen stage. Each row shows the target’s name, a check-type badge, its URL, and an auth tag when the target signs in with credentials. If the stage has no targets, add them under Settings → Pipeline, or pick another stage.
- Run notes (optional, non-secret). A free-text note for this run — for example, “focus on the checkout flow.” Keep it free of secrets.
Run the security suite only against a deploy you’re authorized to test, never against production data. The same caution applies to the regression suite: point it at a safe environment with a clean, dedicated test user.
Select Run Suite to start. The button reads Starting… while the run spins up, and stays disabled until you’ve picked a stage and at least one target. Select Cancel to back out.
Read the run list
Each run appears as a row in the list, newest first. A row shows:
- A status pill —
queued,running,completed, orfailed. The list updates on its own, so the pill advances fromqueuedtorunningtocompletedwithout a refresh. - The short deploy code for the version that was tested, once the run pins one.
- The date it was triggered.
- A delta summary — a short, dotted line of what changed versus the previous run (for example, “2 newly broken · 1 fixed”), or “no change” if nothing did.
- A suspect PR(s) count when the run flags likely culprits for a break.
When a run has completed, the row also shows an Evidence link. It opens the per-deploy evidence package that fuses this run’s results with the others for the same deploy — see Delivery Evidence reports.
Select a row to open the run’s full results.
Read the results
The run detail page opens with a Back to Assurance link, the run’s status pill, its title (Regression Run or Security Regression Run), and a small grid of facts: Triggered, Completed, Deploy commit (the version tested), and Targets (how many ran). Any run notes appear just below.
Delta (vs previous run)
The Delta (vs previous run) card is the headline: what changed since this suite last ran. The first run has no baseline, so it shows First run — no baseline; later runs name the baseline they compared against.
The labels differ by suite:
- Regression — Newly broken (regressions), Reopened, Newly passing (fixed), Still broken, Flaky / quarantined.
- Security Regression — New findings, Reopened (security regression), Resolved, Still open, Flaky / quarantined.
Each label shows a count and the affected case IDs. If nothing moved, the card reads No change vs the previous run. While a run is still going, it reads Delta not available yet (run in progress).
When a run newly broke something, a Send to Requests button appears in this card. Select it to open a Requests thread seeded from the run’s regressions, so you can act on the breaks without retyping them. The button reads Sending… while it works.
Suspect changes (blame window)
When a scenario newly broke, the Suspect changes (blame window) card lists the commits and pull requests merged between the last passing run and this one — the likely culprits to look at first. Each entry shows a short commit code, its pull-request number when there is one, the commit subject, and who made it. Treat these as suspects, not a verdict — a starting point to investigate, not an assigned cause. The card appears only when there are suspects.
Coverage gaps
The Coverage gaps card lists surfaces found in your app that no scenario checks yet. It’s an honest, measured gap list — not a claim that everything else is fully tested. The card appears only when gaps exist.
Case results
The Case results card lists every check the suite replayed, with its outcome: passed, failed, flaky / quarantined, or skipped. Each row shows the case’s name, the surface it covers, its check type, and — when relevant — how many tries it took and its pass rate.
A failing or flaky case expands to drift diagnostics that help you fix it:
- Error — the observed failure.
- Root cause — the likely reason it broke.
- Files — the code it points to, as file references with notes.
- Suggestions — proposed fixes.
The disclaimer
A line at the foot of every run reads: This report is an audit-evidence input; it supports — but does not replace — independent sign-off. The suite gives you evidence to review; it doesn’t sign off on a release for you.
Where to go next
- Assurance suites — what the two suites are and when to use each.
- Delivery Evidence reports — the fused, per-deploy package the Evidence link opens.
- Pipeline stages and gates — the stages you choose an environment from.