Reports
Analyze test run results, identify failure patterns, track long-term trends, and export detailed reports for stakeholders.
What are AutoRunner Reports?
AutoRunner Reports provide detailed analysis of your test execution data. Generate reports at the run, suite, or project level to understand pass rates, failure patterns, performance trends, and overall test health. Export reports in multiple formats for sharing with stakeholders and compliance documentation.
Run Reports
Each completed test run generates a detailed report accessible from the run detail page.
Run Summary
| Metric | Description |
|---|---|
| Pass Rate | Percentage of scenarios that passed |
| Total Scenarios | Number of scenarios executed |
| Passed | Count of successful scenarios |
| Failed | Count of failed scenarios |
| Skipped | Count of skipped scenarios |
| Duration | Total execution time |
| Driver | Browser or device driver used |
| Environment | Target environment for the run |
Scenario Breakdown
A table listing every scenario with its individual result:
| Column | Description |
|---|---|
| Scenario Name | Name of the Gherkin scenario |
| Status | Passed, Failed, or Skipped |
| Duration | Execution time for the scenario |
| Steps | Pass/fail count per step |
| Error | Error message (for failed scenarios) |
Click any scenario row to expand step-by-step details, screenshots, and logs.
Suite Reports
Suite-level reports aggregate results across multiple runs for a single test suite.
Suite Health Metrics
| Metric | Description |
|---|---|
| Average Pass Rate | Mean pass rate across all runs in the selected period |
| Run Count | Total number of runs executed |
| Failure Rate | Percentage of runs with at least one failure |
| Avg. Duration | Average execution time per run |
| Flaky Scenarios | Scenarios that intermittently pass and fail |
Pass Rate Over Time
A line chart showing the suite's pass rate trend across runs. Use this to identify:
- Gradual quality improvements or degradation
- Sudden drops caused by code changes
- Stabilization after fixes
Trend Analysis
Trend analysis provides project-wide and cross-project insights over time.
Available Trend Views
| View | Description |
|---|---|
| Pass Rate Trend | Pass rate progression across days, weeks, or months |
| Run Volume Trend | Number of runs executed per period |
| Duration Trend | Average run duration changes over time |
| Failure Density | Number of unique failures per period |
Filtering Trends
Filter trend data by:
- Time Range — Last 7 days, 30 days, 90 days, or custom range
- Project — Single project or all projects
- Suite — Specific suite or all suites
- Framework — Filter by test framework
- Driver — Filter by browser or device
Failure Analysis
The failure analysis section helps you identify and resolve recurring test failures.
Top Failures
A ranked list of the most frequently failing scenarios across runs:
| Column | Description |
|---|---|
| Scenario | Scenario name and feature file |
| Failure Count | Number of times the scenario failed |
| Last Failed | Date of the most recent failure |
| Error Pattern | Most common error message |
| Suite | Associated test suite |
Flaky Test Detection
AutoRunner identifies flaky tests — scenarios that alternate between pass and fail across consecutive runs:
- Flakiness Score — Percentage of inconsistent results (higher = flakier)
- Run History — Visual pass/fail pattern across recent runs
- Recommendation — Suggested action (stabilize, quarantine, or investigate)
Flaky tests erode confidence in your test suite. Address scenarios with a flakiness score above 20% by reviewing test isolation, timing dependencies, and external service mocks.
Analytics Tab
The Analytics Tab provides three report views for comprehensive test analysis. Access it from the Analytics tab in the project or suite detail page.
Execution Report
The default view displays detailed results for a selected run or set of runs:
- Pass/fail/skip breakdown by scenario and step
- Execution timeline with duration per scenario
- Error clustering to group related failures
- Screenshot gallery for visual verification
Trend Analysis
Track metrics over time to identify patterns and regressions:
- Pass Rate Trend — Line chart showing pass rate progression across days, weeks, or months
- Run Volume Trend — Bar chart of runs executed per period
- Duration Trend — Average run duration changes over time
- Failure Density — Number of unique failures per period
Filter trend data by time range, project, suite, framework, or driver.
Comparison
Compare multiple runs side-by-side to identify what changed between executions:
- Navigate to the Analytics tab and select the Comparison view.
- Select two or more runs to compare (from the same suite or across suites).
- The comparison view displays:
| Column | Description |
|---|---|
| Scenario | Scenario name (shared across compared runs) |
| Run A Result | Status and duration for the first run |
| Run B Result | Status and duration for the second run |
| Delta | Change in status or duration between runs |
| Diff | Highlights scenarios that changed from pass to fail or vice versa |
Use the Comparison view to validate fixes (compare a failing run with the re-run after a fix), assess environment differences (compare the same suite across Development and Staging), or evaluate the impact of code changes on test stability.
Scheduled Reports
Configure automated report generation and delivery on a recurring schedule:
- Navigate to Reports > Scheduled Reports.
- Click Create Schedule.
- Configure the schedule:
| Setting | Description |
|---|---|
| Report Type | Execution Report, Trend Analysis, or Comparison |
| Scope | Project, suite, or cross-project |
| Frequency | Daily, weekly, or monthly |
| Format | PDF or CSV |
| Recipients | Email addresses for report delivery |
Scheduled reports are generated automatically and delivered to the configured recipients via email. Each report includes the data for the selected time window based on the frequency.
Export Options
Export reports in multiple formats:
| Format | Description | Use Case |
|---|---|---|
| Formatted report with charts, tables, and branding | Stakeholder sharing, compliance, print | |
| CSV | Raw data in spreadsheet format | Data analysis, custom reporting |
| JSON | Structured data for programmatic use | Integration with other tools |
Exporting a Report
- Navigate to the report you want to export (run, suite, or trend).
- Click the Export button in the toolbar.
- Select the desired format.
- The file downloads to your browser.
PDF Export Details
PDF reports include:
- Cover page with project name, date range, and RabbitQA branding
- Executive summary with key metrics (pass rate, run count, failure rate)
- Charts and visualizations rendered as high-resolution images
- Scenario breakdown tables with pass/fail status and duration
- Failure details with error messages and screenshots (if captured)
- Page numbering and table of contents for multi-page reports
Use Scheduled Reports to automatically generate and deliver PDF or CSV reports to your team on a recurring basis.
Metrics Breakdown
Detailed performance metrics available in reports:
| Metric | Level | Description |
|---|---|---|
| Pass Rate | Run, Suite, Project | Percentage of passed scenarios |
| Execution Duration | Run, Scenario, Step | Time taken at each level |
| Failure Rate | Suite, Project | Percentage of failing runs |
| Flakiness Score | Scenario | Inconsistency measurement |
| Step Duration | Step | Individual step timing |
| Retry Count | Scenario | Number of retry attempts |
| Screenshot Count | Run | Total captured screenshots |
Next Steps
- Dashboard — View high-level metrics and KPI summaries
- Run Details — Deep-dive into individual run results
- Test Suites — Configure and manage test suites
- Test Runner — Execute tests and monitor active runs
- Test Runners — Register and manage test execution engines
- AI Studio — Generate test scenarios with AI assistance
- Scheduling — Set up recurring runs for consistent trend data
