Definition
- HTML reporter – Built-in Playwright reporter producing a self-contained site with on-failure screenshots, videos, and traces.
- Allure reporter – Rich organizational features: suites, severity, history, trends, and arbitrary file attachments.
- Diagnostic artifacts – Screenshots at failure, video of the sequence, trace with DOM/network/console, error message and stack.
- Environment info – Browser, version, base URL, build commit. Lets developers classify regressions vs environmental differences.
- Purpose – Diagnosis, not scoring. A developer should identify the cause from the report without re-running the test.
When the National Bank of India's overnight tests run, a bare result that just says "something failed" is almost useless. A good report is like a detailed incident file: which step broke, a screenshot at that moment, a video of the whole attempt, a step-by-step trace, the exact error, and which environment and build it ran against.
Understanding the Concepts
How do you generate the HTML report?
Configure the HTML reporter in Playwright config. After a run it produces a self-contained report.
// playwright.config.ts
import { defineConfig } from '@playwright/test';
export default defineConfig({
reporter: [
['html', { open: 'never' }], // Generate but do not auto-open
['list'], // Console output during run
],
use: {
screenshot: 'only-on-failure',
video: 'retain-on-failure',
trace: 'retain-on-failure',
},
});
# View the report after a run
npx playwright show-report
How do you generate an Allure report?
Allure is a two-step flow: write raw results during the run, then render them.
// playwright.config.ts with Allure
export default defineConfig({
reporter: [
['allure-playwright', { outputFolder: 'allure-results' }],
['html'],
],
});
# Generate the Allure report from raw results
npx allure generate allure-results --clean -o allure-report
npx allure open allure-report
What makes a report diagnostic?
Every failure should carry enough evidence for someone who was not watching the run to understand and fix it.
| Artifact | What It Shows | Why It Matters |
|---|---|---|
| Error message + stack | What assertion or action failed | Points to the code |
| Screenshot | Visual state at failure | Shows missing elements, errors |
| Video | Full sequence before failure | Reveals timing, flow issues |
| Trace | DOM, network, console at each step | Most powerful: full context |
| Environment info | Browser, URL, commit, build | Classifies regression vs env |
| Named steps | Which phase failed | Reads as a narrative |
// Named steps for readable reports
test('Transfer between accounts', async ({ page }) => {
await test.step('Navigate to transfer page', async () => {
await page.goto('/transfer');
});
await test.step('Fill transfer form', async () => {
await page.getByLabel('From Account').selectOption('ACC-001');
await page.getByLabel('To Account').fill('ACC-002');
await page.getByLabel('Amount').fill('5000');
});
await test.step('Submit and verify', async () => {
await page.getByRole('button', { name: 'Transfer' }).click();
await expect(page.getByText('Transfer Successful')).toBeVisible();
});
});
How do you include environment metadata?
Record the environment so failures can be classified.
// Custom reporter or fixture that records environment
import { test } from '@playwright/test';
test.beforeAll(async ({}, testInfo) => {
testInfo.annotations.push(
{ type: 'environment', description: process.env.BASE_URL || 'local' },
{ type: 'commit', description: process.env.GITHUB_SHA || 'unknown' },
{ type: 'browser', description: testInfo.project.name },
);
});
How do you publish reports from CI?
# .github/workflows/e2e.yml
jobs:
e2e:
steps:
- run: npx playwright test
- name: Upload HTML report
uses: actions/upload-artifact@v4
if: always()
with:
name: playwright-report
path: playwright-report/
- name: Upload Allure results
uses: actions/upload-artifact@v4
if: always()
with:
name: allure-results
path: allure-results/
Why must reports enable diagnosis, not just scoring?
A bare red/green forces re-running locally. A diagnostic report collapses that loop.
Without diagnostics:
1. Test fails in CI at 2am
2. Developer sees "FAILED" in the morning
3. Tries to reproduce locally (30 min)
4. Cannot reproduce (env difference)
5. Ignores the failure
6. Real regression slips through
With diagnostics:
1. Test fails in CI at 2am
2. Developer opens the report
3. Views screenshot: error banner visible
4. Opens trace: balance API returned 500
5. Checks environment: staging, commit abc123
6. Identifies cause in 5 minutes
7. Fixes it. No re-run needed.
How does Allure add value over the HTML reporter?
Allure provides organizational features for larger programs.
- Suites and features : Group tests by business area.
- Severity labels : Mark critical vs minor tests.
- History and trends : Track pass rate and flakiness over time.
- Retries view : See which tests needed retries (flakiness signal).
What are common interview follow-up questions?
- "How do you handle report storage for many runs?" Archive to S3 or a static hosting service. Allure has a server mode for historical dashboards.
- "How do you notify on failure?" CI sends a Slack or email notification with a link to the report.
- "What is the most useful artifact?" The Playwright trace, because it shows the DOM, network, and console at each step.
Visual Overview
Test Report as a Diagnosis Tool
────────────────────────────────
Test Run
│
├── PASSED tests: status + duration
│
└── FAILED tests: status + duration +
│
├── Error message + stack trace
├── Screenshot (visual state at failure)
├── Video (full sequence)
├── Trace (DOM + network + console)
├── Named steps (which phase failed)
└── Environment info (browser, URL, commit)
│
▼
Developer opens report
-> Identifies cause in minutes
-> No re-run needed
Key Takeaway
I generate reports with both the HTML reporter and Allure. The HTML reporter produces a self-contained site after each run, automatically embedding on-failure screenshots, videos, and traces. Allure adds suites, severity, history, trends, and arbitrary attachments for larger programs.
What makes a report useful is diagnosis, not just red/green. Every failure carries the error message and stack, a screenshot at the failure point, a video of the sequence, the Playwright trace with DOM, network, and console at each step, named steps so the report reads as a narrative, and environment metadata so failures can be classified as regressions vs environmental differences.
The reason reports must enable diagnosis is economics. A bare red/green forces re-running locally, which is slow and often impossible for CI-specific failures. A diagnostic report collapses that loop: a developer opens the report, views the trace, and identifies the cause in minutes without re-running.
At the National Bank of India an on-call engineer opened a failed transfer test's trace and immediately saw the balance API had returned an error, fixed it without ever reproducing locally.
Remember one sentence: a test report should carry enough evidence for someone who was not watching the run to diagnose the failure in minutes, not just show red or green.
---