VisaSpark
Sign in
advanced21 min

Structured Diagnostics and Reporting

Designing what a framework captures on failure -- logs, screenshots, traces -- as structured, queryable diagnostic data, not just raw text, and what a genuinely useful test report communicates beyond pass/fail counts.

What you'll learn

  • Design a structured failure record that captures enough context to begin diagnosing a failure without re-running it
  • Explain why capturing diagnostics only on failure (not on every run) is the practical default for CI performance
  • Explain what a genuinely useful test report communicates beyond a simple pass/fail count

Prerequisites

Explanation

No real trace, screenshot, or CI report is generated by this lesson's exercises -- they model structured-diagnostic-record design as data, using genuine JavaScript/TypeScript execution.

Capturing "some logs" on a test failure is a start, but a genuinely useful framework designs its failure output as structured, queryable data, not a wall of unstructured text — a failure record with clearly separated fields (test name, tags, the specific assertion that failed, the environment/browser it ran against, a timestamp, and a link or path to the trace/screenshot/video captured) can be filtered, searched, and aggregated across many CI runs — "show me every failure of this specific assertion across the last 20 runs" is a query a structured record supports, and unstructured console output effectively does not.

The practical default, matching what Playwright itself does, is to capture the expensive diagnostics — traces, screenshots, video — only on failure (or on the first retry), not on every single successful run: capturing a full trace for every passing test would meaningfully slow down CI and consume storage for artifacts that are, by definition, never actually needed for a test that passed. This is a deliberate performance/completeness tradeoff, not an oversight — the cases where those artifacts matter are overwhelmingly the failing ones.

A genuinely useful test report communicates meaningfully more than a bare pass/fail count. At minimum: which SPECIFIC tests failed (not just "12 failed"), the failure trend across recent runs (is this test newly broken, or has it been reliably failing for a week?), retry counts per test (a test that "passed" only after 2 retries is a very different signal from one that passed immediately, even though both show as green), and direct links to each failure's captured diagnostics, so investigating a failure doesn't require re-running the suite locally just to reproduce and capture the same information the CI run already had.

Example

Modeling a structured failure record's shape and the failure-only diagnostic-capture decision, as data.

function buildFailureRecord(test, error, tracePath) {
  return {
    testName: test.name,
    tags: test.tags,
    failedAssertion: error.assertionDescription,
    environment: test.environment,
    timestamp: test.timestamp,
    tracePath, // a link/path to the captured artifact, not the raw trace data itself
  };
}
const record = buildFailureRecord(
  { name: "checkout completes", tags: ["smoke", "checkout"], environment: "chromium", timestamp: "2026-08-03T10:00:00Z" },
  { assertionDescription: "expected order total to be $42.00" },
  "traces/checkout-completes-run482.zip"
);
console.log(record.failedAssertion); // "expected order total to be $42.00" -- immediately queryable/searchable, not buried in prose

function shouldCaptureExpensiveDiagnostics(testOutcome) {
  return testOutcome === "failed" || testOutcome === "flaky-retry";
}
console.log(shouldCaptureExpensiveDiagnostics("passed")); // false -- capturing a trace here would be pure, unneeded overhead
console.log(shouldCaptureExpensiveDiagnostics("failed"));  // true -- exactly where the diagnostic artifact actually matters

Try it yourself

Call shouldCaptureExpensiveDiagnostics with 'flaky-retry', and confirm a retried-then-passed test is still correctly treated as worth capturing diagnostics for.

Code editor. Press Escape then Tab to leave the editor if keyboard focus becomes trapped. Press Control+Shift+M inside the editor to toggle Tab-key focus trapping.

Loading editor…

Guided exercise

Guided exercise

This models building a structured failure record only -- no real trace or screenshot is captured. Write buildFailureRecord(testName, tags, failedAssertion, tracePath), returning an object with exactly those four fields, named testName, tags, failedAssertion, tracePath.

Checks: correctly preserves the test name · correctly preserves the full tags array · correctly preserves the trace path

Code editor. Press Escape then Tab to leave the editor if keyboard focus becomes trapped. Press Control+Shift+M inside the editor to toggle Tab-key focus trapping.

Loading editor…

Stuck? Get a hint.

Independent exercise

Independent exercise

This models a report's per-test signal beyond bare pass/fail only -- no real report is generated. Write reportSignal(outcome, retryCount): if outcome is 'failed', return 'needs-investigation'. Else if retryCount > 0, return 'passed-but-flaky'. Else return 'genuinely-stable-pass'.

Checks: correctly flags a failure for investigation · correctly distinguishes a retried pass from a genuinely stable one · correctly identifies a genuinely stable, immediate pass

Code editor. Press Escape then Tab to leave the editor if keyboard focus becomes trapped. Press Control+Shift+M inside the editor to toggle Tab-key focus trapping.

Loading editor…

Stuck? Get a hint.

Common mistakes

  • Capturing failure information as unstructured console text instead of a structured record with clearly separated fields -- this makes it effectively impossible to query, filter, or aggregate failures across many CI runs.
  • Capturing full traces, screenshots, and video for every single test run, including passing ones -- this meaningfully slows down CI and wastes storage on artifacts that are never actually needed for a passing test.
  • Reporting only a bare pass/fail count without retry counts or failure trends -- this hides the meaningful difference between a genuinely stable pass and one that only succeeded after retries, and between a newly broken test and a chronically flaky one.

Knowledge check

Knowledge check

1. Why is a structured failure record (with clearly separated fields) preferable to a wall of unstructured console text?
2. Why does a framework typically capture expensive diagnostics (traces, screenshots, video) only on failure, not on every run?
3. Why does a per-test retry count matter in a report, beyond the final pass/fail result?

Takeaway

Design failure output as structured, queryable data with clearly separated fields, not unstructured text. Capture expensive diagnostics (traces, screenshots, video) only on failure, as a deliberate performance tradeoff. Report retry counts and failure trends, not just a bare pass/fail count -- a retried pass is a meaningfully different signal from an immediate one.

Summary

A structured failure record (test name, tags, failed assertion, environment, timestamp, trace path) supports filtering and aggregation across CI runs in a way unstructured console text does not. Capturing expensive diagnostics only on failure is a deliberate, practical performance tradeoff, since those artifacts are needed almost exclusively for failing tests. A genuinely useful report surfaces retry counts and failure trends, not just a bare pass/fail count, since a retried pass is a meaningfully different reliability signal from an immediate one.

References

Your notes

Notes save automatically.

Finished this lesson?

Mark it complete to track your progress and schedule a future review.

Next: CI Pipelines, Quality Gates, and Sharding →