Running Tests

This page covers how to run your generated tests, use reporters, debug failures, and integrate with CI.

Test Commands

The generator adds two scripts to your package.json:

# Run all E2E tests (all spec files in tests/)
yarn test:e2e

# Run only the bundle injection test (quick smoke test)
yarn test:e2e:experiment

Under the hood, these run npx playwright test with the appropriate configuration.

Playwright CLI Flags

You can pass additional flags after the script command:

Run in Headed Mode

Watch the browser as tests execute:

npx playwright test --headed

Run a Specific Test File

npx playwright test tests/e2e/my-experiment/my-experiment.spec.js

Run Tests Matching a Pattern

npx playwright test --grep "should display experiment"

Run in a Specific Browser

# Only Chromium
npx playwright test --project=chromium

# Only Firefox
npx playwright test --project=firefox

# Only WebKit (Safari)
npx playwright test --project=webkit

Run a Single Test

Use test.only() in your spec file:

test.only('should display experiment banner', async ({ page }) => {
  // Only this test will run
});

Skip a Test

test.skip('should handle edge case', async ({ page }) => {
  // This test will be skipped
});

// Conditional skip
test('should display mobile variant', async ({ page }) => {
  test.skip(process.env.CI === 'true', 'Skip on CI - requires mobile emulation');
  // ...
});

Reporters

The generated playwright.config.js uses the HTML reporter by default. You can change or add reporters.

HTML Reporter (Default)

Generates an interactive HTML report after each run:

# View the report
npx playwright show-report

List Reporter

Print results to the terminal:

// playwright.config.js
export default defineConfig({
  reporter: 'list',
});

Multiple Reporters

Use multiple reporters simultaneously:

// playwright.config.js
export default defineConfig({
  reporter: [
    ['html', { open: 'never' }],
    ['json', { outputFile: 'test-results/results.json' }],
    ['junit', { outputFile: 'test-results/results.xml' }],
  ],
});

JUnit Reporter (CI)

Useful for CI pipelines that parse JUnit XML:

reporter: [['junit', { outputFile: 'test-results/junit.xml' }]],

Screenshots and Traces

The generated config includes:

use: {
  trace: 'on-first-retry',  // Record trace on first retry
  screenshot: 'on',          // Take screenshot after each test
  video: 'off',              // No video by default
}

Screenshot Options

Value	Behavior
'on'	Screenshot after every test
'off'	No screenshots
'only-on-failure'	Screenshot only when a test fails

Trace Options

Value	Behavior
'on'	Record trace for every test
'off'	No traces
'on-first-retry'	Record trace on first retry (default)
'retain-on-failure'	Keep trace only for failed tests

Viewing Traces

npx playwright show-trace test-results/trace.zip

This opens the Playwright Trace Viewer, showing a timeline of actions, network requests, DOM snapshots, and console logs.

Enabling Video

use: {
  video: 'on-first-retry',  // or 'on' for every test
}

Debugging

Playwright Inspector

Launch the interactive debugger:

npx playwright test --debug

This opens a browser with the Playwright Inspector panel. You can step through each action, inspect selectors, and see the page state.

Debug a Specific Test

npx playwright test --debug --grep "should display experiment"

Pause in Code

Add page.pause() to stop execution at a specific point:

test('debug this test', async ({ page }) => {
  await page.goto(experimentUrl);
  await page.pause(); // Opens the inspector here
  await expect(page.locator('.banner')).toBeVisible();
});

Verbose Logging

DEBUG=pw:api npx playwright test

Retries

Configure retries for flaky tests:

// playwright.config.js
export default defineConfig({
  retries: 2, // Retry failed tests up to 2 times
});

Or via command line:

npx playwright test --retries=2

Parallel Execution

The generated config sets fullyParallel: true, meaning all tests run in parallel. To run sequentially:

// playwright.config.js
export default defineConfig({
  fullyParallel: false,
  workers: 1, // Single worker
});

Or via command line:

npx playwright test --workers=1

CI Integration

Basic CI Script

# Install Playwright browsers
npx playwright install --with-deps

# Build the experiment first
yarn build

# Run tests
yarn test:e2e

Artifacts

Upload these directories for debugging failed CI tests:

• playwright-report/ - HTML report
• test-results/ - Screenshots, traces, videos
• screenshots/ - Custom screenshots from test helpers

What's Next

• Writing Tests - tutorial on writing tests
• Test Patterns - common testing patterns
• Best Practices - stability and reliability tips