Getting Started

What is Experiment E2E Generator?

Experiment E2E Generator is a CLI tool that scaffolds Playwright end-to-end tests for A/B experiments. Instead of manually creating test files, fixtures, and configuration, you run a single command and answer a few prompts. The generator creates ~8 files with proper structure, installs dependencies, and optionally runs the tests immediately.

Prerequisites

• Node.js >= 16.15.1
• yarn or npm (auto-detected)
• An existing experiment project with a package.json in the root

Installation

No global install required. Run the generator directly with npx:

npx experiment-e2e-generator

Or install globally if you prefer:

npm install -g experiment-e2e-generator

Interactive Setup

The CLI walks you through an interactive prompt sequence:

1. Experiment Name

? What is the experiment name?

The generator auto-detects a default from your package.json name field or the first component in src/components/. You can accept the default or type a custom name.

2. Base URL

? What is the base URL of the application?

Enter the URL where your experiment is deployed (e.g., https://www.example.com). This is used to construct market-specific test URLs.

3. Market Selection

? Select a market group:

An autocomplete menu shows all available market groups and individual markets. Type to filter. See Supported Markets for the full list.

4. ESLint Preference

? Add tests/ to .eslintignore?

If your project uses ESLint, the generator can add tests/ to .eslintignore so that Playwright test files don't trigger linting rules designed for your source code.

What Gets Generated

After completing the prompts, the generator creates 8 files:

File	Purpose
playwright.config.js	Playwright configuration (projects, reporters, timeouts)
tests/config/index.js	Barrel export for all test configuration
tests/config/experiment.config.js	Experiment name, markets, variants, timeouts
tests/config/qa-links.config.js	Market URLs, control/experiment URL resolution
tests/fixtures/test-fixtures.js	Custom Playwright fixtures (experiment context, Adobe preview)
tests/utils/test-helpers.js	Reusable helpers (wait, scroll, screenshot, retry)
tests/e2e/<name>/<name>.spec.js	Live URL tests (control + experiment variants)
tests/e2e/<name>/experiment-test.spec.js	Bundle injection smoke test

Where <name> is the kebab-case version of your experiment name.

Post-Generation Steps

After the generator finishes:

1. Install Dependencies

The generator automatically detects your package manager and runs install. If you skipped this step, run manually:

yarn install
# or
npm install

2. Configure QA Links

Open tests/config/qa-links.config.js and update the URLs for your markets. You can either:

• Set environment variables: CONTROL_URL_NL, EXPERIMENT_URL_NL, etc.
• Edit the default URL patterns in the config file directly

3. Write Test Assertions

Open the generated .spec.js files and replace the TODO comments with assertions specific to your experiment. See Writing Tests for a tutorial.

4. Run Tests

# Run only the bundle injection test (quick smoke test)
yarn test:e2e:experiment

# Run all E2E tests
yarn test:e2e

5. Review the Report

After tests run, Playwright generates an HTML report:

npx playwright show-report

Template Variables

The generator substitutes these placeholders in template files:

Variable	Description	Example
	Original experiment name	My Promo Banner
	Kebab-case version	my-promo-banner
	Application URL	https://www.example.com
	Selected market group code	SEBN
	JSON array of market objects	[{"code":"NL","urlPath":"nl","name":"Netherlands"}]

What's Next

• Supported Markets - see all available markets and market groups
• Writing Tests - tutorial on writing your first test
• Test Patterns - common A/B test patterns
• Architecture - how the generator works under the hood