Workflow

This page documents the end-to-end process from receiving an experiment ticket to having automated tests running.

Overview

Ticket → Build Experiment → Scaffold Tests → Configure → Write → Run → Review → Commit

Step 1: Receive the Experiment Ticket

Review the experiment brief for:

• Experiment name - the identifier for the A/B test
• Target markets - which countries the experiment applies to
• Page paths - which pages the experiment modifies (e.g., product detail page, homepage)
• Variants - what the control looks like vs. what the experiment changes
• Success criteria - what metrics are being tracked (clicks, conversions, impressions)
• QA environment - where to test (staging URL, preview tokens)

Step 2: Build the Experiment

Develop the experiment in your src/ directory using the experiment framework:

1. Create components in src/components/
2. Define market-specific content in src/js/config.js
3. Set up the entry point with runScript() and waitFor()
4. Build with yarn build to produce dist/v1-index.jsx

Step 3: Scaffold Tests

Run the generator from your project root:

npx experiment-e2e-generator

Answer the prompts:

1. Experiment name - accept the auto-detected name or type your own
2. Base URL - enter the QA environment URL
3. Market - select the market group or individual markets from your ticket
4. ESLint - choose whether to exclude tests from linting

The generator creates ~8 files and installs dependencies.

Step 4: Configure QA Links

Open tests/config/qa-links.config.js and replace the default URL patterns with your actual QA URLs.

Option A: Edit the config directly

getUrls(marketCode) {
  const market = this.markets.find((m) => m.code === marketCode);
  return {
    controlUrl: `https://qa.example.com/${market.urlPath}/page/?variant=control`,
    experimentUrl: `https://qa.example.com/${market.urlPath}/page/?variant=experiment`,
  };
}

Option B: Use environment variables

export CONTROL_URL_NL="https://qa.example.com/nl/page/?at_preview_token=ctrl123"
export EXPERIMENT_URL_NL="https://qa.example.com/nl/page/?at_preview_token=exp456"

See Configuration for full details.

Step 5: Write Test Assertions

Open the generated spec files and replace the TODO placeholders with your experiment-specific assertions.

Control Group

Verify the experiment component is not visible in the control:

test('should not display experiment component', async ({ page }) => {
  const component = page.locator('[data-experiment="my-experiment"]');
  await expect(component).not.toBeVisible();
});

Experiment Group

Verify the experiment component appears and works correctly:

test('should display promotional banner', async ({ page }) => {
  const banner = page.locator('[data-experiment="my-experiment"]');
  await expect(banner).toBeVisible();
  await expect(banner).toContainText('Special Offer');
});

See Writing Tests for a full tutorial.

Step 6: Run Tests Locally

Start with the quick smoke test:

yarn test:e2e:experiment

This runs the bundle injection test (experiment-test.spec.js), which loads your compiled bundle into a blank page. If this fails, your experiment has a build or rendering issue.

Then run the full suite:

yarn test:e2e

This runs all specs including live URL tests against the QA environment.

Step 7: Review the HTML Report

After tests complete, open the Playwright report:

npx playwright show-report

Check for:

• Passed tests - green checks, experiment works as expected
• Failed tests - red marks, investigate the failure screenshots and traces
• Skipped tests - yellow marks, tests that were conditionally skipped

Step 8: Iterate on Failures

Common failure patterns and fixes:

Failure	Likely Cause	Fix
Element not visible	Wrong selector or experiment not rendering	Check the selector matches your experiment's DOM
Timeout waiting for element	Page load too slow or element doesn't exist	Increase timeout or add waitForLoadState
URL 404	Wrong QA link	Update qa-links.config.js with correct URLs
Text mismatch	Different copy than expected	Check config.js for the correct market content
Screenshot shows blank page	Bundle not injected or JS error	Check browser console for errors

For each failure:

1. Read the error message and screenshot in the HTML report
2. Run the failing test in debug mode: npx playwright test --debug --grep "test name"
3. Fix the issue (usually a selector or URL update)
4. Re-run the single test to verify
5. Run the full suite again

Step 9: Commit and Push

Once all tests pass:

git add tests/ playwright.config.js
git commit -m "feat: add E2E tests for my-experiment"
git push

Quick Reference

Step	Command	Time
Scaffold	npx experiment-e2e-generator	~2 min
Configure URLs	Edit qa-links.config.js	~5 min
Write assertions	Edit .spec.js files	~15-30 min
Run smoke test	yarn test:e2e:experiment	~1 min
Run full suite	yarn test:e2e	~3-5 min
View report	npx playwright show-report	~1 min

What's Next

• Getting Started - setup and installation
• Writing Tests - detailed test writing tutorial
• Running Tests - CLI flags, reporters, debugging
• Best Practices - tips for reliable tests