Experiment E2E Generator

Appearance

Writing Tests ​

This tutorial walks you through writing your first E2E test for an A/B experiment, starting from the generated scaffolding.

Understanding the Generated Structure ​

After running the generator, you have two test spec files:

  • <experiment-name>.spec.js - Tests that navigate to live QA URLs and verify the control/experiment variants
  • experiment-test.spec.js - A bundle injection test that loads your built experiment into a blank page

Both files contain TODO comments where you add your experiment-specific assertions.

Before writing tests, set up the URLs for your experiment variants. Open tests/config/qa-links.config.js:

js
// Option A: Use environment variables (recommended for CI)
// CONTROL_URL_NL=https://www.example.com/nl/?at_preview_token=abc
// EXPERIMENT_URL_NL=https://www.example.com/nl/?at_preview_token=xyz

// Option B: Edit the default URL pattern in qa-links.config.js
getUrls(marketCode) {
  const market = this.markets.find((m) => m.code === marketCode);
  return {
    controlUrl: `${this.baseUrl}/${market.urlPath}/your-page/?variant=control`,
    experimentUrl: `${this.baseUrl}/${market.urlPath}/your-page/?variant=experiment`,
  };
}

Step 2: Write a Control Group Test ​

The control group test verifies that the experiment component is not visible when the user is in the control group. Open <experiment-name>.spec.js and update the control describe block:

js
test.describe('My Experiment - Control', () => {
  test.beforeEach(async ({ page }) => {
    validateUrls();
    const { controlUrl } = qaLinksConfig.getUrls(qaLinksConfig.markets[0].code);
    await page.goto(controlUrl);
  });

  test('should not display experiment component', async ({ page }) => {
    // Use a selector that matches your experiment's root element
    const experimentBanner = page.locator('[data-experiment="my-experiment"]');

    // In the control group, the experiment should NOT be visible
    await expect(experimentBanner).not.toBeVisible();
  });

  test('should display baseline content', async ({ page }) => {
    // Verify the original page content is intact
    const heading = page.getByRole('heading', { name: 'Welcome' });
    await expect(heading).toBeVisible();
  });
});

Step 3: Write an Experiment Group Test ​

The experiment group test verifies that the experiment component appears and functions correctly:

js
test.describe('My Experiment - Experiment', () => {
  test.beforeEach(async ({ page }) => {
    validateUrls();
    const { experimentUrl } = qaLinksConfig.getUrls(qaLinksConfig.markets[0].code);
    await page.goto(experimentUrl);
  });

  test('should display experiment component', async ({ page }) => {
    const experimentBanner = page.locator('[data-experiment="my-experiment"]');
    await expect(experimentBanner).toBeVisible();
  });

  test('should show correct promotional text', async ({ page }) => {
    const promoText = page.getByText('Special Offer: 20% Off');
    await expect(promoText).toBeVisible();
  });

  test('should have working CTA button', async ({ page }) => {
    const ctaButton = page.getByRole('button', { name: 'Shop Now' });
    await expect(ctaButton).toBeVisible();

    await ctaButton.click();

    // Verify navigation or UI change after click
    await expect(page).toHaveURL(/\/shop\//);
  });
});

Step 4: Add Multi-Market Tests ​

To test across all configured markets, loop over the markets array:

js
import { experimentConfig, qaLinksConfig, validateUrls } from '../../config/index.js';

for (const market of experimentConfig.markets) {
  test.describe(`My Experiment - ${market.name}`, () => {
    test.beforeEach(async ({ page }) => {
      const { experimentUrl } = qaLinksConfig.getUrls(market.code);
      await page.goto(experimentUrl);
    });

    test(`should display experiment in ${market.code}`, async ({ page }) => {
      const banner = page.locator('[data-experiment="my-experiment"]');
      await expect(banner).toBeVisible();
    });
  });
}

Step 5: Use Custom Fixtures ​

The generator creates custom fixtures in tests/fixtures/test-fixtures.js. Use them for shared setup:

js
import { test, expect } from '../fixtures/test-fixtures.js';

test('should use experiment context', async ({ page, experimentContext }) => {
  // Access experiment config through the fixture
  console.log(experimentContext.name);       // experiment name
  console.log(experimentContext.marketGroup); // market group code

  // Build a URL with parameters
  const url = experimentContext.buildUrl('/promo-page/', {
    variant: 'experiment',
    debug: 'true',
  });
  await page.goto(url);
});

Step 6: Add Custom Assertions ​

Common assertion patterns for experiments:

Text Content ​

js
// Exact text match
await expect(page.getByText('Free Shipping')).toBeVisible();

// Partial text match
await expect(page.locator('.promo-banner')).toContainText('limited time');

// Text from config (recommended for multi-market)
await expect(page.locator('.headline')).toHaveText(expectedConfig.headline);

Styling ​

js
// Check background color
await expect(page.locator('.cta-button')).toHaveCSS(
  'background-color',
  'rgb(0, 119, 200)'
);

// Check visibility via CSS
await expect(page.locator('.experiment-overlay')).toHaveCSS('display', 'block');

Interactions ​

js
// Click and verify
await page.getByRole('button', { name: 'Apply Code' }).click();
await expect(page.getByText('Code applied')).toBeVisible();

// Form input
await page.getByLabel('Promo Code').fill('SAVE20');
await page.getByRole('button', { name: 'Apply' }).click();
await expect(page.locator('.discount')).toContainText('20%');

Screenshots ​

js
// Full page screenshot
await page.screenshot({ path: 'screenshots/experiment-visible.png', fullPage: true });

// Element screenshot
await page.locator('.experiment-banner').screenshot({
  path: 'screenshots/banner-close-up.png',
});

What's Next ​