Running PlayWright Tests in CircleCI
In this article, I’d like to share how we can run our PlayWright tests in CI/CD platform, CircleCI. In this attempt, I wanted to learn something that I’d never tried before and my choice landed in PlayWright and CircleCI. What attracted me the most to learning PlayWright is the fact that it allows cross-browser testing with a single API and it is much less flaky and allows faster execution time due to its auto-wait compared to Selenium which I normally use on daily basis. Meanwhile choosing CircleCI as the CI/CD was simply a genuine curiosity about other tools besides Jenkins and GitHub Action which I normally use for test pipelines.
Before we start, Playwright is a Node. js library to automate Chromium, Firefox, and WebKit with a single API. PlayWright is built to enable cross-browser web automation that is evergreen, capable, reliable, and fast. It supports multiple programming languages including TypeScript, JavaScript, Python, .NET, and Java. but this time I’ll be using JavaScript.
Setup PlayWright Test Project
- Run the following command:
npm init playwright@latestThe command above will download the browsers needed as well as create the following files
- playwright.config.ts
- package.json
- package-lock.json
- tests/example.spec.ts
- tests-examples/demo-todo-app.spec.ts
2. Once everything is fully installed, update your test file accordingly. In this example, I’ll create a simple login test within tests/login.spec.js file for the chromium and firefox project.
const { test, expect, devices } = require('@playwright/test');const config = {
reporter: [ ['junit', { outputFile: 'results.xml' }] ],
use: {
trace: 'on',
screenshot: 'on',
video: 'on'
},
projects: [
{
name: 'chromium',
use: { ...devices['Desktop Chrome'] },
},
{
name: 'firefox',
use: { ...devices['Desktop Firefox'] },
},
],
};module.exports = config;test('Login as admin', async ({ page }) => {
await page.goto('https://opensource-demo.orangehrmlive.com/');
await expect(page).toHaveTitle(/OrangeHRM/);
await page.locator('input[name="txtUsername"]').fill('Admin');
await page.locator('input[name="txtPassword"]').fill('admin123');
await page.locator('input:has-text("LOGIN")').click();
await page.waitForURL('**/index.php/dashboard');
await page.locator('id=welcome').click();
});
3. Run your test with the following command:
npx playwright test4. Test report can be displayed by running:
npx playwright show-reportExample of overall test report
Example of individual test report
Setup Test Pipeline in CircleCI
- Link your Git account with CircleCI (Reference: link)
- Once the account is linked, create a .yml file within your project repository e.g. {{REPO URL}} >.circleci > config.yml
- Setup the config.yml file and specify where you’d like to test to run on (here I’m using PlayWright provided docker image) and the job steps:
- Checkout test script from the Git repository
- Install PlayWright and the browser needed
- Specify what type of reports you’d like to generate after the test execution. Here I’m using JUnit .XML report which will provide data such as how many tests passed or failed. Further reading on different report formats such as HTML and JSON can be found here.
version: 2.1jobs:
run-test:
docker:
- image: mcr.microsoft.com/playwright:v1.23.1-focal
# Steps to the job
steps:
- checkout
- run: npm i -D @playwright/test
- run: npx playwright install
- run: npx playwright install chrome
- run:
name: Run test
command: PLAYWRIGHT_JUNIT_OUTPUT_NAME=results.xml npx playwright test --reporter=junit --project=chromium
- store_test_results:
path: results.xml# Invoke jobs via workflows
workflows:
run-test-workflow:
jobs:
- run-test
4. Run your test workflow. One way to do it is by triggering it manually from CircleCI dashboard by selecting the branch and clicking on “Trigger Pipeline”. Your test will start running. Other ways to trigger your test workflow such as through API can be found here.
5. The .XML result will allow you to see test overall test insight such as success rate, duration, total runs,
as well as passed, failed, and flaky tests over time
