Displaying Test Screenshots in GitHub Actions

"It works on my machine." We have often used this sentence, particularly when working on integration tests. How can you debug an end-to-end (e2e) test that fails on the CI but works on your machine?

E2E test runners like Cypress and Playwright can take a screenshot in case of failure. So the challenge boils down to grabbing this screenshot in the workflow, and exposing it to developers to understand why the test failed. Thankfully, there is a solution: build artifacts. Let's explore how to accomplish this.

Cypress with Github Actions

To learn how to install Cypress, please visit the Cypress documentation. For the purpose of this article, I am basing my tests on Cypress examples:

describe('example todo application', () => {
    beforeEach(() => {
        cy.visit('<https://example.cypress.io/todo>');
    });

    it('displays two default items in the todo list', () => {
        cy.get('.todo-list li').should('have.length', 2);
        cy.get('.todo-list li')
            .first()
            .should('have.text', 'Pay electric bill');
        cy.get('.todo-list li')
            .last()
            .should('have.text', 'Walk the dog');
        cy.get('h1').should('have.text', 'TEST FAILED'); // THIS LINE WILL INTENTIONALLY FAIL THE TEST
    });
});

Once everything is set up, you can create a GitHub action associated with your project. In the .github/workflows/main.yml file, you can use the following code:

## Adapt this file to your current configuration
name: Cypress Tests
on: push

jobs:
    cypress-run:
        runs-on: ubuntu-22.04
        steps:
            - name: Checkout
              uses: actions/checkout@v4
            - name: Cypress run
              uses: cypress-io/github-action@v6

Now, you just need to push your changes to GitHub. The action will be triggered and you will be able to see the result in the Actions tab of your repository.

As you can see, the test failed.

However, the tests passed on my machine. How can I check what's different in the CI?

Displaying Screenshots on GitHub Actions

By default, Cypress automatically takes a screenshot in case of failure during the cypress run execution. Screenshots are stored in the cypress/screenshots folder, which you can modify in the configuration.

The following action step downloads the cypress/screenshots folder in case of an error, and creates a GitHub artifact with it. An artifact is a file or a collection of files generated during the execution of a workflow that we can share between different jobs.

- name: Upload screenshots
  uses: actions/upload-artifact@v3
  if: failure()
  with:
    name: cypress-screenshots
    path: cypress/screenshots

In case of failure, the link to the artifact appears on GitHub:

If you download the file, you have a screenshot of the interface at the time of your error:

This trick helps to have a better visualization of what is happening during a failed test.

And Playwright ?

If you are using Playwright, you can use the same solution to display screenshots. To do this, you need to modify your configuration:

playwright.config.js

module.exports = defineConfig({
	...,
  use: {
    ...,
    screenshot: 'only-on-failure', // Take screenshot on failure
  },
  outputDir: './playwright/output', // Screnshoot folder, can be updated.

Then, create an github-action like this:

jobs:
    test:
        timeout-minutes: 60
        runs-on: ubuntu-latest
        steps:
            - uses: actions/checkout@v3
            - uses: actions/setup-node@v3
              with:
                  node-version: 18
            - name: Install dependencies
              run: npm ci
            - name: Install Playwright Browsers
              run: npx playwright install --with-deps
            - name: Run Playwright tests
              run: npx playwright test
            - uses: actions/upload-artifact@v3
              if: always()
              with:
                  name: playwright-report
                  path: playwright-report/
                  retention-days: 30
            - uses: actions/upload-artifact@v3
              if: failure()      
              with:
                  name: playwright-output
                  path: playwright/output ## Folder used in playwright.config.js `outputDir`
                  retention-days: 30

Playwright also offers an automatic report that you can download!

Cypress Cloud

Having a screenshot is great, but it can be difficult to understand what is happening. To solve this problem, Cypress offers a cloud solution that allows you to see the execution of your tests in real time. You can also download the video of the execution. It's called Cypress Cloud. And the configuration is very simple:

Create an account on https://www.cypress.io/ and create a project. Add the project id inside the cypress.config.js

module.exports = {
  projectId: "t1odjw",
  // ...rest of the Cypress project config
}

Then, you will have access to a token that you can add to your GitHub secrets. Then, you can modify your GitHub action like this:

name: Cypress Tests
on: [push]
jobs:
  cypress-run:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v3
      - name: Cypress run
        uses: cypress-io/github-action@v6
        with:
          record: true # Trigger the record for Cypress Cloud
          parallel: true
        env:
          CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

Now each time your action is triggered, you will be able to see the execution of your tests in real time into your Cypress Cloud dashboard.

Be careful, the Cypress Cloud is not free. You can find the pricing here.

Conclusion

Now you know how to display screenshots of your Cypress tests on GitHub Actions. This will allow you to better understand the errors that occur during your tests. You could say to your colleagues: "It works on my machine and on the CI" !

You can find the code of this article on GitHub

Nota Bene: This article talks about GitHub actions, but you can use the same solution with other CI like Gitlab CI. It has also an artifact system and an easy integration for GitLab.