Grouping parallel runs in one entry on Cypress.io using Gitlab CI

Question

I have an Nx project with Gitlab CI configured. My project has 2 kinds of E2E testing included - main project's and Storybook's ones. The tests themselves work fine, but I use cypress.io dashboard for E2E results gathering.

I configured the project to record them and it works, but I want them to be saved as one run, but with 2 groups x 2 browsers (Main/Storybook x Chrome/Firefox). Recording also works fine, cypress.io dashboard recognizes 2 browsers and groups correctly, but I only get results for one browser (there is one result for Chrome for Main and one for Firefox for Storybook - the second browsers are visible in filters, but results for them are empty - please, see the screenshots).

I tried some flags and configurations from https://docs.cypress.io/guides/guides/parallelization.html

Unfortunately, no matter what I try, there is always some kind of error like:

You passed the --parallel flag, but we do not parallelize tests across different environments. This machine is sending different environment parameters than the first machine that started this parallel run. The existing run is: https://dashboard.cypress.io/projects/qwerty/runs/44 In order to run in parallel mode each machine must send identical environment parameters such as:

• specs
• osName
• osVersion
• browserName
• browserVersion (major) This machine sent the following parameters: { "osName": "linux", "osVersion": "Ubuntu - 20.04", "browserName": "Firefox", "browserVersion": "80.0.1", "specs": [ "src/integration/app/app.spec.ts" ] } https://on.cypress.io/parallel-group-params-mismatch

Only one browser per group passes and the 2nd one gets the following error.

These are my scripts in package.json:

# 1st --parallel flag for nx and 2nd after -- for Cypress directly
# $CYPRESS_RECORD_KEY - key provided by cypress.io
# $CI_ID - <branch-name>-<commit-hash> - e.g. my-new-branch-qweqtwerwtreyzsxfc4123dxfv

"e2e:ci": "nx e2e main-e2e --prod --headless --parallel --record --key $CYPRESS_RECORD_KEY --ci-build-id $CI_ID --group MainWeb -- --parallel",
"e2e:ci:chrome": "yarn e2e:ci --browser=chrome",
"e2e:ci:firefox": "yarn e2e:ci --browser=firefox",

"e2e:storybook:ci": "nx e2e storybook-e2e --prod --headless --parallel --record --key $CYPRESS_RECORD_KEY --ci-build-id $CI_ID --group Storybook -- --parallel",
"e2e:storybook:ci:chrome": "yarn e2e:storybook:ci --browser=chrome",
"e2e:storybook:ci:firefox": "yarn e2e:storybook:ci --browser=firefox"

And here is my .gitlab-ci.yml (e2e stage part only):

E2E Main-Web - Chrome:
  stage: e2e
  script:
    - yarn e2e:ci:chrome

E2E Main-Web - Firefox:
  stage: e2e
  script:
    - yarn e2e:ci:firefox

E2E Storybook - Chrome:
  stage: e2e
  script:
    - yarn e2e:storybook:ci:chrome

E2E Storybook - Firefox:
  stage: e2e
  script:
    - yarn e2e:storybook:ci:firefox

I'm not sure what is wrong here. I tried so many configurations, flags, solutions (even parallelization on Gitlab), but always there is something wrong...

Of course I set cypress.io > Project Settings > Parallelization > Run Completion Delay for 60s.

Answer 1

I think the answer lies in the Cypress error you shared:

You passed the --parallel flag, but we do not parallelize tests across different environments.

You are triggering two parallel runs, with each parallel run setting up two browsers. As the error says, this is not expected - each parallel run should have only one browser.

The solution in this case is easy: just configure four groups, one for each test and browser type.

This is how this could look:

"e2e:ci": "nx e2e main-e2e --prod --headless --parallel --record --key $CYPRESS_RECORD_KEY --ci-build-id $CI_ID -- --parallel",
"e2e:ci:chrome": "yarn e2e:ci --browser=chrome --group 'MainWeb Chrome' ",
"e2e:ci:firefox": "yarn e2e:ci --browser=firefox --group 'MainWeb Firefox' ",

"e2e:storybook:ci": "nx e2e storybook-e2e --prod --headless --parallel --record --key $CYPRESS_RECORD_KEY --ci-build-id $CI_ID -- --parallel",
"e2e:storybook:ci:chrome": "yarn e2e:storybook:ci --browser=chrome --group 'Storybook Chrome'",
"e2e:storybook:ci:firefox": "yarn e2e:storybook:ci --browser=firefox --group 'Storybook Firefox'"