Get started with the OyaCloud CLI

Quick setup

1. 1. Create an account

   Sign up — new accounts go through a quick manual approval before the dashboard unlocks.

2. 2. Create a project and copy its API key

   In Projects, choose + New project. The key is shown as soon as the project is created; you can find it again under Settings → API Keys.

3. 3. Install the CLI

   For uploads from your own machine. Requires Node.js 14 or later.

   npm install -g oyacloud

   Copy

   On CI there’s nothing to install — the step in 7 runs it with npx.

4. 4. Choose your test framework

   Steps 5 and 7 update to match.

   Playwright (TypeScript/JS)

   playwright_ts

   Playwright (Python)

   playwright_py

   TestNG (Java)

   testng_java

   pytest (Python)

   pytest_python

   JUnit (Java)

   junit_java

   Cypress

   cypress_js

5. 5. Configure your reporter

   Add the JSON reporter to your Playwright config. Projects created with npm init playwright only have the HTML reporter, which OyaCloud can't read — keep it alongside if you use it:

   // playwright.config.ts
   reporter: [
     ['html'],
     ['json', { outputFile: 'test-results/results.json' }],
   ],

   Copy

   Failure screenshots named in the report are uploaded and linked to their tests. Videos, traces and other files stay on your machine.

   Start each run with a clean results folder

   Nothing to do — Playwright empties test-results/ at the start of every run. That's also why the JSON file goes inside it: Playwright's own example writes results.json to the project root, outside the folder you upload.

   • Sharded runs: upload the merged report from npx playwright merge-reports, or every shard's test-results folder. If both end up in one folder, each test is still counted once.
   • Shards run one after another on one machine: each run empties test-results/, so give each shard its own outputDir, or use the merge step.

6. 6. Store your API key in your CI

   Save it as a secret named OYACLOUD_API_KEY. Pick your platform — step 7 follows it.

   GitHub ActionsGitLab CICircleCIJenkins
   1. In your repository, open Settings → Secrets and variables → Actions.
   2. Choose New repository secret.
   3. Name it OYACLOUD_API_KEY, paste your key, and save.

   On your own machine, save it once instead:

   oyacloud config set-key YOUR_PROJECT_API_KEY

   Copy

7. 7. Add the upload step to your pipeline

   GitHub ActionsGitLab CICircleCIJenkins

   1. In the job that runs your tests, add this step after the test step:

      - name: Upload results to OyaCloud
        if: ${{ !cancelled() }}   # run even when tests fail
        env:
          OYACLOUD_API_KEY: ${{ secrets.OYACLOUD_API_KEY }}
        run: npx oyacloud upload --framework playwright_ts --environment staging --results-dir test-results

      Copy

   2. Make the workflow run on pushes and on pull requests:

      on:
        push:
          branches: [main]
        pull_request:

      Copy

   3. Push a commit and open the job log. On a push the upload step shows:

      ✅ Build Type: post-merge (detected from the GitHub Actions trigger)
      ✅ Uploaded · build …

      Copy

      Results appear on the build shortly. What each run counts as:

      • Push to main → post-merge
      • Pull request → pr, with the PR number and target branch
      • Schedule → regression

      Manual runs (Run workflow) carry no trigger — set OYACLOUD_BUILD_TYPE.

   Good to know

   • --environment is where the tests ran — required for post-merge and regression.
   • The CI image needs Node.js 14 or later, and direct outbound HTTPS to api.oyaplus.com and Amazon S3 (*.amazonaws.com). Proxies (HTTPS_PROXY) aren’t supported yet — contact support@oyaplus.com.
   • Part of your suite? Add --suite-name — Run scope. Preview what would be sent with --dry-run.

   On your own machine, after the tests:

   oyacloud upload --framework playwright_ts --environment staging --results-dir test-results

   Copy

Build types

Every upload has a build type. It decides what the run is compared with.

How the build type is chosen

The first of these that applies wins:

1. The --build-type flag.
2. The OYACLOUD_BUILD_TYPE environment variable.
3. Your CI platform’s trigger — what started the job, not the branch name:

4. Otherwise, from the branch details: a PR number makes it pr; a commit on main, master, develop or your project’s base branch makes it post-merge; anything else is regression.

CI details

The CLI reads branch, commit, PR number, target branch and author from your CI platform's environment and attaches them to every build. A flag always overrides what was detected; a field with no source is left blank.

• GitHub Actions — PR numbers are read for both pull_request and pull_request_target events. On a pull request the commit is the PR’s own latest commit, not the temporary merge commit GitHub tests.
• CircleCI* — depending on project settings, CircleCI may not provide the PR number for PRs from the same repository. It never provides the target branch; pass --target-branch.
• Jenkins — branch, PR number and target branch need a Multibranch Pipeline with PR discovery. Freestyle jobs get the commit only.

Run scope

OyaCloud compares a run only with earlier runs of the same scope, so a smoke run is never measured against your full suite. A full run needs nothing; a partial run needs a name — --suite-name.

Sensitive data

OyaCloud analyses what your test reports contain — test names, error messages, stack traces and screenshots. The CLI doesn't redact anything, so keep secrets and personal data out of your reports.

CLI reference

All options belong to oyacloud upload.

Exit codes

Limits

Per upload: 100 MB zipped, 200 MB of reports unzipped, 2,000 files, and 50 MB per report. Screenshots over 5 MB are skipped and listed, not treated as an error.

Network

On a self-hosted runner behind a firewall, allow outbound HTTPS to both:

• api.oyaplus.com — creates the build and returns its upload link
• *.s3.amazonaws.com — receives your reports as one zip, through a short-lived signed link

Supported frameworks

Troubleshooting

The CLI says…

Your dashboard shows…