Skip to main content

Continuous integration

Testing with GitHub Actions#

GitHub Actions is the Continuous Integration and Deployment tool of GitHub. It allows for running the tests automatically on every push to the Chemotion ELN GitHub repository. Every kind of test (JavaScript, Ruby and acceptance) has its own GitHub Actions workflow.

Skip tests#

If you want to run a subset of the test or commit without tests, you can use skip commands. Multiple skip commands can be combined. Insert one or more of the following skip commands in your commit message, where they are parsed by GitHub:

skip unit js
skip unit rb
skip acceptance
skip all

The acceptance test workflow contains linting, which can be skipped like so:

skip linting

User Interface#

Most of the interactions with GH actions can be done via the Web UI of GitHub. Workflows To view the status of the GH Actions (passing, failed or skipped), click on the Actions Tab in the repository. On the left side all workflows are grouped for each yml file. You can click on a button to view the single workflow runs of a workflow. The name of a button corresponds to the name attribute in the yml file.

The central panel lists all workflow runs of a single workflow yml file. Click on the ... button to delete one workflow run. Click on a single workflow to see the status (passing, failed or skipped) of the single jobs inside the workflow. Clicking on a job shows you the status of the single steps inside the job. In the steps view is a cog symbol in the upper right corner where you can get the log files of the corresponding job. The job and steps view contains a button in the right upper corner to Re-run jobs as well as a ... button allowing you to view the workflow yml file or create a status badge.

Developing CI with GitHub Actions#

In the following we shortly describe how to use GitHub Actions with a focus on the Chemotion ELN. For more information look at the official GitHub documentation for Actions.

GitHub Actions in general#

Runners#

Runners are applications provided by GitHub to execute the workflows. You can install your own self-hosted runner on your system or use runners hosted on GitHub servers. GitHub-hosted runners are specified by the OS keyword : ubuntu-latest, windows-latest, macos-latest etc. For more information see the GitHub documentation:

Self-hosted runners#

You need at least admin rights on a repository to add self-hosted runners. GitHub-hosted runners can be invoked with write access. Self-hosted runners must be linked to a repository (under Settings/Actions/Runners) and installed as a service on a local machine. The GitHub repository then invokes the service if a workflow is triggered and the actions are executed through the service on the local machine. See the GitHub documentation how to install runners. Runners have their own root directory actions-runner.

After installation you have to install the service of the runner:

cd actions-runner ;
sudo ./svc.sh install ;
sudo ./svc.sh start ;

The status of a self-hosted runner can be checked on both sides:

  • in the repository where all self-hosted runners are listed under Settings/Actions/Runners
  • on the local machine:
    cd actions-runner ;
    sudo ./svc.sh status ;

Workflows#

Workflows are a bundle of jobs and steps for creating tests. One workflow is defined in one yml file, there can not be multiple workflows defined within one yml file. Workflows are repeatable. To repeat a workflow without committing the code again click on the Re-run jobs button (see the UI).

Delete multiple workflow runs at once#

The easiest way to delete a whole workflow (i.e., all runs of it) at once is to use the GH CLI tool. There is no option to delete a whole workflow through the UI at the moment. With the CLI you can get all workflows of a repository owned by a user or an organization, then pick a specific workflow by its id, get all runs of this workflow and delete them sequentially.

// GET all workflows
gh api repos/{owner}/{repo}/actions/workflows
// GET all runs from one workflow
gh api repos/{owner}/{repo}/actions/{workflow_id}/runs
// DELETE single run
gh api repos/{owner}/{repo}/actions/runs/{run_id} -X DELETE

Combine the commands in a loop and edit the resulting JSON data with the tool jq (sudo apt-get install jq ) for example on the chemotion_saurus repository:

run_ids=( $(gh api repos/ComPlat/chemotion_saurus/actions/workflows/12327904/runs --paginate | jq '.workflow_runs[].id') )
for run_id in "${run_ids[@]}"
do
echo "Deleting Run ID $run_id"
gh api repos/ComPlat/chemotion_saurus/actions/runs/$run_id -X DELETE
done

Jobs#

Currently, on GitHub Jobs are not repeatable (as opposed to GitLab). To create repeatable tests you have to choose workflows instead of jobs for your tests. You can choose different runners and services for each job and run multiple jobs inside one workflow. By default jobs can run in parallel. If dependencies between jobs are needed, it can be specified with the needs keyword.

jobs:
deploy:
needs: build

Steps and actions#

Steps are sequences of actions. Actions are the smallest unit in GitHub Continuous Integration. Here you can place the commands to execute your tests, e.g., npm test. You can script your own actions or use pre-scripted actions from GitHub Marketplace. Actions inside steps depend on each other and can not run in parallel.

Workspace#

GH by default creates a workspaces to run actions. The path of the workspace can be read via ${{github.workspace}}. You must pay attention on your workspace path when using Docker containers with a WORKDIR. See Docker chapter.

Recommendation

When using some actions from the GitHub Marketplace e.g., the checkout action - the workspace is flushed before the action. Make sure to run other commands after checkout or replace the action with a custom solution. For example if you want to replace the checkout action use git clone ....

Variables#

GH offers different ways and levels to save variables:

  • When you have the appropriate permissions you can save environment variables as so called secrets in your GH repository under Settings/Secrets. This has the advantage, that you can use them in different workflows or across multiple repositories (for organization-level secrets). Call them with ${{ secrets.XXX }}.
  • You can create environment variables inside a workflow file with the env keyword on the workflow, job or step level. Call them with ${{ env.XXX }}. GH sets some default environment variables .
  • On the job level you can create a strategy.matrix. Call variables from that with ${{ matrix.XXX }}
  • On the step level you can create normal shell variables. Call them with $XXX.

Status badge#

A status badge per workflow can also be added to the README file. For quickly getting the Markdown code for a status badge go to the Actions tab on the GitHub repository, click on the workflow, click on the dots on the right upper corner and click on Create status badge.

Docker#

GH actions can run inside a Docker container and can also create a Docker container. For further information look at the official GitHub Documentation:

It is important to know that GitHub actions (for GitHub-hosted runners) automatically mount a Docker container at the WORKDIR of the Docker file, but not in the workspace of the GH actions. Therefore if you have actions that are by default set up in the workspace (e.g., by using actions from the GitHub Marketplace) or if you want to access the files from the Docker container keep in mind to set your directories accordingly, for example with the working-directory attribute in yml or with unix commands (cd).

The official documentation about running actions inside container can be a bit misleading because they suggest to not set the USER, because the default user in GH is root and the GH Actions workspace will not be available. Despite that you can handle this by setting the owners inside the Docker container in GH Actions to the original USER defined in the Docker file with chown and avoid needing the GH workspace with self-scripted actions.

Tools#

GitHub CLI (command line tool)#

See here: https://github.blog/2021-04-15-work-with-github-actions-in-your-terminal-with-github-cli/

GitHub API#

GitHub offers two kind of APIs: REST and GraphQL. Here is the documentation for the GitHub Actions REST API.

Tmate#

To maintain an open SSH session until the workflow run or the SSH session is closed, you can use a tmate action. It is useful for inspecting the server where the actions are running, for example check versions and paths, but the workflows itself can not be rerun.

Visual Studio Code#

One good extension for syntax checking in the workflow yml files is the extension GitHub Actions.

Authentication#

Token#

Some operations need a way to authenticate during a workflow run. GitHub provides an automatic token, which is only valid during a workflow run: GITHUB_TOKEN. In the workflow the token can be accessed like this:

${{ github.token }}
${{ secrets.GITHUB_TOKEN }}
GitHub bot#

Another way to authenticate is by using the bot user provided by GitHub: setup-git-user action. This is useful if you want to push to a repository inside a workflow. The GitHub bot prevents recursive workflows, that is, a push by the bor does not invoke the CI workflows. If you cannot use the setup-git-user action (because you are using a user other than root etc.), you can configure the GitHub bot user in the workflow with these credentials:

git config --global user.email "41898282+github-actions[bot]@users.noreply.github.com"
git config --global user.name "github-actions[bot]"

Node, webpack#

Do not set NODE_ENV=test when running Rails tests. Just use NODE_ENV=test for JavaScript tests. Otherwise the Babel configuration will get confused. The default for the NODE_ENV is development. For testing, set RAILS_ENV=test and make custom configurations for testing for example in webpack/config/development.js and with process.env properties. Have a look at these article for more details: Why doesn't Webpacker use my test config when I run Rails tests?, GitHub issue in the rails/webpacker repository. This article describes a way to customize the devtool setting: How to speed up assets precompile for Ruby on Rails apps. The devtool controls how error paths are resolved in the browser console. In test and production mode no error log is needed, therefore the devtool can be set to the fastest configuration. E. g. use this for compiling and running tests with the environment property DEVTOOL:

RAILS_ENV=test DEVTOOL=eval bundle exec rails webpacker:compile
RAILS_ENV=test bundle exec rspec spec/features

The webpack configuration could then be set like this in config/webpack/development.js:

process.env.NODE_ENV = process.env.NODE_ENV || 'development';
const webpackConfig = require('./base');
const { merge } = require('@rails/webpacker');
if (process.env.DEVTOOL) {
const debugConfig = {
devtool: process.env.DEVTOOL
};
module.exports = merge(webpackConfig, debugConfig);
} else {
module.exports = webpackConfig;
}

Values for the devtool variable are listed in the webpack documentation.

GitHub Actions for Chemotion ELN#

Workflow files#

Each kind of test (Ruby unit tests, JS unit test, acceptance tests) has its own workflow yml file, because that is the only way to make separate tests repeatable in GH Actions. We only use a few pre-scripted actions from the GH Marketplace, because of the special version settings required by nvm and Node and because pre-scripted actions are not easy to handle in our Docker containers.

Docker#

Because of the large Ruby and npm libraries used in Chemotion ELN the jobs are running inside a Docker container with prebuilt libraries. We use this Docker image which is built from the Dockerfile Chemotion ELN/Dockerfile.focal.gitlab-ci. The WORKDIR of the Docker image is set to /home/gitlab-runner. Consequently, all working-directory attributes in the workflows are set to /home/gitlab-runner as well as the HOME environment variable inside the Docker container in the workflow. The default root user inside Docker containers in GH actions is changed to the USER gitlab-runner from the ComPlat Docker image.

PostgreSQL#

For the database a PostgreSQL container is used.

Do not use 12 alpine version

Do not use image: postgres:12-alpine in the Chemotion ELN workflows. Otherwise there will be permission problems. Use image: postgres as a postgres service.

Acceptance tests#

Linting#
Info

This feature is still being reviewed and not available in the current development-5 branch.

Before the acceptance tests, JavaScript and Ruby files will be linted. The files will be automatically pushed to the repository by the GitHub Bot. If no files are changed, git add $FILES_JS $FILES_RB will throw an Error: Process completed with exit code 1., because there are no files to add. But the workflow for acceptance tests will continue because of continue-on-error: true in the workflow. Linting can be skipped with this command in the commit message:

skip linting

Common errors#

GitHub Actions services can be down#

If some GitHub services are not working, have a look at the status page.

Random seed numbers#

The most common error for failing Ruby unit and acceptance tests are caused by seed numbers. TODO: flaky tests caused by seeds are problematic and need to be addressed.

Yarn packages#

A lot of errors can be caused by a wrong Node or npm version or missing yarn packages:

> test

> mocha --require '@babel/register' './spec/javascripts/helper/setup.js' './spec/javascripts/*/.spec.js'

sh: 1: mocha: not found

source ~/.nvm/nvm.sh && nvm use 14.16.0 && yarn install

Database#

Migration error#

After bundle exec db:migrate RAILS_ENV=test this error occurs:

bundle exec rake db: migrate failed

rake aborted! StandardError: An error has occurred, this and all later migrations canceled:

undefined method `id' for nil:NilClass

Reset the database and migrate again:

bundle exec rake db:reset
bundle exec db:migrate RAILS_ENV=test

Npm test#

In JavaScript unit tests:

Error: Cannot find module '../extra/ElementStoreXlisteners'
One possible source for that can be an error in your PostgreSQL set up. Check if databases etc. exist and are properly configured.

Acceptance tests#

Memory (when using GitHub Actions)#

Compiling... Compilation failed:

<--- Last few GCs --->

[2363:0x59c7170] 144557 ms: Mark-sweep (reduce) 2030.2 (2054.6) -> 2029.8 (2055.1) MB, 2075.9 / 0.0 ms (average mu = 0.115, current mu = 0.006) allocation failure scavenge might not succeed [2363:0x59c7170] 147210 ms: Mark-sweep (reduce) 2030.9 (2052.1) -> 2030.3 (2053.1) MB, 2326.8 / 0.0 ms (average mu = 0.119, current mu = 0.123) allocation failure scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Solved with setting a Node flag to increase memory size. Setting this flag as environment variable in GH actions script. Maybe the value in MB needs to be adjusted.
--max_old_space_size=4096

Welcome message#

No route matches [GET] "/welcome-message.md"

See local acceptance testing.

Edit this page
Last updated on by Jan C. Brammer