Continuous integration
Testing with GitHub Actions
GitHub Actions is the Continuous Integration and Deployment tool of GitHub. It allows for running the tests automatically on every push to the Chemotion ELN GitHub repository. Every kind of test (JavaScript, Ruby and acceptance) has its own GitHub Actions workflow.
Skip tests
If you want to run a subset of the test or commit without tests, you can use skip commands. Multiple skip commands can be combined. Insert one or more of the following skip commands in your commit message, where they are parsed by GitHub:
skip unit js
skip unit rb
skip acceptance
skip all
The acceptance test workflow contains linting, which can be skipped like so:
skip linting
User Interface
Most of the interactions with GH actions can be done via the Web UI of GitHub.
To view the status of the GH Actions (passing, failed or skipped), click on the Actions Tab in the repository.
On the left side all workflows are grouped for each yml file.
You can click on a button to view the single workflow runs of a workflow.
The name of a button corresponds to the
name
attribute in the yml file.
The central panel lists all workflow runs of a single workflow yml file.
Click on the ...
button to delete one workflow run.
Click on a single workflow to see the status (passing, failed or skipped) of the single jobs inside the workflow.
Clicking on a job shows you the status of the single steps inside the job.
In the steps view is a cog symbol in the upper
right corner where you can get the log files of the corresponding job.
The job and steps view contains a button in the right upper corner to Re-run jobs
as well as a ...
button
allowing you to view the workflow yml file or create a status badge.
Developing CI with GitHub Actions
In the following we shortly describe how to use GitHub Actions with a focus on the Chemotion ELN. For more information look at the official GitHub documentation for Actions.
GitHub Actions in general
Runners
Runners are applications provided by GitHub to execute the workflows.
You can install your own self-hosted
runner on your system or use runners hosted on GitHub servers.
GitHub-hosted runners are specified by the OS keyword : ubuntu-latest
, windows-latest
, macos-latest
etc.
For more information see the GitHub documentation:
Self-hosted runners
You need at least admin rights on a repository to add self-hosted runners.
GitHub-hosted runners can be invoked with write access.
Self-hosted runners must be linked to a repository (under Settings/Actions/Runners
) and installed as a service on a local machine.
The GitHub repository then invokes the service if a workflow is triggered and the actions are executed through the service on the local machine.
See the GitHub documentation how to install runners.
Runners have their own root directory actions-runner
.
After installation you have to install the service of the runner:
cd actions-runner ;
sudo ./svc.sh install ;
sudo ./svc.sh start ;
The status of a self-hosted runner can be checked on both sides:
- in the repository where all self-hosted runners are listed under
Settings/Actions/Runners
- on the local machine:
cd actions-runner ;
sudo ./svc.sh status ;
Workflows
Workflows are a bundle of jobs
and steps
for creating tests.
One workflow is defined in one yml file, there can not be multiple workflows defined within one yml file.
Workflows are repeatable. To repeat a workflow without committing the code again click on the Re-run jobs
button (see the UI).
Delete multiple workflow runs at once
The easiest way to delete a whole workflow (i.e., all runs of it) at once is to use the GH CLI tool. There is no option to delete a whole workflow through the UI at the moment. With the CLI you can get all workflows of a repository owned by a user or an organization, then pick a specific workflow by its id, get all runs of this workflow and delete them sequentially.
// GET all workflows
gh api repos/{owner}/{repo}/actions/workflows
// GET all runs from one workflow
gh api repos/{owner}/{repo}/actions/{workflow_id}/runs
// DELETE single run
gh api repos/{owner}/{repo}/actions/runs/{run_id} -X DELETE
Combine the commands in a loop and edit the resulting JSON data with the tool jq
(sudo apt-get install jq
) for example on the chemotion_saurus repository:
run_ids=( $(gh api repos/ComPlat/chemotion_saurus/actions/workflows/12327904/runs --paginate | jq '.workflow_runs[].id') )
for run_id in "${run_ids[@]}"
do
echo "Deleting Run ID $run_id"
gh api repos/ComPlat/chemotion_saurus/actions/runs/$run_id -X DELETE
done
Jobs
Currently, on GitHub Jobs are not repeatable (as opposed to GitLab).
To create repeatable tests you have to choose workflows instead of jobs for your tests.
You can choose different runners and services for each job and run multiple jobs inside one workflow.
By default jobs can run in parallel. If dependencies between jobs are needed, it can be specified with the needs
keyword.
jobs:
deploy:
needs: build
Steps and actions
Steps are sequences of actions. Actions are the smallest unit in GitHub Continuous Integration.
Here you can place the commands to execute your tests, e.g., npm test
.
You can script your own actions or use pre-scripted actions from GitHub Marketplace.
Actions inside steps depend on each other and can not run in parallel.
Workspace
GH by default creates a workspaces to run actions. The path of the workspace can be read via ${{github.workspace}}
.
You must pay attention on your workspace path when using Docker containers with a WORKDIR
. See Docker chapter.
Recommendation
When using some actions from the GitHub Marketplace
e.g., the checkout
action - the workspace is flushed before the action. Make sure to run other commands
after checkout
or replace the action with a custom solution.
For example if you want to replace the checkout
action use git clone ...
.
Variables
GH offers different ways and levels to save variables:
- When you have the appropriate permissions you can save environment variables as so called
secrets
in your GH repository underSettings/Secrets
. This has the advantage, that you can use them in different workflows or across multiple repositories (for organization-level secrets). Call them with${{ secrets.XXX }}
. - You can create environment variables inside a workflow file with the
env
keyword on the workflow, job or step level. Call them with${{ env.XXX }}
. GH sets some default environment variables . - On the job level you can create a
strategy.matrix
. Call variables from that with${{ matrix.XXX }}
- On the step level you can create normal shell variables. Call them with
$XXX
.
Status badge
A status badge per workflow can also be added to the README
file.
For quickly getting the Markdown code for a status badge go to the Actions tab on the GitHub repository,
click on the workflow, click on the dots on the right upper corner and click on Create status badge
.
Docker
GH actions can run inside a Docker container and can also create a Docker container. For further information look at the official GitHub Documentation:
It is important to know that GitHub actions (for GitHub-hosted runners)
automatically mount a Docker container at the WORKDIR
of the Docker file,
but not in the workspace
of the GH actions.
Therefore if you have actions that are by default set up in the workspace
(e.g., by using actions from the GitHub Marketplace)
or if you want to access the files from the Docker container keep in mind to set your directories accordingly,
for example with the working-directory
attribute in yml or with unix commands (cd
).
The official documentation about running actions inside container can be a
bit misleading because they suggest to not set the USER
, because the default
user in GH is root and the GH Actions workspace
will not be available.
Despite that you can handle this by setting the owners inside the Docker
container in GH Actions to the original USER
defined in the Docker
file with chown
and avoid needing the GH workspace
with self-scripted actions.
Tools
GitHub CLI (command line tool)
See here: https://github.blog/2021-04-15-work-with-github-actions-in-your-terminal-with-github-cli/
GitHub API
GitHub offers two kind of APIs: REST and GraphQL. Here is the documentation for the GitHub Actions REST API.
Tmate
To maintain an open SSH session until the workflow run or the SSH session is closed, you can use a tmate action. It is useful for inspecting the server where the actions are running, for example check versions and paths, but the workflows itself can not be rerun.
Visual Studio Code
One good extension for syntax checking in the workflow yml files is the extension GitHub Actions.
Authentication
Token
Some operations need a way to authenticate during a workflow run. GitHub provides an automatic token, which is only valid during a workflow run: GITHUB_TOKEN. In the workflow the token can be accessed like this:
${{ github.token }}
${{ secrets.GITHUB_TOKEN }}
GitHub bot
Another way to authenticate is by using the bot user provided by GitHub: setup-git-user action. This is useful if you want to push to a repository inside a workflow. The GitHub bot prevents recursive workflows, that is, a push by the bor does not invoke the CI workflows. If you cannot use the setup-git-user action (because you are using a user other than root etc.), you can configure the GitHub bot user in the workflow with these credentials:
git config --global user.email "41898282+github-actions[bot]@users.noreply.github.com"
git config --global user.name "github-actions[bot]"
Node, webpack
Do not set NODE_ENV=test
when running Rails tests. Just use NODE_ENV=test
for JavaScript tests.
Otherwise the Babel configuration will get confused. The default for the NODE_ENV
is development
.
For testing, set RAILS_ENV=test
and make custom configurations for testing for example in webpack/config/development.js
and with process.env
properties.
Have a look at these article for more details:
Why doesn't Webpacker use my test config when I run Rails tests?,
GitHub issue in the rails/webpacker repository.
This article describes a way to customize the devtool setting:
How to speed up assets precompile for Ruby on Rails apps.
The devtool controls how error paths are resolved in the browser console. In test and production mode no error log is needed,
therefore the devtool can be set to the fastest configuration.
E. g. use this for compiling and running tests with the environment property DEVTOOL
:
RAILS_ENV=test DEVTOOL=eval bundle exec rails webpacker:compile
RAILS_ENV=test bundle exec rspec spec/features
The webpack configuration could then be set like this in config/webpack/development.js
:
process.env.NODE_ENV = process.env.NODE_ENV || 'development';
const webpackConfig = require('./base');
const { merge } = require('@rails/webpacker');
if (process.env.DEVTOOL) {
const debugConfig = {
devtool: process.env.DEVTOOL
};
module.exports = merge(webpackConfig, debugConfig);
} else {
module.exports = webpackConfig;
}
Values for the devtool
variable are listed in the webpack documentation.
GitHub Actions for Chemotion ELN
Workflow files
Each kind of test (Ruby unit tests, JS unit test, acceptance tests) has its own workflow yml file,
because that is the only way to make separate tests repeatable in GH Actions.
We only use a few pre-scripted actions from the GH Marketplace,
because of the special version settings required by nvm
and Node
and because
pre-scripted actions are not easy to handle in our Docker containers.
Docker
Because of the large Ruby and npm libraries used in Chemotion ELN the jobs are running inside a Docker container with prebuilt libraries.
We use this Docker image which is built from the Dockerfile Chemotion ELN/Dockerfile.focal.gitlab-ci
.
The WORKDIR
of the Docker image is set to /home/gitlab-runner
. Consequently, all working-directory
attributes in the workflows are set to /home/gitlab-runner
as well as the HOME
environment variable inside the Docker container in the workflow.
The default root user inside Docker containers in GH actions is changed to the USER gitlab-runner
from the ComPlat Docker image.
PostgreSQL
For the database a PostgreSQL container is used.
Do not use 12 alpine version
Do not use image: postgres:12-alpine
in the Chemotion ELN workflows. Otherwise there will be permission problems.
Use image: postgres
as a postgres service.
Acceptance tests
Linting
Info
This feature is still being reviewed and not available in the current development-5 branch.
Before the acceptance tests, JavaScript and Ruby files will be linted.
The files will be automatically pushed to the repository by the GitHub Bot.
If no files are changed, git add $FILES_JS $FILES_RB
will throw an Error: Process completed with exit code 1.
,
because there are no files to add. But the workflow for acceptance tests will continue because of continue-on-error: true
in the workflow.
Linting can be skipped with this command in the commit message:
skip linting
Common errors
GitHub Actions services can be down
If some GitHub services are not working, have a look at the status page.
Random seed numbers
The most common error for failing Ruby unit and acceptance tests are caused by seed numbers. TODO: flaky tests caused by seeds are problematic and need to be addressed.
Yarn packages
A lot of errors can be caused by a wrong Node or npm version or missing yarn packages:
> mocha --require '@babel/register' './spec/javascripts/helper/setup.js' './spec/javascripts/*/.spec.js'
sh: 1: mocha: not found
source ~/.nvm/nvm.sh && nvm use 14.16.0 && yarn install
Database
Migration error
After bundle exec db:migrate RAILS_ENV=test
this error occurs:
rake aborted! StandardError: An error has occurred, this and all later migrations canceled:
undefined method `id' for nil:NilClass
Reset the database and migrate again:
bundle exec rake db:reset
bundle exec db:migrate RAILS_ENV=test
Npm test
In JavaScript unit tests:
Acceptance tests
Memory (when using GitHub Actions)
<--- Last few GCs --->
[2363:0x59c7170] 144557 ms: Mark-sweep (reduce) 2030.2 (2054.6) -> 2029.8 (2055.1) MB, 2075.9 / 0.0 ms (average mu = 0.115, current mu = 0.006) allocation failure scavenge might not succeed [2363:0x59c7170] 147210 ms: Mark-sweep (reduce) 2030.9 (2052.1) -> 2030.3 (2053.1) MB, 2326.8 / 0.0 ms (average mu = 0.119, current mu = 0.123) allocation failure scavenge might not succeed
<--- JS stacktrace --->
FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
--max_old_space_size=4096