Skip to main content

Testing Code

There are three test classes for Chemotion ELN available: JavaScript (npm) and Ruby unit tests and acceptance (feature) tests.

Local testing for Chemotion ELN#

For locally testing Chemotion ELN on a developer system, change to the root directory of your Chemotion ELN code and type following commands:

Javascript (npm) unit tests#

source ~/.rvm/scripts/rvm && source ~/.nvm/nvm.sh && nvm use && RAILS_ENV=test bundle exec rake db:test:prepare && npm test

Ruby unit tests#

source ~/.rvm/scripts/rvm && RAILS_ENV=test bundle exec rake db:test:prepare && bundle exec rspec --exclude-pattern spec/{features}/**/*_spec.rb

Acceptance tests#

source ~/.rvm/scripts/rvm && source ~/.nvm/nvm.sh && nvm use && RAILS_ENV=test bundle exec rake db:test:prepare && bundle exec rake assets:precompile && bundle exec rspec spec/features

When you have freshly installed Chemotion ELN, make sure to create a welcome-message.md file in the public directory inside Chemotion ELN before running acceptance tests. You can create it from the example file there:

cp public/welcome-message-sample.md public/welcome-message.md

Remote testing with GitHub Actions#

GitHub Actions are the Continuous Integration and Deployment tool of GitHub. They are used for testing newly created code in development and activated on every push to the Chemotion ELN GitHub repository](https://github.com/ComPlat/ Chemotion ELN). Every test class (JavaScript, Ruby and acceptance) has its own workflow.

Skip tests#

If you only want to test a single or two test classes or commit without tests, you can use skip commands. Multiple skip commands can be combined. Insert one or more from the following skip commands in your commit message, where they are parsed by GitHub:

skip unit js
skip unit rb
skip acceptance
skip all

In the file with the acceptance tests is some linting included. Linting can be skipped with this command in the commit message:

skip linting

User Interface#

Most of the interactions with GH actions can be done via the Web UI of GitHub.
Workflows To view the status of the GH Actions (passing, failed or skipped), click on the Actions Tab in the repository. On the left side are all workflows per single yml file grouped. You can click on a button to view the single workflow runs of a workflow. The name of a button corresponds to the name attribute in the yml file.

In the middle part are all workflow runs of a single workflow yml file listed. Click on the ... button to delete one workflow run. Click on a single workflow to see the status (passing, failed or skipped) of the single jobs inside the workflow. Clicking on a job shows you the status of the single steps inside the job. In the steps view is a cog symbol in the upper right corner where you can get the log files of the corresponding job. In the job and steps view is also a button in the right upper corner to Re-run jobs and a ... button where you can view the workflow yml file or create a status badge.

Developing CI with GitHub Actions#

The following text describes shortly how to develop and use GitHub Actions especially for the Chemotion ELN. For deeper and broader information look at the official GitHub documentation for Actions.

GitHub Actions in general#

Runners#

Runners are applications provided by GitHub to execute the workflows. You can install your own self-hosted runner on your system or use runners hosted on GitHub servers. GitHub-hosted runners are specified by the keyword for the OS they are running on: ubuntu-latest, windows-latest, macos-latest etc. For more informations see the GitHub documentation:

Self-hosted runners#

You need at least admin rights on a repository to add self-hosted runners. GitHub-hosted runners can be already invoked with write access. Self-hosted runners must be linked to a repository (under Settings/Actions/Runners) and installed as a service on a computer. The GitHub repository then invokes the service if a workflow is triggered and the actions are executed through the service on the computer. See the GitHub documentation how to install runners. Runners have their own root directory actions-runner.

After installation you have to install the service of the runner:

cd actions-runner ;
sudo ./svc.sh install ;
sudo ./svc.sh start ;

The status of a self-hosted runner can be checked on both sides:

  • in the repository where all self-hsoted runners are listed under Settings/Actions/Runners
  • on the computer:
    cd actions-runner ;
    sudo ./svc.sh status ;

Workflows#

Workflows are a bundle of jobs and steps for creating tests. One workflow is defined in one yml file, there can not be multiple workflows defined within one yml file. Workflows are repeatable. To repeat a workflow without commiting the code again click on the Re-run jobs button (see the UI).

Delete multiple workflow runs at once#

The easiest way to delete a whole workflow (= all runs of it) at once is to use the GH CLI tool. There is no option to delete a whole workflow through the UI at the moment. With the CLI you can get all workflows of a repository owned by a user or a an organisation, then pick a specific workflow by its id, get all runs of this workflow and delete them sequentially.

// GET all workflows
gh api repos/{owner}/{repo}/actions/workflows
// GET all runs from one workflow
gh api repos/{owner}/{repo}/actions/{workflow_id}/runs
// DELETE single run
gh api repos/{owner}/{repo}/actions/runs/{run_id} -X DELETE

Combine the commands in a loop and edit outcoming JSON data with the tool jq (sudo apt-get install jq ) for example on the chemotion_saurus repository:

run_ids=( $(gh api repos/ComPlat/chemotion_saurus/actions/workflows/12327904/runs --paginate | jq '.workflow_runs[].id') )
for run_id in "${run_ids[@]}"
do
echo "Deleting Run ID $run_id"
gh api repos/ComPlat/chemotion_saurus/actions/runs/$run_id -X DELETE
done

Jobs#

Jobs are in the current GH version - in opposite to GitLab CI - not repeatable. To create repeatable tests you have to choose workflows instead of jobs for your tests. You can choose different runners and services for each job and run multiple jobs inside one workflow. By default jobs can run in parallel. If dependencies between jobs are needed, it can be specified with the needs keyword. E. g.:

jobs:
deploy:
needs: build

Steps and actions#

Steps are sequences of actions. Actions are the smallest unit in GitHub Continuous Integration. Here you can place the commands to execute your tests, e. g. npm test. You can script your own actions or use prescripted actions from GitHub Marketplace. Because of their sequence character actions inside steps depend on each other and can not run in parallel.

Workspace#

GH by default creates a workspaces to run actions inside of it. The path of the workspace can be read via ${{github.workspace}}. You must pay attention on your workspace path when using Docker containers with a WORKDIR. See Docker chapter.

Recommendation

When using some actions from the GitHub Marketplace e. g. the checkout action - the workspace is flushed before the action. So make sure to run every other commands after that action or replace the action with a self-scripted code. For example if you want to replace the checkout action use git clone ....

Variables#

GH offers different ways and levels to save variables:

  • When you have the appropriate permissions you can save environment variables as so called secrets in your GH repository under Settings/Secrets. This has the advantage, that you can use them in different workflows or when set on organization level accross multiple repositories. Call them with ${{ secrets.XXX }}.
  • You can create environment variables inside a workflow file with the env keyoword on workflow, job or step level. Call them with ${{ env.XXX }}. There are some default environment variables set in GH.
  • On job level you can create a strategy.matrix. Call variables from that with ${{ matrix.XXX }}
  • On step level you can create normal shell variables. Call them with $XXX.

Status badge#

A status badge per workflow can also be added to the README file. For quickly getting the Markdown code for a status badge go to the Actions tab on the GitHub repository, click on the workflow, click on the dots on the right upper corner and click on Create status badge.

Docker#

GH actions can run inside a Docker container and can also create a Docker container. For furter information look at the official GitHub Documentation:

It is important to know that GitHub actions (for GitHub-hosted runners) automatically mount a Docker container at the WORKDIR of the Docker file, but not in the normal workspace of the GH actions. Therefore if you have actions that are by default set up in the workspace (e. g. by using actions from the GitHub Marketplace) or if you want to access the files from the Docker container keep in mind to set your directories accordingly, for example with the working-directory attribute in yml or with simple unix commands (cd).

The official documentation about running actions inside container can be a bit misleading because they suggest to not set the USER, because the default user in GH is root and the GH Actions workspace will not be available. Despite that you can handle that with setting the owners inside the Docker container in GH Actions again to the original USER defined in the Docker file with chown and avoid needing the GH workspace with self-scripted actions.

Tools#

GitHub CLI (command line tool)#

See here: https://github.blog/2021-04-15-work-with-github-actions-in-your-terminal-with-github-cli/

GitHub API#

GitHub offers two kind of APIs: REST and GraphQL. Here is the documentation for the GitHub Actions REST API.

Tmate#

To create an constant open SSH session until (the workflow run or the SSH session is) closed, you can use a tmate action. It is useful for inspecting the server where the actions are running, for example check versions and paths, but the workflows itself can not be rerun.

Visual Studio Code#

One good extension for syntax checking in the workflow yml files is the extension GitHub Actions.

Authentication#

Token#

Some operations need a way to authenticate during a workflow run. GitHub provides an automatic token, which is only valid during a workflow run: GITHUB_TOKEN. In the workflow the token can be accessed like this:

${{ github.token }}
${{ secrets.GITHUB_TOKEN }}
GitHub bot#

Another way to authenticate is using the bot user provided by GitHub, e. g. with this action: setup-git-user action. This is useful if you want to push to a repository inside a workflow. The GitHub bot prevents recursive workflows, that means an automatic push from it does not invoke the normal CI workflows for the repsoitory. If you can not use the setup-git-user action (because you are using another user than root etc.), you can configure the GitHub bot user in the workflow with these credentials:

git config --global user.email "41898282+github-actions[bot]@users.noreply.github.com"
git config --global user.name "github-actions[bot]"

Node, webpack#

Do not set NODE_ENV=test when running Rails tests. Just use NODE_ENV=test for JavaScript tests. Otherwise the Babel configuration will get confused. The default for the NODE_ENV is development. For testing set RAILS_ENV=test and make custom configurations for testing for example in webpack/config/development.js and with process.env properties. Read for further informations this article: Why doesn't Webpacker use my test config when I run Rails tests? and this GitHub issue in the rails/webpacker repository. This article describes a way to customize the devtool setting: How to speed up assets precompile for Ruby on Rails apps. The devtool controls how error paths are resolved in the browser console. In test and production mode no error log is needed, therefore the devtool can be set to the fastest configuration. E. g. use this for compiling and running tests with the environment property DEVTOOL:

RAILS_ENV=test DEVTOOL=eval bundle exec rails webpacker:compile
RAILS_ENV=test bundle exec rspec spec/features

The webpack configuration could then be set like this in config/webpack/development.js:

process.env.NODE_ENV = process.env.NODE_ENV || 'development';
const webpackConfig = require('./base');
const { merge } = require('@rails/webpacker');
if (process.env.DEVTOOL) {
const debugConfig = {
devtool: process.env.DEVTOOL
};
module.exports = merge(webpackConfig, debugConfig);
} else {
module.exports = webpackConfig;
}

Values for the devtool variable are listed in the webpack documentation.

GitHub Actions for Chemotion ELN#

Workflow files#

Each of the three test classes (Ruby unit tests, JS unit test, acceptance tests) has its own workflow yml file, because that is the only way to make single test classes repeatable in GH Actions. There are just a few prescripted actions from GH Marketplace used, because of the special version settings for nvm and Node and because prescripted actions are not easy to handle in self-written Docker containers.

Docker#

Because of the large Ruby and npm libraries used in Chemotion ELN the jobs are running inside a Docker container with prebuilt libraries, This Dockerimage built from the Dockerfile Chemotion ELN/Dockerfile.focal.gitlab-ci is used. The WORKDIR of the Docker image is set to /home/gitlab-runner therefore all working-directory attributes in the workflows are set to that and also the HOME environment variable inside the Docker container in the workflow. The default root user inside Docker containers in GH actions is changed to the USER gitlab-runner from the complat Docker image.

PostgreSQL#

For the database a PostgreSQL container is used. The databases and users must be extra created.

Do not use 12 alpine version

Do not use image: postgres:12-alpine in the Chemotion ELN workflows. Otherwise there will be permission problems.
Use image: postgres as a postgres service.

Acceptance tests#

Linting#
Info

This feature is still being reviewed and not available in the current development-5 branch.

Before the acceptance tests are running, changed JavaScript or Ruby files will be linted. The changed files will be automatically pushed to the repository via the GitHub Bot. If no files are changed, git add $FILES_JS $FILES_RB will throw an Error: Process completed with exit code 1., because there are no files to add. But the workflow for acceptance tests will continue because of continue-on-error: true in the workflow.
Linting can be skipped with this command in the commit message:

skip linting

Common errors#

With GitHub Actions#

If some GitHub services are not working, may look at the status page.

Random seed numbers#

The most common error for failing ruby unit and acceptance tests are caused by seed numbers. Just run the tests multiple times again, until a suitable seed number is found.

Yarn packages#

A lot of errors are caused by a wrong Node or npm version or missing yarn packages:

> test

> mocha --require '@babel/register' './spec/javascripts/helper/setup.js' './spec/javascripts/*/.spec.js'

sh: 1: mocha: not found

source ~/.nvm/nvm.sh && nvm use 14.16.0 && yarn install

Database#

Migration error#

After bundle exec db:migrate RAILS_ENV=test this error occurs:

bundle exec rake db: migrate failed

rake aborted! StandardError: An error has occurred, this and all later migrations canceled:

undefined method `id' for nil:NilClass

Reset the database and migrate again:

bundle exec rake db:reset
bundle exec db:migrate RAILS_ENV=test

Npm test#

In JavaScript unit tests:

Error: Cannot find module '../extra/ElementStoreXlisteners'
One possible source for that can be an error in your PostGres set up. Check if databases etc. exist and are properly configured.

Acceptance tests#

Memory (when using GitHub Actions)#

Compiling... Compilation failed:

<--- Last few GCs --->

[2363:0x59c7170] 144557 ms: Mark-sweep (reduce) 2030.2 (2054.6) -> 2029.8 (2055.1) MB, 2075.9 / 0.0 ms (average mu = 0.115, current mu = 0.006) allocation failure scavenge might not succeed [2363:0x59c7170] 147210 ms: Mark-sweep (reduce) 2030.9 (2052.1) -> 2030.3 (2053.1) MB, 2326.8 / 0.0 ms (average mu = 0.119, current mu = 0.123) allocation failure scavenge might not succeed

<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory

Solved with setting a Node flag to increase memory size. Setting this flag as environment variable in GH actions script. Maybe the value in MB needs to be adjusted.
--max_old_space_size=4096

Welcome message#

No route matches [GET] "/welcome-message.md"

See local acceptance testing.

Last updated on by maipy