Most Powerful Open Source ERP

ERP5 Unit Tests

Introduction

(please scroll to bottom for test results) ERP5 is used for many critical applications. Our customers depend on it to run their every day business, and they can not accept having regularly bugs coming again and again. Every software issue could have important consequences, especially if you think about banking, aerospace applications and many others. Therefore Nexedi considers the quality as one of its top priorities and runs automated testing since the early age of ERP5.

Unit / Functional / Deployment and Scalability Tests

While the code of ERP5 is evolving, more and more tests are added. When issues are reported, we write the test and fix the problem. Like this we make sure that the same issue is not going to happen again. Also, when new features are done, tests are added to make sure we will provide a well working experience. We have both unit tests and functional tests. Unit tests are used to test isolated software component, while functional tests are used to verify scenarios closer to user experience.

Development policy

We tried for years to keep the quantity of failures as low as possible. While the size of ERP5 was growing up, it started to be more and more time consuming to track regressions. In particular, it's quite common that a change in some part of the code generates a regression in another place that seems unrelated without detailed analysis. And then it happened that we had several errors, and it was difficult to see new ones.

Therefore, after our switch to git, we introduced the rule "contributions must not introduce ANY regression". So we started by fixing test failures, we checked all random issues, and we started to reject any code introducing failures. The idea is simply to check all unit tests before contributing changes to the central place. This can easily be done thanks to our Testing Infrastructure.

Thanks to this rule, about any revision of ERP5 could be considered as stable. This is a really great success. With a quantity of test growing, we have nearly 0 errors/failures all the time.

Testing Infrastructure

We are using what we call "ERP5 Test Nodes". They are processes installed on many different machines thanks to SlapOS cloud tools. We define several lists of tests to run, and ERP5 test nodes will look regularly to several code repositories to detect new commits, then they launch software constructions and finally tests are automatically run. Theses days we are using about 180 CPU cores to run tests for our various projects and for different versions.

Description of tests

Please find below a short description of all tests currently running:

  • ERP5.UnitTest-Master
    A unit test suite which covers all ERP5 code base.
  • ERP5.CodingStyleTest-Master
    Static code and configuration analysis for every business templates.
  • ERP5.ScalabilityTest-Master
    Tests how scalable an ERP5 instance is by simulating heavy intensive operations against a set of cluster configurations
  • ERP5.PerformanceTest-Master
    Tests pure ERP5 performance like creating and storing an object which should happen without any regressions for a already predefined time set
  • ERP5.StandaloneDeploymentScriptTest-Debian.Buster / ERP5.StandaloneDeploymentScriptTest-Debian.Buster.Master / ERP5.StandaloneDeploymentScriptTest-Debian.Bullseye
    A deployment test which assures that we can install ERP5 into a Debian Buster (10) / Bullseye (11) OS. The "Master" variant tests additionally that we can install the latest software release (from master branch), instantiate the product and apply a default automatic configuration. The non "Master" variant uses latest "frozen" version without automatic configuration testing. The non "Master" variant is much faster as software release compilation is downloaded pre-compiled from shacache.
  • Pygolang.UnitTest-Master.Python2 / Pygolang.UnitTest-Master.Python3
    A unit test which covers Pygolang ( Go-like features for Python and Cython )
  • Wendelin.UnitTest-Master
    A unit test suite which covers Wendelin (Big Data Platform) code base
  • Wendelin.CodingStyleTest-Master
    Static code and configuration analysis for Wendelin business templates.
  • Wendelin.core.UnitTest-ZODB4-wc2 / Wendelin.core.UnitTest-ZODB5
    A unit test suite which covers wendelin.core, Out-of-Core processing module from Wendelin (Big Data Platform), for both ZODB4 and ZODB5.
  • Wendelin.StandaloneDeploymentScriptTest-Debian.Buster /  Wendelin.StandaloneDeploymentScriptTest-Debian.Buster.Master / Wendelin.StandaloneDeploymentScriptTest-Debian.Bullseye
    Tests installing Wendelin inside Debian Buster (10) / Bullseye (11) OS.
  • NEO.UnitTest-Master / NEO.UnitTest-Master.ZODB3 / NEO.UnitTest-Master.ZODB5
    A unit test suite which covers the NEO (distributed, redundant and transactional storage DB)
  • Zodbtools.UnitTest-Master.ZODB4.Python2 / Zodbtools.UnitTest-Master.ZODB4.wc2.Python2 / Zodbtools.UnitTest-Master.ZODB5.Python2 / Zodbtools.UnitTest-Master.ZODB5.Python3
    A unit test suite for zodbtools , running for ZODB4, ZODB4 with wendelin.core2 support and ZODB5 on python 2 and python 3
  • JIO.UnitTest-Master
    A unit test suite (javascript) for our  JIO framework running on a web browser
  • JIO.UnitTest-Master.NodeJs
    A unit test suite (javascript) for our  JIO framework running on NodeJS
  • RenderJs.UnitTest-Master
    A unit test suite (javascript) for our RenderJS framework
  • OfficeJS.Appstore.UnitTest-Master
    A unit test which covers the OfficeJS Appstore code base
  • SlapOS.SlapRunner.ResilienceTest-Master
    Tests that the data produced by a simple Software Release inside a resilient webrunner is correctly backup-ed and restored in a backup webrunner, in order ensure a disaster recovery.
  • SlapOS.Theia.ResilienceTest-Master-Peertube
    Tests that the data produced by Peertube Software Release inside a resilient Theia is correctly backup-ed and restored in a backup Theia, in order to ensure a disaster recovery
  • SlapOS.Theia.ResilienceTest-Master-ERP5
    Tests that the data produced by ERP5 Software Release inside a resilient Theia is correctly backup-ed and restored in a backup Theia, in order to ensure a disaster recovery
  • SlapOS.Theia.ResilienceTest-Master-Gitlab
    Tests that the data produced by Gitlab Software Release inside a resilient Theia is correctly backup-ed and restored in a backup Theia, in order to ensure a disaster recovery
  • SlapOS.Eggs.UnitTest-Master.Python2 / SlapOS.Eggs.UnitTest-Master.Python3
    Tests all eggs which are been developed under SlapOS Project, and it may include recipes (slapos.cookbook, slapos.recipe.build...) and also tools (slapos.toolbox). Tests for different Python versions.
  • SlapOS.Buildout.UnitTest-Master.Python2 / SlapOS.Buildout.UnitTest-Master.Python3
    A unit test suite for slapos.buildout, our fork of zc.buildout
  • SlapOS-Master.UnitTest-Master
  • SlapOS.KVM.ResilienceTest-Master
    Tests that the disk used by a KVM is correctly backuped and can be reused by a second KVM, in order to ensure a disaster recovery.
  • SlapOS.SoftwareReleases.IntegrationTest-Master
    On every change on master branch, it compiles several Software Releases, like ERP5, Wendelin, WebRunner, KVM .... (see the full list) on the testnodes (currently we have testnodes running on Debian 10 and Debian 11).
  • SlapOS.SoftwareReleases.IntegrationTest-1.0
    On every change on 1.0 branch  (a.k.a Release Candidate Branch), it compiles several Software Releases, like ERP5, Wendelin, WebRunner, KVM .... (see the full list) on the testnodes (currently we have testnodes running on Debian 10 and Debian 11).
  • SlapOS.SoftwareReleases.IntegrationTest-No-Shacache-Master
    On every change on master branch, it compiles several Software Releases without accessing shacache, like ERP5, Wendelin, WebRunner, KVM .... (see the full list) on the testnodes (currently we have testnodes running on Debian 10 and Debian 11).
  • SlapOS-Master.StandaloneDeploymentScriptTest-Debian.Buster / SlapOS-Master.StandaloneDeploymentScriptTest-Debian.Bullseye
    Tests slapos' master installation on Debian Buster (10) / Bullseye (11) OS
  • nxdtest.UnitTest-Master.Python2 / nxdtest.UnitTest-Master.Python3
    Tox-like tool to run tests for a project under Nexedi testing infrastructure
  • Cloudooo.UnitTest-Master
    A unit test suite which tests our conversion daemon called cloudooo
  • Beremiz.UnitTest-Master.Python2.wxPython4
    A unit test suite  and functional test suite  which covers Beremiz (Beremiz is an integrated development environment for machine automation) code base.
  • Beremiz.IntegrationTest-Master
    An integration test suite which simulates the proper running of a Beremiz runtime environment controlling a virtual coupler device.

Test Results

You can find below the result of automatic test of ERP5 code. Hopefully you are able to see many test results at 0 errors/failures for several thousands of tests. Since we are humans and not perfect, it might happen that a change introduces a regression. But in this case, you could be sure that we will quickly look and solve it.

1 - 48 of 48 records
Date Test title All tests Skips Failures Errors Test Result Build Result
27/03/2024   17:36 ERP5.UnitTest-Master 8296 202 3 0
FAIL
Completed
27/03/2024   16:49 ERP5.CodingStyleTest-Master 7000 3 0 0
PASS
Completed
13/06/2022   21:21 ERP5.ScalabilityTest-Master 40 0 0 0
PASS
Completed
16/10/2023   02:20 ERP5.PerformanceTest-Master 14 0 2 0
FAIL
Completed
28/03/2024   03:28 ERP5.StandaloneDeploymentScriptTest-Debian.Buster 8 1 0 0
PASS
Completed
27/03/2024   17:54 ERP5.StandaloneDeploymentScriptTest-Debian.Buster.Master 8 1 0 0
PASS
Completed
29/12/2023   20:59 ERP5.StandaloneDeploymentScriptTest-Debian.Bullseye 8 1 0 0
PASS
Completed
26/03/2024   15:11 Pygolang.UnitTest-Master.Python2 254 15 0 0
PASS
Completed
26/03/2024   15:12 Pygolang.UnitTest-Master.Python3 254 15 19 2
FAIL
Completed
26/03/2024   15:16 Zodbtools.UnitTest-Master.ZODB4.Python2 48 0 0 0
PASS
Completed
26/03/2024   15:07 Zodbtools.UnitTest-Master.ZODB4.wc2.Python2 48 0 0 0
PASS
Completed
26/03/2024   16:26 Zodbtools.UnitTest-Master.ZODB5.Python2 48 0 0 0
PASS
Completed
26/03/2024   19:33 Zodbtools.UnitTest-Master.ZODB5.Python3 48 0 4 1
FAIL
Completed
27/03/2024   16:42 Wendelin.UnitTest-Master 73 0 0 0
PASS
Completed
26/03/2024   15:13 Wendelin.CodingStyleTest-Master 456 0 3 0
FAIL
Completed
26/03/2024   15:52 Wendelin.core.UnitTest-ZODB4-wc2 1181 8 0 0
PASS
Completed
26/03/2024   15:47 Wendelin.core.UnitTest-ZODB5 1181 8 0 0
PASS
Completed
27/03/2024   22:03 Wendelin.StandaloneDeploymentScriptTest-Debian.Bullseye.Master 8 1 0 0
PASS
Completed
28/03/2024   04:32 Wendelin.StandaloneDeploymentScriptTest-Debian.Buster 8 1 0 0
PASS
Completed
27/03/2024   17:36 Wendelin.StandaloneDeploymentScriptTest-Debian.Buster.Master 8 1 0 0
PASS
Completed
27/03/2024   10:30 Wendelin.StandaloneDeploymentScriptTest-Debian.Bullseye 8 1 0 0
PASS
Completed
27/03/2024   15:48 NEO.UnitTest-Master 1423 9 38 5
FAIL
Completed
02/03/2021   07:59 NEO.UnitTest-Master.ZODB3 841 2 26 1
FAIL
Completed
16/10/2023   01:23 NEO.UnitTest-Master.ZODB5 1402 9 39 13
FAIL
Completed
27/03/2024   02:36 JIO.UnitTest-Master 4341 0 0 0
PASS
Completed
27/03/2024   02:33 JIO.UnitTest-Master.NodeJs 3315 0 0 0
PASS
Completed
27/03/2024   02:32 RenderJs.UnitTest-Master 670 0 5 0
FAIL
Completed
21/12/2023   18:48 OfficeJS.Appstore.UnitTest-Master 19 0 0 0
PASS
Completed
16/01/2023   08:45 SlapOS.SlapRunner.ResilienceTest-Master 1 0 0 0
PASS
Completed
27/03/2024   02:20 SlapOS.Theia.ResilienceTest-Master-ERP5 2 0 0 0
PASS
Completed
27/03/2024   01:52 SlapOS.Theia.ResilienceTest-Master-Peertube 2 0 0 0
PASS
Completed
27/03/2024   00:34 SlapOS.Theia.ResilienceTest-Master-Gitlab 2 0 0 1
FAIL
Completed
28/03/2024   08:10 SlapOS.Eggs.UnitTest-Master.Python2 1371 55 4 0
FAIL
Completed
28/03/2024   08:10 SlapOS.Eggs.UnitTest-Master.Python3 1634 51 4 5
FAIL
Completed
27/03/2024   03:20 SlapOS.Buildout.UnitTest-Master.Python2 380 0 1 0
FAIL
Completed
27/03/2024   03:26 SlapOS.Buildout.UnitTest-Master.Python3 380 0 1 0
FAIL
Completed
28/03/2024   09:36 SlapOS-Master.UnitTest-Master 1336 15 5 0
FAIL
Completed
01/03/2023   13:18 SlapOS.KVM.ResilienceTest-Master 1 0 0 0
PASS
Completed
27/03/2024   06:54 SlapOS.SoftwareReleases.IntegrationTest-Master 1300 13 2 2
FAIL
Completed
27/03/2024   11:24 SlapOS.SoftwareReleases.IntegrationTest-1.0 1305 13 1 0
FAIL
Completed
23/01/2024   04:15 SlapOS-Master.StandaloneDeploymentScriptTest-Debian.Buster 8 1 0 0
PASS
Completed
22/01/2024   19:13 SlapOS-Master.StandaloneDeploymentScriptTest-Debian.Bullseye 8 1 0 0
PASS
Completed
26/03/2024   15:09 nxdtest.UnitTest-Master.Python2 46 3 0 0
PASS
Completed
26/03/2024   19:34 nxdtest.UnitTest-Master.Python3 46 3 0 0
PASS
Completed
26/03/2024   17:31 Cloudooo.UnitTest-Master 205 4 0 0
PASS
Completed
26/03/2024   16:07 Beremiz.UnitTest-Master.Python2.wxPython4 10 0 0 0
PASS
Completed
27/03/2024   19:10 Beremiz.IntegrationTest-Master 1 0 0 0
PASS
Completed
27/03/2024   16:38 MES5.UnitTest-Master 1 0 0 0
PASS
Completed