Automated Testing Framework

OSv comes with over 120 unit tests that are run on every single commit. his patch adds number of shell and Python scripts to help automate integration testing which involves running variety of test apps on OSv and validate their behavior. The motivation is to have more expanded test bed beyond the unit tests to catch any regressions. They will become handy and give more confidence when testing more complicated patches. This new test framework will also help us automate building and running test images in new CI/CD process.

The main idea of the framework is quite simple: have a way to compose test images out of the pre-built packages using capstan and then run those images and validate their behavior by using app-specific test.sh shell scripts found in each app. Also instead of building each app from scratch every time we want to test it, we re-use pre-built capstan packages created using ./scripts/build-capstan-mpm-packages.

The script ./scripts/tests/compose_and_test_selected_apps.sh automates the process of composing the images and running the corresponding test.sh that normally delegates to one of following Python scripts:

• ./scripts/tests/test_app.py - runs an app and validates its standard output

• ./scripts/tests/test_app_with_test_script.py - runs an http, validates it http responses output and stress tests using apache bench

• ./scripts/tests/test_http_app_with_curl_and_ab.py - run an app and delegates to arbitrary Python tester script that would test an app in more sophisticated way like for example using sysbench for mysql or redis-benchmark for redis

Testing of following app has been automated:

• unit tests
• httpserver API
• cli (command line)
• golang-example
• golang-pie-example
• golang-httpserver
• golang-pie-httpserver
• rust-example
• rust-httpserver
• graalvm-example
• graalvm-httpserver
• graalvm-netty-plot
• python2-from-host
• python3-from-host
• node-express-example
• node-socketio-example
• lua-hello-from-host
• jetty
• tomcat
• vertx
• spring-boot-example
• apache-derby
• apache-kafka
• elasticsearch
• stream
• lighttpd
• nginx
• iperf3
• redis-memonly
• mysql
• ffmpeg