Automated Testing Framework

The main idea of the framework is quite simple: have a way to compose test images out of the pre-built packages using capstan and then run those images and validate their behavior by using app-specific test.sh shell scripts found in each app. Also instead of building each app from scratch every time we want to test it, we re-use pre-built capstan packages created using ./scripts/build-capstan-mpm-packages. This means that one can keep handy a capstan repo (or many) with a set of test packages that can be used to test and catch any regressions in OSv beyond what the regular unit tests do.

The script ./scripts/tests/compose_and_test_selected_apps.sh automates the process of composing the images and running the test.sh in the corresponding apps or modules directory. The test.sh normally delegates to one of following Python scripts:

• ./scripts/tests/test_app.py - run an app and validate its standard output for expected text messages

• ./scripts/tests/test_http_app.py- run an http app, validate its http responses and stress test using apache bench (ab) or wrk

• ./scripts/tests/test_app_with_test_script.py - run an app and delegate to arbitrary Python tester script located in the app or module directory that would test an app in a more sophisticated way like for example using sysbench for mysql or redis-benchmark for redis.

As you can see below, the compose_and_test_selected_apps.sh can build and execute all test images (default mode), a category of test images (simple, http, with_tester, unit_tests) or single app. You can also run the script to compose the tests images (-c) or run them only (-r). In addition, you can build images with RoFS or ZFS filesystem. Finally, you can run the tests on QEMU (default) or firecracker (-f). Running all tests on firecracker requires setting up networking that uses NAT setup and requires physical NIC that should be passed by setting the environment variable OSV_FC_NIC.

./scripts/tests/compose_and_test_selected_apps.sh 
Compose and test apps out ouf pre-built capstan packages

Usage: compose_and_test_selected_apps.sh [options] <group_of_tests> | <test_app_name>
Options:
  -c              Compose test app image only
  -r              Run test app only (must have been composed earlier)
  -R              Compose test app image with RoFS (ZFS is the default)
  -l              Use latest OSv kernel from build/last to build test image
  -f              Run OSv on firecracker

Test groups:
  simple - simple apps like golang-example
  http - httpserver apps
  http-java - java httpserver apps
  java <name> - java app
  http-node - node http apps
  node <name> - node app
  with_tester - apps tested with extra tester script like redis
  unit_tests - unit tests
  httpserver_api_tests - httpserver API unit tests
  all - all apps

At this point testing of following apps is automated:

*** Describe how each one is tested

• unit tests
• httpserver API
• cli (command line)
• golang-example
• golang-pie-example
• golang-httpserver
• golang-pie-httpserver
• rust-example
• rust-httpserver
• graalvm-example
• graalvm-httpserver
• graalvm-netty-plot
• python2-from-host
• python3-from-host
• node-express-example
• node-socketio-example
• lua-hello-from-host
• jetty
• tomcat
• vertxTESTER
• spring-boot-example
• apache-derby
• apache-kafka
• elasticsearch
• stream
• specjvm
• lighttpd
• nginx
• iperf3
• redis-memonly
• keydb
• mysql
• ffmpeg

** What has to be installed.

It should be very easy to automate new app by adding relevant test.sh script.

Following environment variables can be used to customize running the tests:

• OSV_KERNEL - the path of the kernel.elf` used to run the tests apps on
• TESTER - name of the tester app used to stress test http apps: ab (default) or wrk
• TESTER_CONCURRENCY - number of concurrent requests used when stress-testing http apps
• TESTER_COUNT - number of requests made by ab
• TESTER_DURATION - duration of stress test run in seconds if TESTER is wrk
• TESTER_THREADS - number of worker threads if TESTER is wrk
• OSV_FC_NIC - name of the physical NIC to be used when setting up firecracker networking (see this wiki page)
• `YCSB_HOME' - location of the YCSB testing project directory used to stress tests redis and keydb

** Example runs: