Half precision, mixed precision, CUDA virtual devices

The release 0.4.1 offers: half precision, mixed precision, proper support for cuda virtual devices, and many bug fixes.

Non-beta release blocked by getting cudajit 0.4.1 in the opam-repository.

From the CHANGELOG:

Added

• Implemented the previously-mocked support for half precision (FP16).
  • We work around the missing Ctypes coverage by not using Ctypes.bigarray_start.
  • We check FP16 constants for overflow.
  • We output half precision specific code from the CUDA backend.
• Finally proper support for mixed precision! Lazy precision defaults and delayed precision setting via Tnode.update_prec.
• A placeholder nn_blocks.ml hinting at an intended design pattern for model components.
• A memory model for the multiple virtual devices per physical device setup, implemented in the CUDA backend.
  • It fixes the CUDA backend behavior in the data parallelism benchmark.

Changed

• Removed the pipes_cc, pipes_gccjit backends (Pipes_multicore_backend) -- I had fixed Pipes_multicore_backend by using the poll library instead of Unix.select, but it turns out to be very very slow.

Fixed

• Log levels related de-confusion:
  • Critical bug: logging of computation traces was not properly converted to ppx_minidebug 2.0.
  • Properly restore log_level and inform about its setting.
  • By default do not log from tests.
  • debug_log_from_routines should only happen when log_level > 1.
• Bugs in Multicore_backend: await was not checking queue emptiness, worker's Condition.broadcast was non-atomically guarded (doesn't need to be), possible deadloop due to the lockfree queue -- now replaced with saturn_lockfree.
• Reduced busy-waiting inside c_compile_and_load, propagating compilation errors now instead of infinite loop on error.
• Fixed loss of significant digits for small numbers when outputting files.
• Added missing mixed-precision conversions in the C_syntax backend builder.
• Restored the functionality of debug logging from the cuda backend.