Zero-copy importer plugin APIs

[mosra]

The end goal for this:

• use Utility::Directory::mapRead() to memory-map a huge image / audio / scene file
• open the memory-mapped file via an importer and parse it without copying the data
• provide a view on the data to the user (again, no copy)
• take small slices of the imported data and upload them to the GPU / play them on the soundcard / ..., again no copy
• have AnimationData, MeshData, ImageData ... work directly on the memory-mapped data

The progress, as of 2021-10-24:

TODO:

• rework doOpenData() to accept an r-value Array and DataFlags describing whether it's temporary, owned by the importer, make openFile() then pass over the allocated array ownership there -- 116d327
  • make a protected openData() with the same signature and the usual assertions from the public openData() so proxy plugins such as AnyImageImporter or opening images from within other plugins can make use of the memory ownership annotations as well
• implement openMemory() as a zero-copy counterpart to openData() in Trade::AbstractImporter + DataFlag::ExternallyOwned, delegate to the above new `doOpenData() -- 27044a4
  • a mutable overload of it, passing DataFlag::Mutable to doOpenData()
  • expose in magnum-sceneconverter -- 03aeb49
  • make magnum-sceneconverter memory-map also referenced files via a callback
• implement openMemory() as a zero-copy counterpart to openData() in Audio::AbstractImporter these interfaces will eventually get deprecated and merged into Trade::AbstractImporter
• implement doOpenData() and openMemory() equivalently in Text::AbstractFont as well
• redesign Trade::MeshData2D / Trade::MeshData3D to eat an array (view) instead of vectors -- done in Mesh Data rework #371 instead
• add an ability to put an array view to Trade::ImageData2D / ... (convert it to an array with a custom no-op deleter internally) -- done in Mesh Data rework #371 instead
• implement Trade::AbstractImporter::mesh2DMemory() / ... that's able to pass through the mapped data in case the file was opened via openMemory() done differently in Mesh Data rework #371
• implement Trade::AbstractImporter::image2DMemory() / ... that's able to pass through the mapped data in case the file was opened via openMemory() done differently in Mesh Data rework #371
• implement Audio::Importer::data() that's able to pass through the mapped data in case the file was opened via openMemory() these interfaces will eventually get deprecated and merged into Trade::AbstractImporter
• Add an ImporterFlag::ZeroCopy flag, which will enable importers to pass the input data from openMemory() through to AnimationData / MeshData / ImageData / ... without copies
  • and silently falling back to copies if the data is not directly representable there (because it needs to be modified, flipped, swizzled, deinterleaved, whatever)
  • if the importer doesn't support it, it behaves same as without the flag (no error)
• Add an ImporterFlag::ForceZeroCopyThing for animations, images, mesh index/vertex buffers, ... that turns the above opt-in behavior to enforced
  • and a ImporterFeature::ZeroCopyThing with which the importer advertises ability to zero-copy certain data, which then gets assert for presence if the corresponding flag gets set
  • useful for example when editing image/mesh/scene files in-place, to ensure we're not editing an in-memory copy
  • or to ensure the app doesn't have any suboptimal behavior by accident
  • should the output data be checked? probably not, the base class doesn't remember the memory range anywhere
  • Also ImporterFeature::ZeroCopyScenes and ImporterFlag::ForceZeroCopyScenes once SceneData rework #525 is in
  • And apparently SkinData needs to expose jointDataFlags() & inverseBindMatrixDataFlags()
  • All those inflate the flag/feature enums which now need to be 16-bit (or maybe more with the YUp etc flags?), bump importer plugin interfaces for that
  • AnyImageImporter etc. flag propagation now needs to handle the cases of the flag not being supported in the target and provide a runtime error instead of an assert
• ImporterFlag::YUp & ImageConverterFlag::YUp to control whether images get Y-flipped on import and on conversion (otherwise it's not really possible to implement zero-copy import), similarly for ZBackward (and XLeft?)
  • ImageFlag::YUp to annotate what orientation the image is in so this information doesn't get lost after
  • Probably also need ImporterFlag::YDown / ImageConverterFlag::YDown and a presence of neither meaning "keep it in whatever orientation it was" to make the zero-copy import not annoying to set up
  • Decide how should pixels() behave -- they should probably return the view with X right, Y down, Z forward always to be consistent? That'll break existing code tho.
• Implement this in WavAudioImporter once Trade::AbstractImporter can do audio as well
• Implement this in FreeTypeFont
• Implement this in TgaImporter once the YUp flag is in (it also converts from ARGB or something, what to do ?)
• Implement this in DdsImporter once the YUp flag is in
• Implement this in KtxImporter once the YUp flag is in
• Implement this in StanfordImporter in progress, zerocopy branch
• Implement this in TinyGltfImageImporter (impossible) CgltfImporter
• Optional, for fun: implement ImageView::slice() and then upload a slice to GPU memory

Open questions / followup tasks:

• might be worth to investigate hugepages support for faster memory reservation (https://chrisgreendevelopmentblog.wordpress.com/2021/10/14/fun-with-large-memory-allocations-well-not-really-fun-at-all/) and expose such feature somehow

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 97.12%. Comparing base (9870cd7) to head (4446300).
Report is 2067 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff             @@
##           master     #240       +/-   ##
===========================================
+ Coverage   79.97%   97.12%   +17.15%     
===========================================
  Files         511      237      -274     
  Lines       32354    14863    -17491     
===========================================
- Hits        25875    14436    -11439     
+ Misses       6479      427     -6052     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.