Replace Marshal by Granular_marshal in ocaml-index #1889
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
voodoos
added a commit
to voodoos/merlin
that referenced
this pull request
Jan 21, 2025
voodoos
added a commit
to voodoos/merlin
that referenced
this pull request
Jan 22, 2025
voodoos
force-pushed
the
granular-marshal
branch
from
bc7eb27 to
e6b4bd3
Compare
Optimize use of reachable_words Fix fd leak: always close granlar_marshal in_channel Compressed index_format by de-duplicating filenames/longidents Fix fd leak: keep latest in_channel open for granular_marshal
voodoos
added a commit
to voodoos/merlin
that referenced
this pull request
Jan 22, 2025
voodoos
force-pushed
the
granular-marshal
branch
from
4c6e689 to
2f18e9b
Compare
voodoos
added a commit
to voodoos/merlin
that referenced
this pull request
Jan 22, 2025
voodoos
force-pushed
the
granular-marshal
branch
from
2f18e9b to
a1495ad
Compare
in the union find structures
voodoos
force-pushed
the
granular-marshal
branch
from
a1495ad to
250b46e
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Initial PR presentation by @Lucccyo:
The current implementation of ocaml-index uses Marshal to store on the disk the data.
Searching for occurrences on massive projects is time-consuming because the search loads all the data structures from the disk to perform the search.
This Pull Request aims to replace Marshal with a granular version to make the ocaml-index more efficient in reading.
It comes with two granular implementations of the data structures set and map, based on the Stdlib implementation.
During a search operation, the program lazily loads only the required part of the ocaml-index.
It works because the heavy nodes of the granular_map and granular_set have link indirections,
introducing serialization boundaries, which allows Marshal to delay the deserialization of their children.