-
-
Notifications
You must be signed in to change notification settings - Fork 3.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Check for windows NTFS compression on monerod startup (fixes #9617) #9664
base: master
Are you sure you want to change the base?
Check for windows NTFS compression on monerod startup (fixes #9617) #9664
Conversation
Should probably only need to check the example
Testnet & stagenet will end up in subfolders like |
If there is no edge case where
That should be care of with |
Changed so that only the database directory is checked, and if it doesnt exist, it falls back to checking the data-dir. |
Either the software supports a compressed database/data dir, or it doesn't (I'm not certain as to the policy here, though compression + memory mapping seems like a bad idea); logging an ERROR and proceeding with startup doesn't make sense to me, and doesn't "fix" much of anything. |
My reasoning was that this error only seems to become a problem if the blockchain exceeds 70GB (see #9569 (comment)), so its not a reason to halt if e.g. running for a smaller chain such as the testnet (or if the on disk chain becomes smaller in the future due to e.g. further pruning improvements). Im very open to discussions as I am just getting familiar with the codebase. |
I think you're on the right track here, but, conceptually, the blockchain grows forever, so allowing startup in a configuration which will eventually break wastes the feature detection and proves quite irritating for users (mainnet is the most critical use case and it exceeds the failure threshold). I may have uncovered the underlying problem: "If you compress a file that is larger than 30 gigabytes, the compression may not succeed." (https://learn.microsoft.com/en-us/windows/win32/fileio/file-compression-and-decompression). Rather than simply warning the user, or aborting, I suggest transparently fixing the problem in the BlockchainLMDB class:
You can find info on the relevant windows APIs here:
Edit: Removed "maybe" qualifier from (2), as I think it's necessary because some users clone databases/data directories across systems, so the design shouldn't ignore existing, compressed database files. |
Very good points, I think this is the better approach. I will start implementing this in the new year and update this PR |
ceff273
to
50adf42
Compare
50adf42
to
ce21549
Compare
Thank you again for the great feedback. Please see the latest commit. |
ee9edf6
to
9ad2712
Compare
A clarification on handling directory-level compression for other reviewers: One could argue that if the directory already exists, the program should preserve its compression attribute; in fact, I suggested as much in earlier comments. However, upon further consideration, I think it makes sense to choose the most intrusive option (always uncompress the directory) because:
|
Can you confirm that this modification runs normally on Windows and results in an an unencrypted data directory + db post-shutdown in the following startup scenarios?
|
Yes, tested those configurations, as well as data-dir compressed but database-file not compressed and vice versa, with no issues. Should I write an additional unit-test? How is the procedure, should unit-tests be provided for any added functionality? I develop from Linux, so I cross-compile and test manually, but I can set up a development environment on Windows to write a unit test if desired. |
In most projects, I'd add tests for new behavior. Here, the tests ordinarily don't run on windows, so extra tests don't do much. If you want to add tests, you have two mutually exclusive options:
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There's a failing build. It looks like you need to explicitly add:
#include <boost/filesystem/fstream.hpp>
Everything else looks good. Thanks for your work on this issue!
One last thing: Probably a good idea to squash commits for merge. |
85baef4
to
feccf6b
Compare
Fixed and squashed. Thanks again for the good feedback! |
Fixes #9617
If the data-dir is compressed and the lmdb folder is (check for both, because if data-dir is compressed and lmdb doesnt exist yet, it is likely that the compression is configured to be applied recursively):
If just the db folder is compressed:
If not compressed at all:
Just logging an ERROR message, just like if Fat32 is used, not aborting, because for a small blockchain this not be a critical issue.
Can we assume that "lmdb" stays the name of the db folder? Since it is hardcoded here. Otherwise, one would need to define it somewhere central. I assumed so, because
BlockchainLMDB::get_db_name()
also has it hardcoded.