How to reduce the number of times memory written to disk in the database analyze process.

[Eathonhsu]

We had a large project (based on cpp development) try to execute the CodeQL.

We're running into slow database analysis times.

We tracked it down - looks like some tests (ReturnStackAllocatedMemory, RedundantNullCheckParam) are constantly writing memory to disk.

Example: (Counts about 16500 time the similar messages).

...
[2024-12-27 07:47:19] Pausing evaluation to evict 238.89MiB ARRAYS at sequence stamp o+13268168
[2024-12-27 07:47:19] Unpausing evaluation: 118.33MiB in memory written to disk, 238.99MiB forgotten: 120.66MiB UNREACHABLE (156 items up to o+13268158) 118.33MiB VITAL (153 items up to o+13248308)
[2024-12-27 07:47:19] Pausing evaluation to evict 238.89MiB ARRAYS at sequence stamp o+13268479
[2024-12-27 07:47:19] Unpausing evaluation: 119.11MiB in memory written to disk, 238.99MiB forgotten: 119.88MiB UNREACHABLE (155 items up to o+13268475) 119.11MiB VITAL (154 items up to o+13248615)
[2024-12-27 07:47:19] Pausing evaluation to evict 238.89MiB ARRAYS at sequence stamp o+13268787
[2024-12-27 07:47:20] Unpausing evaluation: 121.43MiB in memory written to disk, 238.99MiB forgotten: 117.56MiB UNREACHABLE (152 items up to o+13268778) 121.43MiB VITAL (157 items up to o+13248929)
[2024-12-27 07:47:20] Pausing evaluation to evict 238.89MiB ARRAYS at sequence stamp o+13269098
[2024-12-27 07:47:20] Unpausing evaluation: 118.33MiB in memory written to disk, 238.99MiB forgotten: 120.66MiB UNREACHABLE (156 items up to o+13269090) 118.33MiB VITAL (153 items up to o+13249236)
[2024-12-27 07:47:20] Pausing evaluation to evict 238.89MiB ARRAYS at sequence stamp o+13269405
[2024-12-27 07:47:20] Unpausing evaluation: 119.88MiB in memory written to disk, 238.99MiB forgotten: 119.11MiB UNREACHABLE (154 items up to o+13269397) 119.88MiB VITAL (155 items up to o+13249548)
[2024-12-27 07:47:20] Pausing evaluation to evict 238.89MiB ARRAYS at sequence stamp o+13269719
[2024-12-27 07:47:20] Unpausing evaluation: 118.33MiB in memory written to disk, 238.99MiB forgotten: 120.66MiB UNREACHABLE (156 items up to o+13269711) 118.33MiB VITAL (153 items up to o+13249855)
[2024-12-27 07:47:20] Pausing evaluation to evict 238.89MiB ARRAYS at sequence stamp o+13270031
[2024-12-27 07:47:21] Unpausing evaluation: 118.33MiB in memory written to disk, 238.99MiB forgotten: 120.66MiB UNREACHABLE (156 items up to o+13270022) 118.33MiB VITAL (153 items up to o+13250157)
[2024-12-27 07:47:21] Pausing evaluation to evict 238.89MiB ARRAYS at sequence stamp o+13270333
[2024-12-27 07:47:21] Unpausing evaluation: 122.20MiB in memory written to disk, 239.09MiB forgotten: 116.88MiB UNREACHABLE (152 items up to o+13270327) 122.20MiB VITAL (158 items up to o+13250479)
...

NOTE:
Another project execution time is much faster and the similar messages counts about 200 times.

Can we optimize the parameters to improve the balance between memory utilization, cache efficiency, and disk I/O operations?
or have any suggestions?

Thank you for your valuable suggestions and guidance. 😊

Ethan

[smowton]

Could you provide a log from the offending run so we can see what is being evaluated when we start to need to write temporary data to disk like this?

Meanwhile, could you try setting --ram to the number of megabytes you'd like CodeQL to top out at, to see if that helps?

[Eathonhsu]

I upload the CodeQL database analysis logs for reference.
codeql-analysis-log.zip

The process log: (ReturnStackAllocatedMemory and RedundantNullCheckParam, these two processes are exhibiting notably slow execution times)
[1/24 eval 15.2s] Evaluation done; writing results to codeql\cpp-queries\Best Practices\Unused Entities\UnusedStaticVariables.bqrs.
[2/24 eval 15.2s] Evaluation done; writing results to codeql\cpp-queries\Best Practices\Likely Errors\OffsetUseBeforeRangeCheck.bqrs.
[3/24 eval 15.2s] Evaluation done; writing results to codeql\cpp-queries\Likely Bugs\Arithmetic\ComparisonPrecedence.bqrs.
[4/24 eval 15.2s] Evaluation done; writing results to codeql\cpp-queries\Likely Bugs\ContinueInFalseLoop.bqrs.
[5/24 eval 15.3s] Evaluation done; writing results to codeql\cpp-queries\Likely Bugs\Arithmetic\UnsignedGEZero.bqrs.
[6/24 eval 15.3s] Evaluation done; writing results to codeql\cpp-queries\Best Practices\Unused Entities\UnusedLocals.bqrs.
[7/24 eval 15.5s] Evaluation done; writing results to codeql\cpp-queries\jsf\4.07 Header Files\AV Rule 35.bqrs.
[8/24 eval 16.1s] Evaluation done; writing results to codeql\cpp-queries\Likely Bugs\Conversion\ImplicitDowncastFromBitfield.bqrs.
[9/24 eval 1m23s] Evaluation done; writing results to codeql\cpp-queries\jsf\4.13 Functions\AV Rule 114.bqrs.
[10/24 eval 10m38s] Evaluation done; writing results to codeql\cpp-queries\Security\CWE\CWE-190\ComparisonWithWiderType.bqrs.
[11/24 eval 10m47s] Evaluation done; writing results to codeql\cpp-queries\Security\CWE\CWE-835\InfiniteLoopWithUnsatisfiableExitCondition.bqrs.
[12/24 eval 70m41s] Evaluation done; writing results to codeql\cpp-queries\Likely Bugs\Memory Management\ReturnStackAllocatedMemory.bqrs.
[13/24 eval 70m53s] Evaluation done; writing results to codeql\cpp-queries\Likely Bugs\Memory Management\PointerOverflow.bqrs.
[14/24 eval 70m53s] Evaluation done; writing results to codeql\cpp-queries\experimental\Best Practices\UselessTest.bqrs.
[15/24 eval 75m9s] Evaluation done; writing results to codeql\cpp-queries\Likely Bugs\RedundantNullCheckSimple.bqrs.
[16/24 eval 82m26s] Evaluation done; writing results to codeql\cpp-queries\Critical\OverflowStatic.bqrs.
[17/24 eval 82m26s] Evaluation done; writing results to codeql\cpp-queries\Likely Bugs\Conversion\LossyFunctionResultCast.bqrs.
[18/24 eval 82m26s] Evaluation done; writing results to codeql\cpp-queries\Critical\OverflowDestination.bqrs.
[19/24 eval 82m26s] Evaluation done; writing results to codeql\cpp-queries\Security\CWE\CWE-131\NoSpaceForZeroTerminator.bqrs.
[20/24 eval 82m26s] Evaluation done; writing results to codeql\cpp-queries\Critical\OverflowCalculated.bqrs.
[21/24 eval 82m26s] Evaluation done; writing results to codeql\cpp-queries\Likely Bugs\Likely Typos\UsingStrcpyAsBoolean.bqrs.
[22/24 eval 82m26s] Evaluation done; writing results to codeql\cpp-queries\Security\CWE\CWE-119\OverflowBuffer.bqrs.
[23/24 eval 82m26s] Evaluation done; writing results to codeql\cpp-queries\Security\CWE\CWE-120\VeryLikelyOverrunWrite.bqrs.
[24/24 eval 118m42s] Evaluation done; writing results to codeql\cpp-queries\experimental\Likely Bugs\RedundantNullCheckParam.bqrs.

execute-queries-20241227.064755.300.log
...
[2024-12-27 06:58:58] [PROGRESS] execute queries> [11/24 eval 10m47s] Evaluation done; writing results to codeql\cpp-queries\Security\CWE\CWE-835\InfiniteLoopWithUnsatisfiableExitCondition.bqrs.
[2024-12-27 06:58:58] Pausing evaluation to evict 238.87MiB ARRAYS at sequence stamp o+10871931
[2024-12-27 06:58:58] Unpausing evaluation: 69.62MiB in memory written to disk, 239.05MiB forgotten: 92.81MiB UNREACHABLE (121 items up to o+10871927) 2.37MiB UNIMPORTANT (12 items up to o+10871577) 143.87MiB VITAL (181 items up to o+10852707)
[2024-12-27 06:58:58] Pausing evaluation to evict 238.87MiB ARRAYS at sequence stamp o+10872287
[2024-12-27 06:58:58] Unpausing evaluation: 141.54MiB in memory written to disk, 239.00MiB forgotten: 72.70MiB UNREACHABLE (94 items up to o+10872284) 166.29MiB VITAL (216 items up to o+10853137)
[2024-12-27 06:58:59] Pausing evaluation to evict 238.87MiB ARRAYS at sequence stamp o+10872651
[2024-12-27 06:58:59] Unpausing evaluation: 125.30MiB in memory written to disk, 238.99MiB forgotten: 81.98MiB UNREACHABLE (106 items up to o+10872644) 157.01MiB VITAL (203 items up to o+10853564)
...
[2024-12-27 07:58:52] [PROGRESS] execute queries> [12/24 eval 70m41s] Evaluation done; writing results to codeql\cpp-queries\Likely Bugs\Memory Management\ReturnStackAllocatedMemory.bqrs.
...

About the parameters --ram I will try this, thanks.

[smowton]

Thanks, that's useful -- it shows that either your database has a very large codebase extracted (e.g., there are evidently a lot of different control-flow nodes that guard others, and a lot of variable-accesses, looking at two of the predicates that cause significant stress and return hundred-million-scale results). Are you able to share any details about the code you're analysing?

One thing that could be useful is subdividing your analysis into different projects. For example, often if a repository contains millions of lines of code, actually it can be subdivided into subsets which are interesting to security-analyse together, such as programs and their dependent libraries, vs. those which don't interact in this sense, e.g. a pair of unrelated programs neither of which calls the other. If this sounds like your use case, one route to optimising your analysis could be to prepare more fine-grained databases each of which is analysed individually.