Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Investigate if worth staging/caching selected files #7

Open
eford opened this issue Jan 6, 2022 · 0 comments
Open

Investigate if worth staging/caching selected files #7

eford opened this issue Jan 6, 2022 · 0 comments
Assignees
Labels
optimization Performance optimization

Comments

@eford
Copy link
Member

eford commented Jan 6, 2022

Looking into possibility of caching some files to a local disk and/or a RAM disk.
(Chuck also mentioned something about Optane memory, but I don't know if that would be a good fit.)
The obvious candidates are:

  • calc_ccfs: reads the day's L2 files, particularly once we start processing them with multiple configurations in one pass (rather than as separate jobs).
  • calc_rvs: Currently reads the daily_ccfs_?.jld2 files. (In the future, we hope to have an alternative algorithm that would need access to the L2 files, rather than the daily_ccfs_?.jld2 files.)
@eford eford added the optimization Performance optimization label Jan 6, 2022
@eford eford assigned dshao and unassigned nucci6 Jun 2, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
optimization Performance optimization
Projects
None yet
Development

No branches or pull requests

3 participants