Are the contents of a task work directory worth adding to the task level logs?

[abhi18av]

Hi everyone,

I'm wondering whether it's worth adding a .directory.tree (or a better named file) which contains the output of tree command when the task exits.

Often times while debugging, we need to inspect the contents of the directory and having a dedicated file would help automate the process. We could add a separate Directory contents tab in Tower for every process window as well.

Please let me know if this proposal makes sense or not.

Best,
Abhinav

[pditommaso]

That could be useful BUT, lot of users are already concerned that NF creates too many hidden files (see https://docs.nersc.gov/jobs/workflow/nextflow/). Also, tree is not a universally available tool => -1

[abhi18av]

Hmm, that's an interesting thought. Howerver in the end these files are there to help debug more effectively.

1. For the first point, maybe we can add this to a current file or append to .command.log when the task exits.
2. True - for this we could include a util function and rely on a native java implementation of the unix tree command.

[KevinSayers]

Depending on the filesystem too, using a non-filesystem specific method to list the files could be very detrimental for the filesystem performance with a large number of files or processes.

[pditommaso]

Point 1, could be double but 2 it's not possible because the job script is autonomous, only depends on the underlying system tool. NF cannot helps. Considering also other comments, it looks there's consensus to not to do it.

[abhi18av]

Yeah, I also feel that this might not be as useful for other's as I imagined earlier 😅 .

[ewels]

Honestly - it's not a command that I find myself running manually very often with debugging.. I can see that it could be nice to see the file contents of the work directory in Tower directly, but I'd personally put it as a fairly low priority feature.

[abhi18av]

@pditommaso , just out of curiosity - I wasn't able to locate the relevant code. Could you please point out the code which is concerned with the generation of these dotfiles in the task directory?

[pditommaso]

Have fun https://github.com/nextflow-io/nextflow/blob/66f36c769827bf0a45a37a06878c215be6dc140c/modules/nextflow/src/main/groovy/nextflow/executor/BashWrapperBuilder.groovy

[abhi18av]

Actually, I just realized that this could also be resolved using a design pattern. For example consider a process with the following afterScript

process listDirContents {

  afterScript "tree -ahugf > .directorytree"

  """
  touch random_file.txt
  """

}

And we obtain a .directorytree with the following contents

.
├── [eklavya  staff       0]  ./.command.begin
├── [eklavya  staff     176]  ./.command.err
├── [eklavya  staff    6.2K]  ./.command.log
├── [eklavya  staff       0]  ./.command.out
├── [eklavya  staff    2.6K]  ./.command.run
├── [eklavya  staff      39]  ./.command.sh
├── [eklavya  staff     176]  ./.directorytree
└── [eklavya  staff       0]  ./random_file.txt

0 directories, 7 files

So at this point my personal use-case is satisfied ✅

Something that I'm musing about, from the point of view of Tower, I think it might be possible to inject this afterScript to all processes using processSelectors (stored in a base .nextflow.config) while launching a pipeline from Tower.

This way when the user's config is merged with global $HOME/.nextflow/config during config resolution - we might have successfully injected this to all processes by default.

[pditommaso]

Indeed. Put that into the nextflow.config and you have done. 👍