datalad

[quentinduche]

Après avoir déplacé ma local database d'un espace privé sur le NAS à un espace partagé sur le NAS
mv /data/share/users/qduche/VHD/cohortevhd/ /data/share/projects/vhd

Tout nouveau téléchargement aboutit sur l'erreur suivante, à l'étape datalad :

2024-12-03 10:30:37,525 [    INFO] [Node] Finished "convert", elapsed time 4.35778s.
Traceback (most recent call last):
  File "/home/qduche/Projects/VHD/download_cohort/./vhd_dl_datasets.py", line 180, in <module>
    main()
  File "/home/qduche/Projects/VHD/download_cohort/./vhd_dl_datasets.py", line 177, in main
    s2b.download()
  File "/home/qduche/Dev/shanoir_downloader/shanoir2bids.py", line 707, in download
    self.download_subject(subject_to_search=subject_to_search)
  File "/home/qduche/Dev/shanoir_downloader/shanoir2bids.py", line 679, in download_subject
    workflow(**workflow_params)
  File "/home/qduche/Projects/VHD/env/lib/python3.12/site-packages/heudiconv/main.py", line 512, in workflow
    add_to_datalad(outdir, study_outdir, msg, bids_options)
  File "/home/qduche/Projects/VHD/env/lib/python3.12/site-packages/heudiconv/external/dlad.py", line 116, in add_to_datalad
    ds.save(path=[".gitattributes"], message="Custom .gitattributes", to_git=True)
  File "/home/qduche/Projects/VHD/env/lib/python3.12/site-packages/datalad/distribution/dataset.py", line 502, in apply_func
    return f(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^
  File "/home/qduche/Projects/VHD/env/lib/python3.12/site-packages/datalad/interface/base.py", line 772, in eval_func
    return return_func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/qduche/Projects/VHD/env/lib/python3.12/site-packages/datalad/interface/base.py", line 762, in return_func
    results = list(results)
              ^^^^^^^^^^^^^
  File "/home/qduche/Projects/VHD/env/lib/python3.12/site-packages/datalad/interface/base.py", line 872, in _execute_command_
    for r in _process_results(
  File "/home/qduche/Projects/VHD/env/lib/python3.12/site-packages/datalad/interface/utils.py", line 319, in _process_results
    for res in results:
  File "/home/qduche/Projects/VHD/env/lib/python3.12/site-packages/datalad/core/local/save.py", line 222, in __call__
    for s in Status()(
  File "/home/qduche/Projects/VHD/env/lib/python3.12/site-packages/datalad/interface/base.py", line 872, in _execute_command_
    for r in _process_results(
  File "/home/qduche/Projects/VHD/env/lib/python3.12/site-packages/datalad/interface/utils.py", line 319, in _process_results
    for res in results:
  File "/home/qduche/Projects/VHD/env/lib/python3.12/site-packages/datalad/core/local/status.py", line 405, in __call__
    for r in yield_dataset_status(
  File "/home/qduche/Projects/VHD/env/lib/python3.12/site-packages/datalad/core/local/status.py", line 159, in yield_dataset_status
    status = repo.diffstatus(
             ^^^^^^^^^^^^^^^^
  File "/home/qduche/Projects/VHD/env/lib/python3.12/site-packages/datalad/support/gitrepo.py", line 3099, in diffstatus
    to_state = self.get_content_info(
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/qduche/Projects/VHD/env/lib/python3.12/site-packages/datalad/support/gitrepo.py", line 2866, in get_content_info
    submodules = [s["path"].relative_to(self.pathobj).as_posix()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/qduche/Projects/VHD/env/lib/python3.12/site-packages/datalad/support/gitrepo.py", line 2460, in get_submodules_
    for r in self.call_git_items_(
  File "/home/qduche/Projects/VHD/env/lib/python3.12/site-packages/datalad/dataset/gitrepo.py", line 524, in call_git_items_
    for file_no, line in self._generator_call_git(
  File "/home/qduche/Projects/VHD/env/lib/python3.12/site-packages/datalad/dataset/gitrepo.py", line 362, in _generator_call_git
    for file_no, content in generator:
  File "/home/qduche/Projects/VHD/env/lib/python3.12/site-packages/datalad/runner/gitrunner.py", line 305, in run_on_filelist_chunks_items_
    yield from chunk_generator
  File "<frozen _collections_abc>", line 356, in __next__
  File "/home/qduche/Projects/VHD/env/lib/python3.12/site-packages/datalad/runner/nonasyncrunner.py", line 99, in send
    return self._locked_send(message)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/qduche/Projects/VHD/env/lib/python3.12/site-packages/datalad/runner/nonasyncrunner.py", line 124, in _locked_send
    self._check_result()
  File "/home/qduche/Projects/VHD/env/lib/python3.12/site-packages/datalad/runner/nonasyncrunner.py", line 95, in _check_result
    self.runner._check_result()
  File "/home/qduche/Projects/VHD/env/lib/python3.12/site-packages/datalad/runner/nonasyncrunner.py", line 297, in _check_result
    raise CommandError(
datalad.runner.exception.CommandError: CommandError: 'git -c diff.ignoreSubmodules=none -c core.quotepath=false ls-files --stage -z -- .heudiconv' failed with exitcode 128

[alexpron]

Hello @quentinduche
thanks for taking the time to report.

Reproduction of the bug

1. Checkout on main branch (e1b0d65)
2. Downloaded datasets in the s2b_example_config.json of the repository (Aneravimm study) with default argument
   python shanoir2bids.py -u apron -j s2b_example_config.json -of /home/alpron/data/debug-mv-dataset
3. Moved (mv without options) the whole output directory elsewhere.
   mv /home/alpron/data/debug-mv-dataset /home/alpron/debug-mv-dataset
4. Download the same files into the moved datalad dataset
   python shanoir2bids.py -u apron -j s2b_example_config.json -of /home/alpron/debug-mv-dataset

[alexpron]

I have got an other error (not yours)

Traceback (most recent call last):
  File "/home/alpron/code/shanoir_downloader/shanoir2bids.py", line 840, in <module>
    main()
  File "/home/alpron/code/shanoir_downloader/shanoir2bids.py", line 836, in main
    stb.download()
  File "/home/alpron/code/shanoir_downloader/shanoir2bids.py", line 753, in download
    self.download_subject(subject_to_search=subject_to_search)
  File "/home/alpron/code/shanoir_downloader/shanoir2bids.py", line 724, in download_subject
    workflow(**workflow_params)
  File "/home/alpron/softs/miniconda3/envs/test-env/lib/python3.10/site-packages/heudiconv/main.py", line 479, in workflow
    prep_conversion(
  File "/home/alpron/softs/miniconda3/envs/test-env/lib/python3.10/site-packages/heudiconv/convert.py", line 259, in prep_conversion
    convert(
  File "/home/alpron/softs/miniconda3/envs/test-env/lib/python3.10/site-packages/heudiconv/convert.py", line 597, in convert
    convert_dicom(
  File "/home/alpron/softs/miniconda3/envs/test-env/lib/python3.10/site-packages/heudiconv/convert.py", line 752, in convert_dicom
    compress_dicoms(
  File "/home/alpron/softs/miniconda3/envs/test-env/lib/python3.10/site-packages/heudiconv/dicoms.py", line 582, in compress_dicoms
    dcm_time = get_reproducible_int(dicom_list)
  File "/home/alpron/softs/miniconda3/envs/test-env/lib/python3.10/site-packages/heudiconv/dicoms.py", line 492, in get_reproducible_int
    dicom = dcm.dcmread(dicom_list[0], stop_before_pixels=True, force=True)
  File "/home/alpron/softs/miniconda3/envs/test-env/lib/python3.10/site-packages/pydicom/filereader.py", line 1002, in dcmread
    fp = open(fp, 'rb')
FileNotFoundError: [Errno 2] No such file or directory: '/home/alpron/data/debug-mv-dataset/tmp_dicoms/VS_Aneravimm_010/391710/VS_Aneravimm_010_20210512_t2_tse_HR_cor_MTL_1.4.9.12.34.1.8527.1158820958174222669888838509284917382343.dcm'

This seems to be due to https://heudiconv.readthedocs.io/en/latest/custom-heuristic.html#the-heudiconv-hidden-directory (the mapping used is the one inside .heudiconv/subject file and in this case the directory should be removed
but this imply that configuration file used contain all the files inside otherwise mapping will be erased (replaced by incomplete mapping)

[quentinduche]

I've got the same error when performing step 4.

[alexpron]

After further investigation:

at each subject/session BIDS conversion using heudiconv , heudiconv generate files to parameterize the conversion

• heuristic.py (scripted conversion)
• auto.txt
• edit.txt (final mapping between DICOM Series ID and BIDS key)
• filegroup.json (DICOM Series to DICOM instances) where DICOM path to instance are absolute.

From my understanding pre conversion step

• presence of the edit.txt is checked
• difference on the heuristic.py files (stored and in generation are checked (MD5)
  If no differences are found edit.txt and filegroup.json are used

When moving the datalad dataset using mv the filegroup.json is not updated properly.

Note that in our configuration it should not happen as I passed the overwrite argument
(files are regenerated, an other option would be not to overwrite and catch the exception thrown.

I will open an issue on heudiconv as as far as I understand the overwrite argument of the prep_conversion function
is not taken into account

Other solutions:

• delete .heudiconv dir and regenerate it (dirty and computation for nothing)
• write a script to properly move such dataset by recomputing path inside filegroup.json accordingly

[alexpron]

Here is the opened issue on heudiconv repository nipy/heudiconv#807

[alexpron]

Regarding the initial error the command line introducing the error is :
git -c diff.ignoreSubmodules=none -c core.quotepath=false ls-files --stage -z -- .heudiconv failed with exitcode 128
This command list files contained inside the .heudiconv directory. It may be due to broken git repo (git status to check) or permission error (more likely) (either locked files or mixed owner files)