using aws fargate with wave - CannotPullContainerError

[itamary1]

Bug report

Expected behavior and actual behavior

I am using nextflow fargate for batch analysis, I have a pipeline with four steps, and I run it on about 30 samples (a total of 120 jobs)..
The jobs use public docker images from dockerhub and wave(which is mandatory for aws fargate).

I am getting randomly the error(see Program output):
error: nextflow.exception.ProcessException: CannotPullContainerError:

I thought it is related to dockerhub usage limit so I tried to use tower token with dockerhub credentials and it still happen.
I also checked that my personal dockerhub user didnt reach the pull limitation and got:
ratelimit-limit: 200;w=21600
ratelimit-remaining: 196;w=21600
(I have 196 pulls remain for next 6 hours, also its look like it nextflow didn't use my dockerhub credentials because I ran 100 jobs)

I run the pipeline in the past several times (even one time on 15,000 samples) and everything was ok with fargate+wave.

Thanks for your help!

Steps to reproduce the problem

run multiple jobs on aws fargate.

Program output

its look like this:
Jan-19 10:17:10.904 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 22; name: cmpileup (cmpileup SRR7660770); status: COMPLETED; exit: -; error: nextflow.exception.ProcessException: CannotPullContainerError: ref pull has been retried 1 time(s): failed to copy: httpReadSeeker: failed open: unexpected status code https://wave.seqera.io/v2/wt/3deb0fa443bf/levanonlab/cmpileup/blobs/sha256:1d71e290059824641352aefb09300471c4c331a29bf5f107cba46eb13a4584cb: 400 Bad Request; workDir: s3://levanon-lab-sambuseq-challenge-common/nextflow_workdir/7_SE_Stranded_75_35S/2b/e65acdc960560d9a6f4681564514d7]

(I upload the .nextflow.log file, and replace my towerAccessToken to @@@ for privacy.)

.nextflow.log

Environment

• Nextflow version: 24.11.0-edge.5929
• Java version: 17.0.10
• Operating system: Linux
• Bash version: 5.2.21

[pditommaso]

Does this problem happen consistently even after you specified your own credentials?