Connecting subworkflows with DSL2 #1918
Unanswered
RenzoTale88
asked this question in
Q&A
Replies: 5 comments
-
|
Hi
This caught me out too!
I think the solution is that when calling output from your subworkflows you have to reference them by index number and not by the names you gave them in the emit statements in the subworkflows.
Your main workflow should then be:
include {TWOBIT; SIZES} from './modules/subworkflows/preprocess' params(params)
workflow {
main:
TWOBIT()
TBS = TWOBIT.out.[0]
TBT = TWOBIT.out.[1]
SIZES(TBS, TBT)
}
Hope this helps
Bob
|
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Hi @bobamess just gave it a go, now the workflow looks like yours: But ont it complains about something different: This is again probably trivial, but I came across this error and can't find any documentation on how to fix it. Andrea |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Also, if I add only one of the workflows, as here: It works fine. But when I add the second stage: Then the problem occurs: |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Oops! I was a bit too quick there!
It should have been:
include {TWOBIT; SIZES} from './modules/subworkflows/preprocess' params(params)
workflow {
main:
TWOBIT()
TBS = TWOBIT.out[0]
TBT = TWOBIT.out[1]
SIZES(TBS, TBT)
}
i.e. without the “.” Before the indexed output [0] and [1].
From: Robert Amess
Sent: 16 November 2020 18:04
To: nextflow-io/nextflow <[email protected]>
Subject: RE: [nextflow-io/nextflow] Connecting subworkflows with DSL2 (#1794)
Hi
This caught me out too!
I think the solution is that when calling output from your subworkflows you have to reference them by index number and not by the names you gave them in the emit statements in the subworkflows.
Your main workflow should then be:
include {TWOBIT; SIZES} from './modules/subworkflows/preprocess' params(params)
workflow {
main:
TWOBIT()
TBS = TWOBIT.out.[0]
TBT = TWOBIT.out.[1]
SIZES(TBS, TBT)
}
Hope this helps
Bob
From: RenzoTale88 <[email protected]<mailto:[email protected]>>
Sent: 16 November 2020 17:08
To: nextflow-io/nextflow <[email protected]<mailto:[email protected]>>
Cc: Subscribed <[email protected]<mailto:[email protected]>>
Subject: [nextflow-io/nextflow] Connecting subworkflows with DSL2 (#1794)
Hello,
I've been trying to learn how to migrate my workflows from nextflow DSL1 to DSL2, and I got a question concerning the use of sub-workflows.
I'm trying to migrate a small workflow that convert two genomes in 2bit format, and then generates the sizes files from those. Each of the two is a separate small workflow (creation of 2bit, followed by generation of the sizes), and the output of the first should be passed to the second.
I've created a structure of folders as follow:
./
./modules/
./modules/subworkflow
./modules/processes
./main.nf
With the subworkflow contiaining the sub-workflow in .nf format and processes containing the processed that can be included by the different functions.
I got a main workflow as follow:
include {TWOBIT; SIZES} from './modules/subworkflows/preprocess' params(params)
workflow {
main:
TWOBIT()
TBS = TWOBIT.out.twoBitS
TBT = TWOBIT.out.twoBitT
SIZES(TBS, TBT)
}
This is calling these two sub-workflows, that look as follow:
include {make2bitS; make2bitT; makeSizeS; makeSizeT} from "../processes/preprocess" params(params)
workflow TWOBIT {
main:
// Make 2bit genomes
make2bitS()
make2bitT()
emit:
twoBitS = make2bitS.out.twoBsrc
twoBitT = make2bitT.out.twoBtgt
}
workflow SIZES {
take:
2bitS
2bitT
main:
makeSizeS(2bitS)
makeSizeT(2bitT)
emit:
twoBitS = makeSizeS.out.sizesSrc
twoBitT = makeSizeT.out.sizesTgt
}
These in turn call the following functions:
process make2bitS {
tag "twoBit"
publishDir "$params.outdir/genome2bit"
output:
path "source.2bit", emit: twoBsrc
script:
"""
faToTwoBit ${params.source} source.2bit
"""
}
process makeSizeS {
tag "twoBit"
publishDir "$params.outdir/genome2bit"
input:
path src
output:
path "source.sizes", emit: sizesSrc
script:
"""
twoBitInfo ${src} source.sizes
"""
}
process make2bitT {
tag "twoBit"
publishDir "$params.outdir/genome2bit"
output:
path "target.2bit", emit: twoBtgt
script:
"""
faToTwoBit ${params.target} target.2bit
"""
}
process makeSizeT {
tag "twoBit"
publishDir "$params.outdir/genome2bit"
input:
path tgt
output:
path "target.sizes", emit: sizesTgt
script:
"""
twoBitInfo ${tgt} target.sizes
"""
}
However, when I run the pipeline, I get the following error:
Module compilation error
- file : ./modules/subworkflows/preprocess.nf
- cause: Workflow malformed parameter definition @ line 32, column 9.
2bitS
^
Module compilation error
- file : ./modules/subworkflows/preprocess.nf
- cause: Workflow malformed parameter definition @ line 33, column 9.
2bitT
^
2 errors
Sorry, it's probably a very simple mistake, but I'm not sure to understand what it is it doesn't like.
Thank you in advance
Andrea
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub<#1794>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ANZBGH3U522MGGNANPYHKJDSQFL75ANCNFSM4TXN4RYA>.
|
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Hi,
Further to this, here’s a simplified workflow.
The main points are:
1) When there is only one output from a process there is no need to name it with the emit option as you can just use <process_name>.out
2) Unless you are modifying the output from a process, or workflow, within a workflow there is no need to name it before including it in the emit statement of your workflow since you can reference it directly, as in the lines below from my simplified version of your workflow:
emit:
make2bitS.out; make2bitT.out
Simplified workflow:
include {TWOBIT; SIZES} from './modules/subworkflows/preprocess' params(params)
workflow {
main:
TWOBIT()
SIZES(TWOBIT.out[0], TWOBIT.out[1])
}
//This is calling these two sub-workflows, that look as follow:
include {make2bitS; make2bitT; makeSizeS; makeSizeT} from "../processes/preprocess" params(params)
workflow TWOBIT {
main:
// Make 2bit genomes
make2bitS()
make2bitT()
emit:
make2bitS.out; make2bitT.out
}
workflow SIZES {
take:
2bitS
2bitT
main:
makeSizeS(2bitS)
makeSizeT(2bitT)
emit:
makeSizeS.out; makeSizeT.out
}
//These in turn call the following functions:
process make2bitS {
tag "twoBit"
publishDir "$params.outdir/genome2bit"
output:
path "source.2bit"
script:
"""
faToTwoBit ${params.source} source.2bit
"""
}
process makeSizeS {
tag "twoBit"
publishDir "$params.outdir/genome2bit"
input:
path src
output:
path "source.sizes"
script:
"""
twoBitInfo ${src} source.sizes
"""
}
process make2bitT {
tag "twoBit"
publishDir "$params.outdir/genome2bit"
output:
path "target.2bit"
script:
"""
faToTwoBit ${params.target} target.2bit
"""
}
process makeSizeT {
tag "twoBit"
publishDir "$params.outdir/genome2bit"
input:
path tgt
output:
path "target.sizes"
script:
"""
twoBitInfo ${tgt} target.sizes
"""
}
From: Robert Amess
Sent: 17 November 2020 12:20
To: nextflow-io/nextflow <[email protected]>
Subject: RE: [nextflow-io/nextflow] Connecting subworkflows with DSL2 (#1794)
Oops! I was a bit too quick there!
It should have been:
include {TWOBIT; SIZES} from './modules/subworkflows/preprocess' params(params)
workflow {
main:
TWOBIT()
TBS = TWOBIT.out[0]
TBT = TWOBIT.out[1]
SIZES(TBS, TBT)
}
i.e. without the “.” Before the indexed output [0] and [1].
From: Robert Amess
Sent: 16 November 2020 18:04
To: nextflow-io/nextflow <[email protected]<mailto:[email protected]>>
Subject: RE: [nextflow-io/nextflow] Connecting subworkflows with DSL2 (#1794)
Hi
This caught me out too!
I think the solution is that when calling output from your subworkflows you have to reference them by index number and not by the names you gave them in the emit statements in the subworkflows.
Your main workflow should then be:
include {TWOBIT; SIZES} from './modules/subworkflows/preprocess' params(params)
workflow {
main:
TWOBIT()
TBS = TWOBIT.out.[0]
TBT = TWOBIT.out.[1]
SIZES(TBS, TBT)
}
Hope this helps
Bob
From: RenzoTale88 <[email protected]<mailto:[email protected]>>
Sent: 16 November 2020 17:08
To: nextflow-io/nextflow <[email protected]<mailto:[email protected]>>
Cc: Subscribed <[email protected]<mailto:[email protected]>>
Subject: [nextflow-io/nextflow] Connecting subworkflows with DSL2 (#1794)
Hello,
I've been trying to learn how to migrate my workflows from nextflow DSL1 to DSL2, and I got a question concerning the use of sub-workflows.
I'm trying to migrate a small workflow that convert two genomes in 2bit format, and then generates the sizes files from those. Each of the two is a separate small workflow (creation of 2bit, followed by generation of the sizes), and the output of the first should be passed to the second.
I've created a structure of folders as follow:
./
./modules/
./modules/subworkflow
./modules/processes
./main.nf
With the subworkflow contiaining the sub-workflow in .nf format and processes containing the processed that can be included by the different functions.
I got a main workflow as follow:
include {TWOBIT; SIZES} from './modules/subworkflows/preprocess' params(params)
workflow {
main:
TWOBIT()
TBS = TWOBIT.out.twoBitS
TBT = TWOBIT.out.twoBitT
SIZES(TBS, TBT)
}
This is calling these two sub-workflows, that look as follow:
include {make2bitS; make2bitT; makeSizeS; makeSizeT} from "../processes/preprocess" params(params)
workflow TWOBIT {
main:
// Make 2bit genomes
make2bitS()
make2bitT()
emit:
twoBitS = make2bitS.out.twoBsrc
twoBitT = make2bitT.out.twoBtgt
}
workflow SIZES {
take:
2bitS
2bitT
main:
makeSizeS(2bitS)
makeSizeT(2bitT)
emit:
twoBitS = makeSizeS.out.sizesSrc
twoBitT = makeSizeT.out.sizesTgt
}
These in turn call the following functions:
process make2bitS {
tag "twoBit"
publishDir "$params.outdir/genome2bit"
output:
path "source.2bit", emit: twoBsrc
script:
"""
faToTwoBit ${params.source} source.2bit
"""
}
process makeSizeS {
tag "twoBit"
publishDir "$params.outdir/genome2bit"
input:
path src
output:
path "source.sizes", emit: sizesSrc
script:
"""
twoBitInfo ${src} source.sizes
"""
}
process make2bitT {
tag "twoBit"
publishDir "$params.outdir/genome2bit"
output:
path "target.2bit", emit: twoBtgt
script:
"""
faToTwoBit ${params.target} target.2bit
"""
}
process makeSizeT {
tag "twoBit"
publishDir "$params.outdir/genome2bit"
input:
path tgt
output:
path "target.sizes", emit: sizesTgt
script:
"""
twoBitInfo ${tgt} target.sizes
"""
}
However, when I run the pipeline, I get the following error:
Module compilation error
- file : ./modules/subworkflows/preprocess.nf
- cause: Workflow malformed parameter definition @ line 32, column 9.
2bitS
^
Module compilation error
- file : ./modules/subworkflows/preprocess.nf
- cause: Workflow malformed parameter definition @ line 33, column 9.
2bitT
^
2 errors
Sorry, it's probably a very simple mistake, but I'm not sure to understand what it is it doesn't like.
Thank you in advance
Andrea
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub<#1794>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ANZBGH3U522MGGNANPYHKJDSQFL75ANCNFSM4TXN4RYA>.
|
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Hello,
I've been trying to learn how to migrate my workflows from nextflow DSL1 to DSL2, and I got a question concerning the use of sub-workflows.
I'm trying to migrate a small workflow that convert two genomes in 2bit format, and then generates the sizes files from those. Each of the two is a separate small workflow (creation of 2bit, followed by generation of the sizes), and the output of the first should be passed to the second.
I've created a structure of folders as follow:
With the
subworkflowcontiaining the sub-workflow in .nf format andprocessescontaining the processed that can be included by the different functions.I got a main workflow as follow:
This is calling these two sub-workflows, that look as follow:
These in turn call the following functions:
However, when I run the pipeline, I get the following error:
Sorry, it's probably a very simple mistake, but I'm not sure to understand what it is it doesn't like.
Thank you in advance
Andrea
Beta Was this translation helpful? Give feedback.
All reactions