Introducing WorkFlow Package Manager (WFPM) CLI

[junjun-zhang]

Hi Nextflow community members,

Following the suggestions by @pditommaso and @abhi18av, I am glad to use this Show and tell section to share with you all a tool we recently developed to support workflow code reuse via shareable packages. The tool is called WorkFlow Package Manager (WFPM) CLI. Of course, it's open source: https://github.com/icgc-argo/wfpm. Feel good to have something share back to the open source community. Needless to say, without the awesome Nextflow framework, nothing about WFPM would be possible.

As a key concept in WFPM, a package is the smallest unit of shareable workflow code bundle. Concretely, a package can include one or more of these items: process, function or workflow. Each package must contain a pkg.json file to keep basic information, such as package name, version, main entry point, source code repository, container registry, dependencies etc.

The WFPM CLI development aims to make it as easy as possible to collaboratively develop ICGC ARGO bioinformatics workflows that conform to ARGO's established best practice guidelines: reproducible, portable, composable, findable and testable.

Noticeable features supported by WFPM CLI:

• auto-generated templates including workflow code and testing code
• seamless integration with GitHub Actions for automated CI/CD (support for other platform like GitLab is under consideration)
• everything developed using WFPM CLI is a package
• every package is independently developed and published with strict release versioning
• every package must include tests that will be automatically invoked by CI/CD processes
• published package is immutable and permanently avaialble
• anyone can write and publish a package
• a published package can be imported by anyone into her own workflow code base
• package import must explicitly specify a particular version to be imported to ensure reproducibility, this is similar to the idea of the package-lock.json file in NPM to ensure reproducible builds
• a package can import other packages (which are also known as dependencies)
• the above feature enables the support for subworkflows, ie, importing a published workflow as a subworkflow to build larger workflows

Code reuse in many general purpose programming languages is natively supported and often taken for granted, however it's not easy to do in workflow development. For example, you can do import yaml in Python to bring in all the great functionalities supported by the package which you rarely know or need to know who developed it. With features offered by WFPM, similar code import in workflow development becomes possible and easy to do using reusable and shareable packages.

The WFPM CLI development is at early stage, but we feel it has sufficient features for users of the amazing Nextflow community to try it out. As we have been mainly focusing on ARGO's primary needs, undoubtedly, there are weak areas we need to improvement on. We'd love to hear feedback and suggestions from you, what you like or dislike, all is greatly appreciated!

[pditommaso]

Hi Junjun, this looks great!

Naive question does it implements also a registry or modules are Github projects ?

[junjun-zhang]

@pditommaso this is a great question, I certainly didn't make it clear. It is the latter, all packages are released at its source code management system, at this moment github. Centralized package registry could be a thing in the future.