Support multiple deployment YAML files with shared definitions for better maintainability

[hamzamogni]

Describe the current behavior

The --prefect-file flag when running prefect deploy only accepts a single prefect.yaml file, this leads to maintenance issues when having big number of deployments: prefect.yaml could become a big file that is hard to maintain and navigate.

When dealing with large number of deployments, all using shared definitions, a good pattern would be to split deployments in different yaml files, then perform a preprocessing step to compile them into a single prefect.yaml before running prefect deploy.

Describe the proposed behavior

This enhancement is to give an option to automatically merge deployment yaml files, along with global definitions when running prefect deploy. This could be possible by making it possible to define a global-definitions.yaml file, and put deployment yaml files in a directory.

Example Use

this could be used in the CLI as in prefect deploy --global-definitions global.yaml --deployments-directory deployments

Additional context

No response

[zzstoatzz]

this issue came from a thread in slack

[OP] Hello @Nate, yes we are creating multiple flows for same deployments, i was only curious how they manage it using that approach (docstrings).
yes I tried using --prefect-file but my understanding is that it only takes one file path as an argument, meaning there would be a need for a preprocessing step to merge definitions from separate yaml files into a single one and then feed the output to --prefect-file.

[nate] aha yes you're correct if deployments in all files would depend on the same definitions that would indeed be required
alternatively you can use the python interface if you want to employ a modular approach to apply common deployment config and organize it however you can organize python code
👍
1

[nate] it feels like there could also be a reasonable enhancement request related to "global definitions" or something like that that could be referenced in any deployment related to a given server / workspace (regardless of file)