editable requirements are not allowed as constraints with the new backtracking resolver

[mmerickel]

I'm using a pattern of 3 requirement files:

• global constraints (base.txt)
• setup
• src
• dev

Setup includes a local package that is used by later src packages as a custom setuptools build class.

Since things are ordered, we use the pins from the previous file as constraints for the next file to ensure that all the files are pinned to the same versions. Thus:

# setup.in
-c base.txt
-e file:src/my-setup-package#egg=my-setup-package

# src.in
-c base.txt
-c setup.txt  # <------------ note this is the output pin from setup.in
-e file:src/my-web-app#egg=my-web-app

With the pip-tools workflow we then update each .txt file in order:

$ pip-compile --find-links wheels --no-emit-find-links --resolver=backtracking setup.in
$ pip-compile --find-links wheels --no-emit-find-links --resolver=backtracking src.in
$ pip-compile --find-links wheels --no-emit-find-links --resolver=backtracking dev.in

This workflow only fails when using --resolver=backtracking.

Tested with pip-tools 6.11.0.

[atugushev]

Constraints files are requirements files that only control which version of a requirement is installed, not whether it is installed or not. Their syntax and contents is a subset of Requirements Files, with several kinds of syntax not allowed: constraints must have a name, ——> they cannot be editable <—— , and they cannot specify extras

https://pip.pypa.io/en/stable/user_guide/#constraints-files

New resolver has more strict syntax in constraints files and does not support editables there since pip-20.3.

[mmerickel]

Is there any suggested workaround here to solve the goals of:

• ensure all setup/source/dev dependencies are pinned to the same version
• editable requirements for locally developed packages

[mmerickel]

Ok but I mean ... constructively, the only difference from what I'm doing and the examples in the README for pip-tools is that I'm using editable installs - a super common thing to do.

It seems I have to do something like pip-compile a file, then post-process it to remove any editable requirements so that I can use that file as constraints for the next pip-compile?

pip-compile setup.in
grep -v '^-e file:'  setup.txt > setup.constraints.txt
pip-compile src.in
grep -v '^-e file:'  src.txt > src.constraints.txt
pip-compile dev.in

With the following .in files:

# setup.in
-c base.txt
-e file:src/my-setuptools-extension#egg=my-setuptools-extension

# src.in
-c base.txt
-c setup.constraints.txt
-e file:src/my-package#egg=my-package

# dev.in
-c base.txt
-c setup.constraints.txt
-c src.constraints.txt
pip-tools
flake8
black

Is there a better way? Based on the pip docs, I'm also screwed if I use things like extras, ignoring the editable installs. I would need to parse and rewrite the dependencies to remove extras. It indicates that this pattern itself is just "wrong" because they don't anticipate this type of cascading constraints idea. But again, this idea is in the pip-tools README so I just wonder if there is a plan on how to deal with this. I know I'm coming in hot/desperate but pip-tools is the closest thing I have to a "python workspace manager" and it's stressful keeping it working lol.

[mmerickel]

I see there is also pip-compile --strip-extras indicating I could run pip-compile multiple times, I guess, alongside my grep:

pip-compile --strip-extras setup.in -o setup.constraints.tmp
grep -v '^-e file:'  setup.constraints.tmp > setup.constraints.txt
rm -f setup.constraints.tmp
pip-compile --strip-extras src.in -o src.constraints.tmp
grep -v '^-e file:'  src.constraints.tmp > src.constraints.txt
rm -f src.constraints.tmp

pip-compile setup.in
pip-compile src.in
pip-compile dev.in

[atugushev]

A few options as a workaround I could think of:

• post-processing by stripping editables from constrain files
• pip-compile --unsafe-package my-setup-package to exclude editable package from the output

[alflanagan]

At a very minimum, the README section Workflow for Layered Requirements needs to be updated with new advice (I'm not sure what the advice should be). Users following it who hit this error are going to have no idea what to do.

[carlosabalde]

I'm one of those users following the requirements.in (base dependencies) + dev-requirements.in (additional dependencies only useful in dev / CI envs) pattern suggested in the README and I'm not sure what to do in order to use --resolver=backtracking when compiling dev-requirements.in. What's the advise here?

[carlosabalde]

After some more research, specially after checking #1613, is now clear that the trick is always using --strip-extras in order to keep --resolver=backtracking happy. Extras are not relevant in the output file and that way I can continue using -c requirements.txt in dev-requirements.in.

[webknjaz]

Why don't you pass setup.cfg/pyproject.toml as a CLI argument instead of having it in the in-file? Also, note that you don't need to store the post-processed file on disk, instead — feed it to stdin, if you go this route.

[carlosabalde]

I guess it depends on the scenario. In my particular scenario I don't want to use setup.cfg or pyproject.toml files because this software is not installed using pip. I have *.in files, I have full control of the production env, and I need to keep the *.txt compiled versions in Git in order to have fully reproducible deployments. In any case, I think that's a different discussion. The point here is --strip-extras should be used together with --resolver=backtracking in order to create constraint-compatible requirement files.

[scottjmaddox]

Extras are not relevant in the output file and that way I can continue using -c requirements.txt in dev-requirements.in.

Why are extras not relevant in the output file? If requirements.txt is used to create the production environment, isn't it critical that it specifies any needed extras?

I'm running into this same issue, and trying to figure out how to deal with it.

[carlosabalde]

Let's assume in requirements.in you include celery[redis]==x.y.z as a dependency. When requirements.txt is generated, having celery[redis]==x.y.z in there (instead of just celery==x.y.z when using --strip-extras) is not relevant because when requirements.txt is generated the optional component [redis] is considered and included in requirements.txt (as additional dependencies like redis==..., async-timeout==..., etc.). Do you have an example contradicting this?

[webknjaz]

because this software is not installed using pip. I have *.in files

Sounds like you should be waiting for PEP 735 to land. What you describe is essentially "dependency groups". And I totally understand the importance.

Why are extras not relevant in the output file? If requirements.txt is used to create the production environment, isn't it critical that it specifies any needed extras?

I'm running into this same issue, and trying to figure out how to deal with it.

Because what you pass to -c in the pip install command must be compliant with the pip constraints file format, while things passed to -r may contain even more syntax. They are related but totally different concepts. You can separate your direct requirements from a tree of transitive deps that are pinned exactly. I often offer people to treat constraints as lockfiles (they can be used as such to certain extend). Abusing constraints as requirements may often be problematic and I strongly advise against doing that: #2051 (comment).