Using pip-tools as an API?

[woodruffw]

Hi there! I'm reaching out as the lead engineer working on pip-audit, a tool that Google and Trail of Bits are developing to allow developers to audit their Python environments for packages with known vulnerabilities

One of our goals is to support a requirements-file mode, and pip-tools' support for resolving entire dependency trees and pinning versions seemed like a strong choice for that task.

What's the problem this feature will solve?

Currently, pip-tools exposes two CLI tools: pip-compile and pip-sync. These work great, but they're difficult to interact with directly via Python: in order to utilize pip-compile's dependency resolution, we'd need to create a subprocess, control where pip-compile puts its outputs, and then read those outputs back in.

It would be great if we could skip that subprocess step and import pip-tools directly as a Python module!

Describe the solution you'd like

Our ideal solution looks something like this rough sketch:

import pip_tools

resolved_dep_list = pip_tools.compile(some_requirements_string)

(Where some_requirements_string is a string in Requirements File Format and resolved_dep_list is something resembling a List[Tuple[str, Version]].

Alternative Solutions

As mentioned above: the next best thing for our use case is probably to use a subshell and hack together an environment wherein we're relatively confident that running pip-compile <some-input> won't cause any externally observable side effects. But we think that a real Python API would be a lot cleaner, and a net win for the packaging ecosystem!

Additional context

The work we're doing on pip-audit is currently funded, and we have budget set aside for making changes/improvements to the Python packaging tooling ecosystem. We're happy to use some of our engineering time here, if it would help!

[AndydeCleyre]

I think this ought to be considered related to #1377 (dependent on?), and I encourage you to help build a consensus on the ideal structured output object there.

---

In the hacks department, here are two examples with plumbum:

In [1]: from plumbum.cmd import pip_compile

In [2]: deps = (pip_compile['-', '-o', '-'] << "requests")()

In [3]: print(deps)
#
# This file is autogenerated by pip-compile with python 3.9
# To update, run:
#
#    pip-compile --output-file=- -
#
certifi==2021.5.30
    # via requests
charset-normalizer==2.0.6
    # via requests
idna==3.2
    # via requests
requests==2.26.0
    # via -r -
urllib3==1.26.6
    # via requests

In [1]: from plumbum import local

In [2]: pip_compile = lambda reqsin: [tuple(line.split('==')) for line in (local['pip-compile']['--no-annotate', '--no-header', '-', '-o', '-'] << reqsin)().splitlines()]

In [3]: deps = pip_compile("requests")

In [4]: print(deps)
[('certifi', '2021.5.30'), ('charset-normalizer', '2.0.6'), ('idna', '3.2'), ('requests', '2.26.0'), ('urllib3', '1.26.6')]

[woodruffw]

Thanks for following up! I'll go ahead and opine on #1377 as well, but FWIW: our ideal output for an API would be nearly identical to the one currently produced by the CLI; something to the effect of List[Requirement], where Requirement is or is similar to packaging.requirements.Requirement from the PyPA's packaging package.

[webknjaz]

our ideal output for an API would be nearly identical to the one currently produced by the CLI

What if this gets replaced by the lockfile proposal currently being discussed on dpo, that would fulfill the request, right?

[woodruffw]

What if this gets replaced by the lockfile proposal currently being discussed on dpo, that would fulfill the request, right?

Yes! A full standard lockfile format would fulfill this and make changes to pip-tools completely unnecessary 🙂