Add new source hashing methods: content_sha256, content_sha384, content_sha512

[jaimergp]

Description

• Closes Better hashing of sources #4762.
• Relevant CEP submitted at Standardize algorithm for directory hashing ceps#100.

Checklist - did you ...

• Add a file to the news directory (using the template) for the next release's release notes?
• Add / update necessary tests?
• Add / update outdated documentation?

[codspeed-hq]

CodSpeed Performance Report

Merging #5277 will not alter performance

_{Comparing jaimergp:content-hash (33eb089) with main (b8724c6)}

Summary

✅ 5 untouched benchmarks

[wolfv]

I think this is cool. It would also work nicely with the new proposal for "rendered recipes" (conda/ceps#74).

On that note - should we continue adding features to conda-build without any standardization (e.g. CEP) process?

[jaimergp]

should we continue adding features to conda-build without any standardization (e.g. CEP) process?

I'm planning to submit a CEP. I opened this draft to explore what kind of things are needed for a stable yet robust logic, cross platform. Things like permissions and so on don't translate well to Windows.

[wolfv]

Awesome. Yeah, I also recently looked at a few content hash implementations in Rust but didn't find anything super convincing yet. There are a bunch though (https://crates.io/search?q=content%20hash)

[jaimergp]

So far the scheme I followed looks a lot like https://github.com/DrSLDR/dasher?tab=readme-ov-file#hashing-scheme. Things to standardize would be how the tree is sorted, the normalization of the path, the separators (to prevent this), and the allowed algorithms.

I've seen a few merkle tree based packages but we don't need all the proof stuff, or leaf querying; just comparing the root hash.

Maybe it could be implemented in a recursive way that doesn't involve obtaining the whole file tree beforehand if that increases performance or simplifies implementation elsewhere. IMO this feels like one of those CEPs that does require prototyping first to see which things have to be standardized.

[jaimergp]

pre-commit.ci autofix

[jaimergp]

I would add a CLI option in rattler-build that would look something like [...]

Hm, true, we could add a little subcommand here to make it easier. Although honestly, I usually run conda-build, wait for the hashing mismatch error, and then copy the correct one 😬 (That's why I amended the text in the errors hah).

[schuylermartin45]

I don't think it is wise to add support for two hashing algorithms with known vulnerabilities in 2024. Although it may be unlikely, that smells like an avenue for a supply chain attack to me.

I think if we were to support multiple hashing algorithms, we should support algorithms that are still deemed viable and secure, like the other SHA-* bit lengths.

[jaimergp]

#4793 will probably land soon. I'll update this branch with it once it reaches main and then remove the weak hashes. If we are that concerned about md5 and sha1 being used, we should also study the possibility of deprecating them or at least warning about them in the logs.

[beckermr]

We'll need a lot of advanced notice to deprecate them on conda-forge. For now we should probably add a lint + minimigrator to move to sha256.

[wolfv]

I am not concerned, I just don't understand why we want to add them? IMO it doesn't add value. Having the MD5 hash available for the regular hash makes sense because some pacakges might publish the known-good value (and that can be used in the recipe), but for something that we have invented ourselves I don't see a use-case where it is justified to use anything other than the best available hash.

[beckermr]

LGTM! We should probably vote on the CEP before merging.

[jezdez]

Setting as blocked on the CEP vote

[jaimergp]

@jezdez, the CEP passed, so I guess this review can be dismissed now.

[beckermr]

Some comments.

[jaimergp]

CI errors are unrelated. Investigating in #5577.