chore(deps): update dependency llvm_zstd to v1.5.6 #469
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![renovate[bot]](https://avatars.githubusercontent.com/in/2740?s=60&v=4){kind=small}
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR contains the following updates:
Test plan: CI should pass with updated dependencies. No review required: this is an automated dependency update PR.
Release Notes
facebook/zstd (llvm_zstd)
v1.5.6: Zstandard v1.5.6 - Chrome EditionCompare Source
This release highlights the deployment of Google Chrome 123, introducing
zstd-encodingfor Web traffic, offered as a preferable option for compression of dynamic contents. With limited web server support forzstd-encodingdue to its novelty, we are launching an updated Zstandard version to facilitate broader adoption.Improved latency (time to first byte) for web pages
Using
zstdcompression for large documents over the Internet, data is segmented into smaller blocks of up to 128 KB, for incremental updates. This is crucial for applications like Chrome that process parts of documents as they arrive. However, on slow or congested networks, there can be some brief unresponsiveness in the middle of a block transmission, delaying update. To mitigate such scenarios,libzstdintroduces the new parameterZSTD_c_targetCBlockSize, enabling the division of blocks into even smaller segments to enhance initial byte delivery speed. Activating this feature incurs a cost, both runtime (equivalent to -2% speed at level 8) and a slight compression efficiency decrease (<0.1%), but offers some desirable latency reduction, notably beneficial in areas with more congested network infrastructure.Improved compression ratio at high levels
Highest compression levels (typically 18+) receive some compression ratio improvement. The improvement is really noticeable for 32-bit structures, like arrays of
intfor example. A real-world example would the.debug_str_offsetssection ofDWARFdebug info withinELFexecutables, mentioned in #2832, for which the compression effectiveness increases by +35%. It's not rare for many files or objects to contain sections of 32-bit structures, resulting in corresponding compression ratio improvements.Granular binary size selection
libzstdprovides build customization, including options to compile only the compression or decompression modules, minimizing binary size. Enhanced inv1.5.6(source), it now allows for even finer control by enabling selective inclusion or exclusion of specific components within these modules. This advancement aids applications needing precise binary size management.Miscellaneous Enhancements
This release includes various minor enhancements and bug fixes to enhance user experience. Key updates include an expanded list of recognized compressed file suffixes for the
--exclude-compressedflag, improving efficiency by skipping presumed incompressible content. Furthermore, compatibility has been broadened to include additional chipsets (sparc64,ARM64EC,risc-v) and operating systems (QNX,AIX,Solaris,HP-UX).Change Log
api: Promote
ZSTD_c_targetCBlockSizeto Stable API by @felixhandteapi: new experimental
ZSTD_d_maxBlockSizeparameter, to reduce streaming decompression memory, by @terrellnperf: improve performance of param
ZSTD_c_targetCBlockSize, by @Cyan4973perf: improved compression of arrays of integers at high compression, by @Cyan4973
lib: reduce binary size with selective built-time exclusion, by @felixhandte
lib: improved huffman speed on small data and linux kernel, by @terrelln
lib: accept dictionaries with partial literal tables, by @terrelln
lib: fix CCtx size estimation with external sequence producer, by @embg
lib: fix corner case decoder behaviors, by @Cyan4973 and @aimuz
lib: fix zdict prototype mismatch in static_only mode, by @ldv-alt
lib: fix several bugs in magicless-format decoding, by @embg
cli: add common compressed file types to
--exclude-compressedby @daniellerozenblit (requested by @dcog989)cli: fix mixing
-cand-ocommands with--rm, by @Cyan4973cli: fix erroneous exclusion of hidden files with
--output-dir-mirrorby @felixhandtecli: improved time accuracy on BSD, by @felixhandte
cli: better errors on argument parsing, by @KapJI
tests: better compatibility with older versions of
grep, by @Cyan4973tests: lorem ipsum generator as default content generator, by @Cyan4973
build: cmake improvements by @terrelln, @sighingnow, @gjasny, @JohanMabille, @Saverio976, @gruenich, @teo-tsirpanis
build: bazel support, by @jondo2010
build: fix cross-compiling for AArch64 with lld by @jcelerier
build: fix Apple platform compatibility, by @nidhijaju
build: fix Visual 2012 and lower compatibility, by @Cyan4973
build: improve win32 support, by @DimitriPapadopoulos
build: better C90 compliance for zlibWrapper, by @emaste
port: make: fat binaries on macos, by @mredig
port: ARM64EC compatibility for Windows, by @dunhor
port: QNX support by @klausholstjacobsen
port: MSYS2 and Cygwin makefile installation and test support, by @QBos07
port: risc-v support validation in CI, by @Cyan4973
port: sparc64 support validation in CI, by @Cyan4973
port: AIX compatibility, by @likema
port: HP-UX compatibility, by @likema
doc: Improved specification accuracy, by @elasota
bug: Fix and deprecate ZSTD_generateSequences (#3981), by @terrelln
Full change list (auto-generated)
https://github.com/facebook/zstd/pull/36343634
DESTDIRandBINDIRseparately by @paulmenzel in https://github.com/facebook/zstd/pull/3752do { } while (0)by @terrelln in https://github.com/facebook/zstd/pull/38315921623by @Cyan4973 in https://github.com/facebook/zstd/pull/39005131069by @Cyan4973 in https://github.com/facebook/zstd/pull/3902targetCBlockSizeby @Cyan4973 in https://github.com/facebook/zstd/pull/3917zstd::libzstdCMake target if only static or dynamic linkage is specified. by @teo-tsirpanis in https://github.com/facebook/zstd/pull/3811ZSTD_RowFindBestMatchcomment by @yoniko in https://github.com/facebook/zstd/pull/3947utimensat()on FreeBSD by @felixhandte in https://github.com/facebook/zstd/pull/3960--output-dir-mirrorby @felixhandte in https://github.com/facebook/zstd/pull/3963ZSTD_c_targetCBlockSizeParameter to Stable API by @felixhandte in https://github.com/facebook/zstd/pull/3964New Contributors
Full Changelog: facebook/zstd@v1.5.5...v1.5.6
v1.5.5: Zstandard v1.5.5Compare Source
This is a quick fix release. The primary focus is to correct a rare corruption bug in high compression mode, detected by @danlark1 . The probability to generate such a scenario by random chance is extremely low. It evaded months of continuous fuzzer tests, due to the number and complexity of simultaneous conditions required to trigger it. Nevertheless, @danlark1 from Google shepherds such a humongous amount of data that he managed to detect a reproduction case (corruptions are detected thanks to the checksum), making it possible for @terrelln to investigate and fix the bug. Thanks !
While the probability might be very small, corruption issues are nonetheless very serious, so an update to this version is highly recommended, especially if you employ high compression modes (levels 16+).
When the issue was detected, there were a number of other improvements and minor fixes already in the making, hence they are also present in this release. Let’s detail the main ones.
Improved memory usage and speed for the
--patch-frommodeV1.5.5introduces memory-mapped dictionaries, by @daniellerozenblit, for both posix #3486 and windows #3557.This feature allows
zstdto memory-map large dictionaries, rather than requiring to load them into memory. This can make a pretty big difference for memory-constrained environments operating patches for large data sets.It's mostly visible under memory pressure, since
mmapwill be able to release less-used memory and continue working.But even when memory is plentiful, there are still measurable memory benefits, as shown in the graph below, especially when the reference turns out to be not completely relevant for the patch.
This feature is automatically enabled for
--patch-fromcompression/decompression when the dictionary is larger than the user-set memory limit. It can also be manually enabled/disabled using--mmap-dictor--no-mmap-dictrespectively.Additionally, @daniellerozenblit introduces significant speed improvements for
--patch-from.An
I/Ooptimization in #3486 greatly improves--patch-fromdecompression speed on Linux, typically by+50%on large files (~1GB).Compression speed is also taken care of, with a dictionary-indexing speed optimization introduced in #3545. It wildly accelerates
--patch-fromcompression, typically doubling speed on large files (~1GB), sometimes even more depending on exact scenario.This speed improvement comes at a slight regression in compression ratio, and is therefore enabled only on non-ultra compression strategies.
Speed improvements of middle-level compression for specific scenarios
The row-hash match finder introduced in version 1.5.0 for levels 5-12 has been improved in version 1.5.5, enhancing its speed in specific corner-case scenarios.
The first optimization (#3426) accelerates streaming compression using
ZSTD_compressStreamon small inputs by removing an expensive table initialization step. This results in remarkable speed increases for very small inputs.The following scenario measures compression speed of
ZSTD_compressStreamat level 9 for different sample sizes on a linux platform running an i7-9700k cpu.v1.5.4(MB/s)v1.5.5(MB/s)The second optimization (#3552) speeds up compression of incompressible data by a large multiplier. This is achieved by increasing the step size and reducing the frequency of matching when no matches are found, with negligible impact on the compression ratio. It makes mid-level compression essentially inexpensive when processing incompressible data, typically, already compressed data (note: this was already the case for fast compression levels).
The following scenario measures compression speed of
ZSTD_compresscompiled withgcc-9for a ~10MB incompressible sample on a linux platform running an i7-9700k cpu.v1.5.4(MB/s)v1.5.5(MB/s)Miscellaneous
There are other welcome speed improvements in this package.
For example, @felixhandte managed to increase processing speed of small files by carefully reducing the nb of system calls (#3479). This can easily translate into +10% speed when processing a lot of small files in batch.
The Seekable format received a bit of care. It's now much faster when splitting data into very small blocks (#3544). In an extreme scenario reported by @P-E-Meunier, it improves processing speed by x90. Even for more "common" settings, such as using 4KB blocks on some "normally" compressible data like
enwik, it still provides a healthy x2 processing speed benefit. Moreover, @dloidolt merged an optimization that reduces the nb ofI/Oseek()events during reads (decompression), which is also beneficial for speed.The release is not limited to speed improvements, several loose ends and corner cases were also fixed in this release. For a more detailed list of changes, please take a look at the changelog.
Change Log
mmaplarge dictionaries to save memory, by @daniellerozenblit--patch-frommode (~+50%) (#3545) by @daniellerozenblitzstdno longer crashes when requested to write into write-protected directory (#3541) by @felixhandte-o(#3584, @Cyan4973) reported by @georgmucmakedoes no longer require 3.18 as minimum version (#3510) by @koutests/fullbenchcan benchmark multiple files (#3516) by @dloidoltFull change list (auto-generated)
f-variants ofchmod()andchown()by @felixhandte in https://github.com/facebook/zstd/pull/3479Configuration
📅 Schedule: Branch creation - "on the 1st through 7th day of the month" in timezone America/Los_Angeles, Automerge - At any time (no schedule defined).
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.
This PR was generated by Mend Renovate. View the repository job log.