Skip to content

Commit

Permalink
Fix bug with RANS_ORDER_STRIPE on large blocks.
Browse files Browse the repository at this point in the history
The meta-data is 2 bytes for order + N, ? bytes for original size, and
?*N bytes for the N compressed sizes.  We forgot about the original
size so only left room for 2+5*N (with 5 being the maximum size of a
32-bit var int).  So blocks larger than 2^28 risked not fitting in the
allocated meta-data size.

Reported by Divon Lan.
  • Loading branch information
jkbonfield committed Aug 9, 2022
1 parent 3891cb6 commit 3ef17f6
Show file tree
Hide file tree
Showing 3 changed files with 12 additions and 7 deletions.
6 changes: 4 additions & 2 deletions NEWS.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
Release 1.3.0: 19th July 2022
-----------------------------
Release 1.3.0: 9th August 2022
------------------------------

The primary change in this release is a new SIMD enabled rANS codec.

Expand Down Expand Up @@ -49,6 +49,8 @@ Bug fixes
(Additionally blocks above 2GB now error, rather than crashing or
returning incorrect results.)

- Fix encode error with large blocks using RANS_ORDER_STRIPE.


Release 1.2.2: 1st April 2022
-----------------------------
Expand Down
9 changes: 6 additions & 3 deletions htscodecs/arith_dynamic.c
Original file line number Diff line number Diff line change
Expand Up @@ -75,11 +75,14 @@
#define MAGIC 8

unsigned int arith_compress_bound(unsigned int size, int order) {
int N = (order>>8) & 0xff;
if (!N) N=4;
return (order == 0
? 1.05*size + 257*3 + 4
: 1.05*size + 257*257*3 + 4 + 257*3+4) +
: 1.05*size + 257*257*3 + 4 + 257*3+4) + 5 +
((order & X_PACK) ? 1 : 0) +
((order & X_RLE) ? 1 + 257*3+4: 0) + 5;
((order & X_RLE) ? 1 + 257*3+4: 0) +
((order & X_STRIPE) ? 7 + 5*N: 0);
}

#ifndef MODEL_256 // see fqzcomp_qual_fuzz.c
Expand Down Expand Up @@ -735,7 +738,7 @@ unsigned char *arith_compress_to(unsigned char *in, unsigned int in_size,
c_meta_len += var_put_u32(out+c_meta_len, out_end, in_size);
out[c_meta_len++] = N;

out2_start = out2 = out+2+5*N; // shares a buffer with c_meta
out2_start = out2 = out+7+5*N; // shares a buffer with c_meta
for (i = 0; i < N; i++) {
// Brute force try all methods.
// FIXME: optimise this bit. Maybe learn over time?
Expand Down
4 changes: 2 additions & 2 deletions htscodecs/rANS_static4x16pr.c
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ unsigned int rans_compress_bound_4x16(unsigned int size, int order) {
((order & RANS_ORDER_PACK) ? 1 : 0) +
((order & RANS_ORDER_RLE) ? 1 + 257*3+4: 0) + 20 +
((order & RANS_ORDER_X32) ? (32-4)*4 : 0) +
((order & RANS_ORDER_STRIPE) ? 1 + 5*N: 0);
((order & RANS_ORDER_STRIPE) ? 7 + 5*N: 0);
return sz + (sz&1) + 2; // make this even so buffers are word aligned
}

Expand Down Expand Up @@ -1181,7 +1181,7 @@ unsigned char *rans_compress_to_4x16(unsigned char *in, unsigned int in_size,
unsigned char *out_best = NULL;
unsigned int out_best_len = 0;

out2_start = out2 = out+2+5*N; // shares a buffer with c_meta
out2_start = out2 = out+7+5*N; // shares a buffer with c_meta
for (i = 0; i < N; i++) {
// Brute force try all methods.
int j, m[] = {1,64,128,0}, best_j = 0, best_sz = in_size+10;
Expand Down

0 comments on commit 3ef17f6

Please sign in to comment.