Add Iguana container + LZ77 layer to hwy/contrib/iguana - #3342
Open
tvost2 wants to merge 2 commits into
Open
Conversation
First step of the "Iguana" op_wishlist item: a Highway port of Iguana's ANS32 entropy stage - the 32-way interleaved rANS coder that is the "decompress at 10+ GB/s" core of that compressor (github.com/SnellerInc/ sneller, ion/zion/iguana). Ported from the pure-Go reference (ans32.go, ans_statistics.go); the bitstream is byte-for-byte compatible. - ans.h/.cc: the frequency model (histogram + normalization, incl. the empty-input and single-symbol edge cases), dense-table (de)serialization, the scalar ANS32 encoder, and a scalar reference decoder. - ans-inl.h: SIMD Ans32Decode. Each round gathers 32 dense-table entries, does one Mul/Add, and renormalizes the lanes that fell below 2^16 by pulling one 16-bit word each from the two ends of the payload, vectorized with Expand (+ Reverse for the backward half). CappedTag<uint32_t, 16>, so 4/8/16 lanes per group. HWY_SCALAR and the scalable targets fall back to the scalar reference, which produces identical output. - ans_test.cc: round-trips a range of sizes and models and cross-checks the SIMD decoder against the scalar reference. Passes on AVX3/AVX2/SSE4/SSSE3/SSE2/EMU128 locally; strict -Werror and clang-format are clean. Iguana's LZ77 layer, ANS1/ANS_nibble, and a fully-vectorized tail are follow-ups.
Completes the "Iguana" op_wishlist item on top of the ANS32 entropy stage: hwy/contrib/iguana/ is now a working portable Iguana codec (github.com/SnellerInc/sneller). Ported from the pure-Go reference; the bitstream is compatible with it. - iguana.h/.cc: the container format (backward control varints; CopyRaw / DecodeIguana / DecodeANS32 commands), the LZ77 stage (token loop + overlap-aware wild copy), and the scalar encoder - a Lizard-style hash-chain match finder with the reference's repeat-offset and end-of-buffer handling. DecompressScalar ties it together. - iguana-inl.h: Decompress - the same container loop, but each entropy- coded stream is routed through the SIMD ANS32 decoder. The container parsing and LZ token loop are inherently serial, so they stay in the target-independent helpers. Output matches DecompressScalar. - iguana_test.cc: round-trips a range of sizes and data models, the >64 KiB far-offset path, and a re-sliced short string; cross-checks the SIMD path against the scalar reference. Covers the EncodingIguana / EntropyANS32 pipeline. ANS1 / ANS_nibble stream modes and a vectorized wild copy are follow-ups. Passes on AVX3/AVX2/SSE4/SSSE3/SSE2/EMU128 locally; strict -Werror and clang-format are clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Completes the Iguana
op_wishlist.mditem: the container + LZ77 layer ontop of the ANS32 entropy stage from #3341, so
hwy/contrib/iguana/is now aworking portable Iguana codec (github.com/SnellerInc/sneller, ion/zion/iguana).
Stacked on #3341 — review that one first; the diff here is only the new
iguana.{h,cc}/iguana-inl.h/iguana_test.cc.Ported from the pure-Go reference (
decoder.go,encoder.go,stream.go,recovered via the Go module proxy). Bitstream-compatible with it.
What's here
iguana.h/iguana.ccCopyRaw/DecodeIguana/DecodeANS32commands), the LZ77 stage (DecompressIguanaLZ: token loop + overlap-aware wild copy), and the scalar encoder — a Lizard-style hash-chain match finder (kChainBits,kHashBytes,kHistSize) with the reference's repeat-offset and end-of-buffer handling.DecompressScalarties it together.iguana-inl.hDecompress— the same container loop, but each entropy-coded stream goes through the SIMD ANS32 decoder from #3341 (iguana_ans::Ans32Decode). The container parsing and LZ token loop are inherently serial, so they stay in the target-independent helpers. Output is identical toDecompressScalar.iguana_test.ccCovers the
EncodingIguana/EntropyANS32pipeline thatEncoder.Compressproduces.
ANS1/ANS_nibblestream modes are declared-but-rejected (smallscalar coders; a follow-up).
Validation
iguana_testpasses on AVX3 / AVX2 / SSE4 / SSSE3 / SSE2 / EMU128 locally.every test case, including the far-offset path; the encoder here produces the
same block and both decoders reproduce the input.
-Werror(-Wconversion -Wsign-conversion -Wshadow -Wcast-align …)and
clang-formatare clean.Follow-ups
The LZ wild copy could use wide Highway loads/stores for non-overlapping
matches;
ANS1/ANS_nibble; aHWY_DYNAMIC_DISPATCHentry point.