Level 7: filling the decode-speed gap between lz4hc and zstd

[hellobertrand]

Why this discussion

In a recent DwarFS thread, @solbjorn made a point that matches what I thinking about for a while:

currently there's a huge gap between lz4hc and zstd levels which give better ratio — the decompression speed decreases significantly. I'd love to have something in-between.

That gap is real. On silesia (lzbench):

Codec	Ratio	Decode speed
lz4hc -9	36.75%	4.8 GB/s (Apple M3)
lz4hc -12	36.45%	4.8 GB/s (Apple M3)
zxc -6	36.27%	on par with lz4hc -12
(the gap)
zstd 1..3	31–34%	1.9-1.8 GB/s

Everything between 36% and ~31% with decode speed above 2 GB/s is essentially unserved. That's where a zxc level 7 could live.

What I'm exploring

I have an early prototype (format v7) that adds, on top of level 6:

• Huffman-coded sequence tokens (level 6 only entropy-codes literals),
• longer literal Huffman codes (max code length 8 -> 11), which mainly pays on wide-alphabet data,
• a deeper optimal parse.

Early numbers on silesia (Apple M-series, lzbench): 33.78% ratio at 4 GB/s decode roughly 2.7 ratio points better than the lz4hc class, while still decoding 2x faster than zstd. Compression is very slow (7 MB/s), which I consider acceptable for a compress-once / read-many level.
An another prototype give 35.2% ratio at 5 GB/s decode.

Two design principles:

1. Levels 1–6 don't change. Level 6 stays byte-identical and keeps its decode speed.
2. Decode speed stays in the multi-GB/s class. If a lever costs more decode speed than it pays in ratio, it's out; level 7 will not become "zstd but slower to decompress".

What I'd like your input on

1. Where's your threshold? For your use case (DwarFS, SquashFS, game assets, FOTA…), what ratio gain would justify what decode-speed loss? Would 33% at 4 GB/s change your choice vs 36% @ 6 GB/s? Would 35% at 5 GB/s?
2. Compression speed floor. Level 7 compresses slowly by design. Is ~5–10 MB/s per core acceptable in your pipelines, or is there a floor below which it becomes unusable?
3. Anything else in the gap? If you've benchmarked other codecs in this zone I'm interested in what worked and what didn't.

[hellobertrand]

Proto v1 (level -7)

On my laptop (Apple M3)
lzbench 2.3 | Clang 21.0.0 | 64-bit MacOS |

Compressor name	Compression	Decompress.	Compr. size	Ratio	Filename
zxc 0.13.0 -6	14.6 MB/s	6149 MB/s	76882010	36.27	../corpus/silesia.tar
zxc 0.13.0 -7	6.95 MB/s	3992 MB/s	71603349	33.78	../corpus/silesia.tar
lz4hc 1.10.0 -9	48.0 MB/s	4760 MB/s	77884448	36.75	../corpus/silesia.tar
lz4hc 1.10.0 -12	16.1 MB/s	4876 MB/s	77262620	36.45	../corpus/silesia.tar
zstd 1.5.7 -1	711 MB/s	1990 MB/s	73193704	34.53	../corpus/silesia.tar
zstd 1.5.7 -3	434 MB/s	1897 MB/s	66133500	31.20	../corpus/silesia.tar

[hellobertrand]

Proto v2 (level -7)

On my laptop (Apple M3)
lzbench 2.3 | Clang 21.0.0 | 64-bit MacOS |

Compressor name	Compression	Decompress.	Compr. size	Ratio	Filename
zxc 0.13.0 -6	14.4 MB/s	6119 MB/s	76882010	36.27	../corpus/silesia.tar
zxc 0.13.0 -7	7.00 MB/s	4968 MB/s	74598267	35.20	../corpus/silesia.tar
lz4hc 1.10.0 -9	51.0 MB/s	4880 MB/s	77884448	36.75	../corpus/silesia.tar
lz4hc 1.10.0 -12	16.5 MB/s	4871 MB/s	77262620	36.45	../corpus/silesia.tar
zstd 1.5.7 -1	707 MB/s	1991 MB/s	73193704	34.53	../corpus/silesia.tar
zstd 1.5.7 -3	436 MB/s	1888 MB/s	66133500	31.20	../corpus/silesia.tar

[solbjorn]

33.78% ratio at 4 GB/s decode

Great numbers!

Is ~5–10 MB/s per core acceptable

To me -- yes, sure.

Anything else in the gap?

In real tests, at least on my engine, the floor is 3 Gb/s, after which the game loading time starts decreasing due to CPU usage. This depends on different external factors like SSD etc. But I'd say the whole 3-4 Gb/s area is an object of interest -- every ratio point can exchange SSD bandwidth for CPU and give better results -- if not during the initial load, then in-game.

[hellobertrand]

@solbjorn this is the last benchmarks in CI. This is a v7 format breaking changes...

As you see level 7 is 1.8x decode speed of zstd -1 with better ratio on Zen 3 AMD 7763. And on Apple M2 is about 2x than zstd -1.
Level 6 on x86 is at the same speed than lz4hc -9 with better ratio. On apple M2 decode speed is 6% better than v0.12.0

Benchmark Result (ubuntu-latest)

lzbench 2.3 | GCC 14.2.0 | 64-bit Linux | AMD EPYC 7763 64-Core Processor

Compressor name	Compression	Decompress.	Compr. size	Ratio	Filename
memcpy	22348 MB/s	22182 MB/s	211947520	100.00	1 files
zxc 0.13.0 -1	664 MB/s	7145 MB/s	130356147	61.50	1 files
zxc 0.13.0 -2	445 MB/s	5967 MB/s	113633866	53.61	1 files
zxc 0.13.0 -3	189 MB/s	3947 MB/s	97051444	45.79	1 files
zxc 0.13.0 -4	131 MB/s	3794 MB/s	90392857	42.65	1 files
zxc 0.13.0 -5	79.0 MB/s	3658 MB/s	85341256	40.27	1 files
zxc 0.13.0 -6	9.68 MB/s	3409 MB/s	76878179	36.27	1 files
zxc 0.13.0 -7	6.51 MB/s	2208 MB/s	69953576	33.01	1 files
lz4 1.10.0	582 MB/s	3559 MB/s	100880800	47.60	1 files
lz4 1.10.0 --fast -17	1017 MB/s	4114 MB/s	131732802	62.15	1 files
lz4hc 1.10.0 -9	33.9 MB/s	3410 MB/s	77884448	36.75	1 files
lzav 5.9 -1	433 MB/s	2703 MB/s	84644732	39.94	1 files
snappy 1.2.2	613 MB/s	1591 MB/s	101512076	47.89	1 files
zstd 1.5.7 --fast --1	447 MB/s	1629 MB/s	86916294	41.01	1 files
zstd 1.5.7 -1	407 MB/s	1223 MB/s	73193704	34.53	1 files
zstd 1.5.7 -3	212 MB/s	1102 MB/s	66133500	31.20	1 files
zlib 1.3.2 -1	98.3 MB/s	328 MB/s	77259029	36.45	1 files

Benchmark Result (macos-26-xlarge)

lzbench 2.3 | Clang 21.0.0 | 64-bit MacOS |

Compressor name	Compression	Decompress.	Compr. size	Ratio	Filename
memcpy	52735 MB/s	52750 MB/s	211947520	100.00	1 files
zxc 0.13.0 -1	880 MB/s	12768 MB/s	130356147	61.50	1 files
zxc 0.13.0 -2	587 MB/s	10581 MB/s	113633866	53.61	1 files
zxc 0.13.0 -3	252 MB/s	7022 MB/s	97051444	45.79	1 files
zxc 0.13.0 -4	173 MB/s	6652 MB/s	90392857	42.65	1 files
zxc 0.13.0 -5	103 MB/s	6257 MB/s	85341256	40.27	1 files
zxc 0.13.0 -6	12.8 MB/s	5912 MB/s	76878179	36.27	1 files
zxc 0.13.0 -7	8.72 MB/s	3652 MB/s	69953576	33.01	1 files
lz4 1.10.0	809 MB/s	4768 MB/s	100880800	47.60	1 files
lz4 1.10.0 --fast -17	1347 MB/s	5602 MB/s	131732802	62.15	1 files
lz4hc 1.10.0 -9	47.8 MB/s	4508 MB/s	77884448	36.75	1 files
lzav 5.9 -1	670 MB/s	3879 MB/s	84644732	39.94	1 files
snappy 1.2.2	877 MB/s	3255 MB/s	101415443	47.85	1 files
zstd 1.5.7 --fast --1	722 MB/s	2512 MB/s	86916294	41.01	1 files
zstd 1.5.7 -1	644 MB/s	1792 MB/s	73193704	34.53	1 files
zstd 1.5.7 -3	391 MB/s	1689 MB/s	66133500	31.20	1 files
zlib 1.3.2 -1	149 MB/s	408 MB/s	77259029	36.45	1 files

[solbjorn]

@hellobertrand, huge thanks man. Level 7 does exactly what I needed!

I hope it won't take too long for mhx to merge your PR to upstream DwarFS. In the meantime, could you maybe keep your dwfs fork up to date with the upstream branch? I'll be still using it in the engine.

[hellobertrand]

@solbjorn I'm glad level 7 meets your needs. There is indeed a gap to fill between lz4hc and zstd.

I think mhx will hold off on merging until zxc v1.0. That's when I lock the format and guarantee backward compatibility, which is what you'd want before a filesystem commits to it.

Meanwhile I'll definitely keep the fork current: I'll update the PR once v0.13.0 hits vcpkg (microsoft/vcpkg#52894), and I'll ping you here when it's ready.

[hellobertrand]

@solbjorn mhx/dwarfs#362 updated with zxc v0.13.0