Bounds-check XtcBasedEncoder and Lossless_bypass against corrupted input

[alexey-milovidov]

Problem

XtcBasedEncoder::decode and Lossless_bypass::decompress read sizes and indices from the compressed data without validating them. On corrupted or adversarial input (e.g. when fuzzing the decompressor) this leads to out-of-bounds writes/reads and a memory leak:

• XtcBasedEncoder::decode allocates buffer.data twice — the first allocation is immediately overwritten and leaked.
• smallIdx, read from the compressed data, indexes the fixed-size magicInts table (magicInts[smallIdx], magicInts[smallIdx - 1]) with no range check — an out-of-bounds read of a static array.
• buffer.index, an attacker-controlled byte count read from the data, is memcpy'd into buffer.data without checking it against the buffer capacity — a heap buffer overflow.
• The buffer.data / index-buffer allocations are not null-checked.
• Lossless_bypass::decompress does not null-check its malloc before memcpy.

Changes

Bounds checks and null checks only; valid data is unaffected:

• drop the leaked first buffer.data allocation;
• reject smallIdx outside [0, LASTIDX) before indexing magicInts;
• reject a packed size (buffer.index) larger than the buffer.data capacity before the copy loop;
• null-check the buffer.data and index-buffer allocations;
• null-check the Lossless_bypass allocation.

Notes

XtcBasedEncoder::decode does not receive a length for its input buffer, so the reads from the input pointer itself can not be fully bounded without threading a length through the EncoderInterface::decode signature; this change bounds the out-of-bounds writes and the static-array index, which are the memory-corruption issues.

Context

Found while integrating SZ3 as an experimental compression codec in ClickHouse and fuzzing the decompressor: ClickHouse/ClickHouse#108788