Add support for non-byte-aligned bitstrings

[pguyot]

These changes are made under both the "Apache 2.0" and the "GNU Lesser General
Public License 2.1 or later" license terms (dual license).

SPDX-License-Identifier: Apache-2.0 OR LGPL-2.1-or-later

[petermm]

PR Review — c8fd1c1ed (“Add support for non-byte-aligned bitstrings”)

Verdict

Changes requested. The representation and matching changes are generally coherent, but construction still truncates dynamically supplied bitstrings, partial-width little-endian integers use the wrong bit layout, and one JIT matching path still measures only complete bytes.

Findings

1. High — dynamic bitstring segments are accepted and then truncated to complete bytes

bs_append, bs_private_append, bs_put_binary, and bs_create_bin now accept term_is_bitstring(src), but continue to derive the source length from term_binary_size(src) and copy with memcpy (src/libAtomVM/opcodesswitch.h:3448-3477, src/libAtomVM/opcodesswitch.h:3596-3632, and src/libAtomVM/opcodesswitch.h:5626-5663, src/libAtomVM/opcodesswitch.h:5872-5910). term_binary_size intentionally excludes the trailing partial byte, so accepting the wider type silently loses data and places subsequent segments at the wrong offset.

The JIT duplicates this behavior: its all size calculation reads the complete-byte count (libs/jit/src/jit.erl:2811-2826), and jit_bitstring_copy_binary copies that many bytes with memcpy (src/libAtomVM/jit.c:1711-1730).

This was reproduced against the rebuilt VM:

copy(Bits) -> <<Bits/bitstring>>.
append(Bits) -> <<Bits/bitstring, 16#AB:8>>.

Bits = <<1:1>>,
{copy(Bits), append(Bits)}.

OTP returns {<<1:1>>, <<1:1, 16#AB:8>>}; AtomVM returns {<<>>, <<16#AB>>}.

The complete fix is not a one-line change: size segments in bits (term_bit_size(src) for all), validate explicit bit lengths against term_bit_size(src), copy the requested number of bits at the current destination bit offset, and make the interpreter/JIT paths identical. Until that is implemented, restoring binary-only validation on paths that can only copy bytes is safer than accepting and corrupting the value.

At minimum, add a regression that prevents this behavior from returning unnoticed:

diff --git a/tests/erlang_tests/test_bs.erl b/tests/erlang_tests/test_bs.erl
--- a/tests/erlang_tests/test_bs.erl
+++ b/tests/erlang_tests/test_bs.erl
@@
 test_create_with_unaligned_int_size() ->
@@
     true = is_bitstring(B),
+    Bits = id(<<1:1>>),
+    <<1:1>> = copy_bitstring(Bits),
+    <<1:1, 16#AB:8>> = append_to_bitstring(Bits),
     ok.
+
+copy_bitstring(Bits) -> <<Bits/bitstring>>.
+append_to_bitstring(Bits) -> <<Bits/bitstring, 16#AB:8>>.

2. High — non-byte-sized little-endian integer fields have the wrong layout

The commit enables non-byte-aligned construction, but the generic little-endian helpers byte-swap ceil(N / 8) whole bytes and then read/write only N bits (src/libAtomVM/bitstring.c:32-58, src/libAtomVM/bitstring.c:61-112). For widths that are not multiples of eight, Erlang instead lays out complete low-order bytes first and then the remaining high-order bits.

For a dynamically constructed <<16#ABC:12/little>>, OTP produces <<16#BC, 16#A:4>>. The rebuilt AtomVM produces a first byte of 16#C0 (and the current printer hides the final four bits). Extraction applies the corresponding incorrect whole-value byte swap, so matching is affected too; a pack/unpack round trip may conceal the paired error.

Preserve the existing byte-aligned fast path. For a partial width, process complete eight-bit chunks in low-byte-first order, then process the final N rem 8 high-order bits; extraction must perform the inverse mapping.

Add byte-level, OTP-compatible tests rather than only round trips:

diff --git a/tests/erlang_tests/test_bs.erl b/tests/erlang_tests/test_bs.erl
--- a/tests/erlang_tests/test_bs.erl
+++ b/tests/erlang_tests/test_bs.erl
@@
 test_create_with_int_little_endian() ->
     <<2, 1>> = create_int_binary_little_endian(16#0102, 16),
+    <<16#BC, 16#A:4>> = create_int_binary_little_endian(16#ABC, id(12)),

The test matrix should include widths 1, 4, 7, 9, 12, 15, and 63, both at bit offset zero and after an unaligned prefix.

3. High — JIT bs_match_string rejects literals that consume trailing bits

The interpreter correctly changed its capacity check to term_bit_size(bs_bin) (src/libAtomVM/opcodesswitch.h:4173), but the JIT primitive still checks term_binary_size(bs_bin) * 8 (src/libAtomVM/jit.c:1899-1906). A one-bit bitstring therefore has zero capacity according to the JIT and a valid literal match fails only when JIT-compiled.

diff --git a/src/libAtomVM/jit.c b/src/libAtomVM/jit.c
--- a/src/libAtomVM/jit.c
+++ b/src/libAtomVM/jit.c
@@ -1902,7 +1902,9 @@ static bool jit_bitstring_match_module_str(Context *ctx, JITState *jit_state, te
     size_t remaining = 0;
     const uint8_t *str = module_get_str(jit_state->module, str_id, &remaining);
 
-    if (term_binary_size(bs_bin) * 8 - bs_offset < MIN(remaining * 8, bits)) {
+    size_t capacity_bits = term_bit_size(bs_bin);
+    if (bs_offset > capacity_bits
+        || capacity_bits - bs_offset < MIN(remaining * 8, bits)) {
         return false;
     }

Add an interpreter-versus-JIT test that matches a literal such as <<1:1>> from the trailing partial byte.

4. Medium — ETF serialization preserves insignificant padding instead of canonicalizing it

BIT_BINARY_EXT decoding accepts non-zero insignificant low bits in the final byte, as OTP does (src/libAtomVM/external_term.c:842-869). Serialization then copies the entire backing byte unchanged (src/libAtomVM/external_term.c:395-406). Consequently, semantically equal bitstrings can have different external encodings depending on their origin.

For example, AtomVM accepts <<131,77,0,0,0,1,1,255>> as <<1:1>>, but term_to_binary/1 emits the dirty final byte 255; OTP emits the canonical final byte 128.

diff --git a/src/libAtomVM/external_term.c b/src/libAtomVM/external_term.c
--- a/src/libAtomVM/external_term.c
+++ b/src/libAtomVM/external_term.c
@@ -401,6 +401,9 @@ static int serialize_term(uint8_t *buf, term t, GlobalContext *glb)
                 WRITE_32_UNALIGNED(buf + 1, nbytes);
                 buf[5] = last_byte_bits;
                 memcpy(buf + 6, data, nbytes);
+                if (nbytes != 0) {
+                    buf[6 + nbytes - 1] &= (uint8_t) (0xFFU << (8 - last_byte_bits));
+                }
             }
             return 6 + nbytes;

Add a decode/re-encode test using a final byte of 255 with one significant bit and assert that the serialized final byte is 128.

5. Low — printing a bitstring omits its trailing partial field

term_funprint now enters the binary branch for every bitstring, but still iterates over only term_binary_size(t) complete bytes (src/libAtomVM/term.c:346-399). Thus <<1:1>> prints as <<>>, and <<255,5:7>> prints as <<255>>, which misrepresents values in diagnostics and crash output.

diff --git a/src/libAtomVM/term.c b/src/libAtomVM/term.c
--- a/src/libAtomVM/term.c
+++ b/src/libAtomVM/term.c
@@ -391,6 +391,17 @@ int term_funprint(PrinterFun *fun, term t, const GlobalContext *global)
                 ret += printed;
             }
         }
+        uint8_t trailing_bits = (uint8_t) (term_bit_size(t) % 8);
+        if (trailing_bits != 0) {
+            uint8_t value = binary_data[len] >> (8 - trailing_bits);
+            int printed = fun->print(fun, "%s%u:%u", len != 0 ? "," : "",
+                (unsigned) value, (unsigned) trailing_bits);
+            if (UNLIKELY(printed < 0)) {
+                return printed;
+            }
+            ret += printed;
+        }
         int printed = fun->print(fun, ">>");

Validation performed

• cmake --build build -j4 — passed.
• build/tests/test-erlang — passed.
• Rebuilt AtomVM execution of test_bs.beam and test_binary_to_term.beam — both returned 0.
• Focused runtime probes confirmed findings 1, 2, and 4.
• Static interpreter/JIT comparison confirmed finding 3.