Pad binary block headers so block data is aligned for its datatype

[embray]

This pertains to asdf-format/libasdf#227 which provides more background.

When reading an uncompressed binary block, the natural implementation is to expose a zero-copy view of the block payload> np.ndarray(shape, dtype, buffer=mmap, offset=...) in Python, or an mmap pointer as in libasdf. The alignment of that pointer is whatever the block payload's byte offset in the file happens to be.

That offset is block_magic_offset + 4 (magic) + 2 (header_size) + header_size, and block_magic_offset depends on the length of the YAML tree above it. So the alignment of a user's array data is an accident of how long the tree serialized to, and it changes whenever anything in the tree changes — adding a history entry can flip an array from 8-byte aligned to misaligned.

In practice, for a file written by asdf today, an int64 array, for example, typically lands on an offset that is not a multiple of 8.

• In C/C++ implementations, forming a const int64_t * to a misaligned address and dereferencing it is undefined behavior, independent of whether the hardware tolerates it. UBSan flags it; it faults on strict-alignment targets; and compilers are entitled to assume alignment when vectorizing a typed dereference, though modern compilers implement various workarounds to this.

• In numpy, an array built over a misaligned buffer pushes ufuncs onto buffered/unaligned code paths (copying into aligned scratch buffers, or memcpy-based inner loops) rather than the faster aligned kernels.

Neither of these are a significant problem for most users today, though it can have subtle performance effects. This also only relevant for non-compressed data blocks of course--if they are compressed, they have to be decompressed anyways so zero-copy reads are not an issue.

My recommendation: When writing an uncompressed binary block (though it can't hurt to do even if compressed), pad the block header by increasing header_size beyond the standard 48 bytes, so that the block payload begins at a 16-byte boundary. The ASDF specification already specifically allows for this, and asdf also handles this case fine.

This requires no format change. header_size is already a variable field that every conforming reader honors, so existing readers (including older asdf versions) read such files correctly with no changes. The cost is at most 15 bytes per block.

(The standard also permits arbitrary padding bytes before the block magic, but in practice header padding is the compatible mechanism and in a way is more self-documenting.)

Padding unconditionally to 16 bytes is a superset of every numeric scalar datatype supported so it's the natural choice (more complex record/array datatypes are not relevant here, since they can only be handled through unaligned code paths anyways).

On the read side, asdf already handles this fine (it allows block headers larger than the minimum 48 bytes and treats additional bytes in the block header as padding), so this is just a small fix on the write side.

Caveat: If a user hand-edits the tree then all bets are off; typically this is not the case but library authors still should be prepared for the possibility that alignment of the data block cannot be guaranteed.

I would suggest a complementary patch to the ASDF specification about this too, as a recommendation to ASDF library implementers.