The compressor interface allows decompression in chunks (needed to support lazy decompression) but does not yet support chunked compression. Each compressor, given a data buffer to compress, just compresses the whole thing in one go in memory. This was the naïve approach taken initially just to get the feature done, and in fact Python asdf does the same thing from what I could tell when implementing it.
All of the compressors do, in principle, suppport incremental compression though, which makes for significantly more efficient memory usage on write. This should be improved then in a future version (the compressor interface is in principle pluggable for adding new compressors, but it's not documented yet and should be safe to change in a future release as needed. It's only an internal API at the moment.
The compressor interface allows decompression in chunks (needed to support lazy decompression) but does not yet support chunked compression. Each compressor, given a data buffer to compress, just compresses the whole thing in one go in memory. This was the naïve approach taken initially just to get the feature done, and in fact Python asdf does the same thing from what I could tell when implementing it.
All of the compressors do, in principle, suppport incremental compression though, which makes for significantly more efficient memory usage on write. This should be improved then in a future version (the compressor interface is in principle pluggable for adding new compressors, but it's not documented yet and should be safe to change in a future release as needed. It's only an internal API at the moment.