add PDF stream writer

[milahu]

currently pymupdf does not support writing PDF streams

currently i have to buffer the whole output document in RAM
and then i can write the complete document to disk

# pseudo code

import pymupdf

input_doc = pymupdf.open("input.pdf")
output_doc = pymupdf.open()  # empty PDF

# buffer to RAM
for page_idx in range(100):
    out_page = output_doc.new_page(width=100, height=200)
    colorspace = pymupdf.csGRAY
    page_image = get_page_image(page_idx) # np.array
    rect = input_doc[page_idx].rect
    h, w = page_image.shape[:2]
    pix = pymupdf.Pixmap(colorspace, w, h, page_image.tobytes(), False)
    out_page.insert_image(rect, pixmap=pix)

# write to disk
output_doc.save("output.pdf")
output_doc.close()

problem:
this fails if the output document is bigger than RAM

solution:
stream-write the output document to disk

i would need to pass the output file path to pymupdf.open
(which currently fails if the output file does not exist)
and maybe explicitly enable output-streaming

output_doc = pymupdf.open("output.pdf", write_stream=True)

so then i can remove the output_doc.save call
and the output file is finalized in output_doc.close()

proof of concept:
i have implemented a minimal streaming PDF writer in
rahimnathwani/binarize-pdf#2
as expected, this uses significantly less memory

[JorjMcKie]

I don't see why this would be necessary, respectively create a truly stable solution.
Inserting raw pixmaps as images is the most memory-expensive method. It will also end up in PNG images, which are not the best option for PDFs - JPEG is better.
In any case, there will always be a case where memory will get exhausted, forcing you to all sorts of fallbacks.
This script inserts JPG images instead of raw pixmaps, giving a file size of 627 MB (the full 1310 pages adobe manual), while memory consumption of the run stayed below 900 MB all the time.

Whereas pixmaps lead to a file size of almost 6 GB (!!!).

import pymupdf

input_doc = pymupdf.open("adobe.pdf")
output_doc = pymupdf.open()  # empty PDF

# buffer to RAM
for i in range(input_doc.page_count):
    out_page = output_doc.new_page(width=100, height=200)
    colorspace = pymupdf.csRGB
    input_pix = input_doc[i].get_pixmap(dpi=150)
    rect = input_doc[i].rect
    h, w = input_pix.w, input_pix.h
    pix = pymupdf.Pixmap(colorspace, w, h, input_pix.samples, False)
    # out_page.insert_image(out_page.rect, pixmap=pix)
    out_page.insert_image(out_page.rect, stream=pix.tobytes("jpg"))

# write to disk
output_doc.save("output.pdf")
output_doc.close()

Instead of JPG images like pix.tobytes("jpg") you can of course use Pillow to convert the pixmap to other formats, or play with JPEG quality, etc.

[milahu]

there will always be a case where memory will get exhausted, forcing you to all sorts of fallbacks

streaming to disk is such a fallback : )

i was just surprised that this is not implemented in any PDF library
given that this is rather trivial to implement
since the xref table is written at the end

obviously this breaks all PDF features which require a full document tree
but there are use cases where such features are not needed
use cases like "convert a million 600dpi images to PDF for printing"

but probably pymupdf is the wrong place for this...
probably this feature would have to be implemented upstream in mupdf

    # out_page.insert_image(out_page.rect, pixmap=pix)
    out_page.insert_image(out_page.rect, stream=pix.tobytes("jpg"))

thanks for the suggestion
but for my use case (binarize-pdf)
it is even more efficient to insert CCITT-G4-compressed images
CCITT-G4 versus JPEG = 13 versus 311 MB = 24x smaller
but i dont see a way to do this with pymupdf

related

• use doxapy, use stream processing rahimnathwani/binarize-pdf#2
• Multi-strip usage when encoding group4 TIF breaks use in PDF python-pillow/Pillow#5740 (comment)