Skip to content

0019. PDF and spreadsheet reading tools

Status

Accepted

Context

Agents in consuming projects are regularly handed PDFs (vendor quotes, invoices, design documents) and Excel workbooks (inventories, cost sheets). The image could read neither:

  • Excel. pandas has been in the image since 2.4.0, but pandas.read_excel() needs a separate engine per format: openpyxl for .xlsx/.xlsm, xlrd for legacy .xls. None was installed, so every call failed with a missing optional dependency error.
  • PDF. There was no PDF tooling at all: no poppler (pdftotext, pdftoppm), no qpdf or mupdf, and no Python PDF library.
  • Claude Code's Read tool uses poppler twice. It runs pdfinfo to count a PDF's pages, and pdftoppm -jpeg -r 100 -f <first> -l <last> to render the page ranges asked for with its pages parameter. Without poppler it cannot count pages, so it sends every PDF whole, and any pages request fails with "pdftoppm is not installed".

Installing at runtime does not fix this. apk add needs root (sudo is scoped to four maintenance scripts), packages pip-installed into the mise Python are lost on rebuild (ADR 0014), and the examples/ egress allowlist ships with PyPI commented out.

Constraints specific to this image:

  • Alpine/musl (ADR 0001) on amd64 and arm64 (ADR 0012).
  • Wheels by choice. Nothing forces pip to use wheels, and the oci-cli pipx install in Layer E already compiles PyYAML from source. The main Layer E pip install has stayed wheel-only anyway, which avoids compile time and fragile musl builds. New packages and their compiled dependencies should ship wheels for CPython 3.14 on both arches.

Decision

Add a read-focused set:

  • poppler-utils via apk in the Layer A package block, unpinned, tracking the alpine:3.24 base like the other apk packages. It provides pdftotext (-layout keeps column alignment), pdfinfo, pdfimages, and pdftoppm.
  • Four Renovate-pinned PyPI ARGs, installed in Layer E with the other mise-Python packages:
  • openpyxl: .xlsx/.xlsm read/write, and pandas' default engine for them.
  • xlrd: legacy .xls, and pandas' default engine for it.
  • pypdf: pure-Python text, metadata, split/merge, and decrypt (it uses the cryptography package already in the image).
  • pdfplumber: layout-aware text and table extraction. It pulls in pdfminer.six, pypdfium2 (PDFium), and Pillow.

pip download --only-binary=:all: for musllinux_1_2_x86_64 and musllinux_1_2_aarch64 (cp314) resolved the full dependency closure wheel-only on both arches before this change was made.

Rejected:

  • odfpy (.ods): unmaintained, with a single sdist from January 2020 and no wheel. It is pure Python and would build, but it would be the only sdist in the main pip install, and .ods files are rare here.
  • python-calamine (.xlsb and .ods): ships musllinux wheels, but pandas uses it only when asked (engine="calamine"), and these formats are rare here. Easy to add if a project needs them.
  • PyMuPDF: AGPL-3.0 (or a commercial licence). Code that imports it takes on AGPL obligations, which consuming projects should not get by default.
  • LibreOffice (headless conversion, formula recalculation): hundreds of MB.
  • OCR (tesseract + pytesseract) for scanned PDFs: a separate need that brings its own language data. Revisit if requested.
  • qpdf: for reading, pypdf already covers decrypting and splitting.

Consequences

Accepted:

  • pandas.read_excel() works out of the box for .xlsx, .xlsm, and .xls. pdftotext -layout file.pdf - and pdfplumber read PDFs, including tables. Claude Code's Read tool can read PDF page ranges.
  • pandas 3.1 deprecates the xlrd engine in favour of python-calamine (pandas-dev/pandas#56542): .xls reads still work but warn, and the image ships no calamine. python-calamine has cp314 musllinux wheels for both arches, so adding it later is cheap.
  • Behaviour change: with pdfinfo available, Claude Code's Read tool refuses whole-file reads of PDFs over 10 pages ("Use the pages parameter"). Agents read long PDFs with pages (up to 20 pages per call, returned as page images) or as text with pdftotext -layout. This is how Claude Code behaves on any machine with poppler installed; before, the limit went unenforced only because poppler was missing.
  • The image grows by roughly 55–60 MB uncompressed. About 45 MB is new wheel content, nearly all of it pdfplumber's dependencies (unpacked: Pillow ~20 MB, pypdfium2 ~13 MB, pdfminer.six ~9 MB), and pip compiles another 6–8 MB of bytecode at install. The apk side is about 5 MB (poppler, poppler-utils, tiff, openjpeg), because the image already ships the other shared libraries poppler links (cairo, fontconfig, freetype, lcms2, libjpeg-turbo, libpng, nss) for Chromium.
  • Renovate tracks four more pins, all in the existing python-tools group. pdfplumber pins an exact pdfminer.six version, so a pdfplumber bump carries it along.
  • poppler is GPL, but it is used only as standalone CLI binaries. pypdfium2 bundles PDFium (BSD/Apache).
  • Scanned (image-only) PDFs have no text layer, so pdftotext, pypdf, and pdfplumber return nothing for them, and the image has no OCR. The Read tool can still show their pages as images. .xlsb and .ods files are not supported.