72 wheels: shipping a Rust extension as a Python package when your dependency has no ABI

The previous post was about an algorithm: cache a Polars query plan once, replay it against a stream of DataFrames, skip the optimizer. That part was fun to build and is maybe 400 lines of Rust. Getting it onto PyPI as a wheel anyone could pip install was harder, stranger, and took most of the actual engineering time, and there’s almost nothing written about this particular problem, because it only shows up when your Rust crate links another project’s internal Rust crates rather than calling a stable C ABI. This is that story.

The ABI problem nobody warns you about

rtlf doesn’t just depend on Polars the Python package. It depends on polars-lazy, polars-plan, polars-stream and nine other polars-* Rust crates directly, because the whole trick (patching the optimized IR arena, hooking the physical-plan executor) only works from inside Polars’ own query-planning code. There’s no public API for any of it.

That means the compiled .so isn’t just linked against “Polars” in the abstract. It embeds DSL_SCHEMA_HASH, a SHA256 of the exact Rust type definitions that make up Polars’ plan IR, baked in at compile time. The installed Python polars wheel has to produce the same hash from its own compiled extension, or the two sides disagree about what a plan node even looks like: not a type error, not a nice Python exception, just a mismatch at import time. There’s no version range that’s “close enough.”

The first complication is that crates.io’s polars versions and PyPI’s polars versions aren’t the same artifact. The published Rust crates lag, diverge, and don’t correspond one-to-one with the wheels PyPI ships. So Cargo.toml patches all 21 polars-* crates (and pyo3-polars, which is built from the same monorepo) straight to a git tag of the polars repository itself:

[patch.crates-io]
polars = { git = "https://github.com/pola-rs/polars", tag = "py-1.38.1" }
polars-arrow = { git = "https://github.com/pola-rs/polars", tag = "py-1.38.1" }
polars-buffer = { git = "https://github.com/pola-rs/polars", tag = "py-1.38.1" }
polars-compute = { git = "https://github.com/pola-rs/polars", tag = "py-1.38.1" }
# … 17 more lines, one per polars-* crate, all pinned to the same tag
pyo3-polars = { git = "https://github.com/pola-rs/polars", tag = "py-1.38.1" }

(Cargo.toml:72-93.) py-1.38.1 is the tag Polars cuts for its own Python release: building against anything else, including a perfectly valid crates.io release, produces a hash mismatch. The [profile.release] block two lines below is the other half of why this build is expensive: codegen-units = 1 and lto = "fat" (Cargo.toml:95-98), full link-time optimisation across a dependency tree that includes most of Polars’ execution engine. That’s not a knob turned up for fun: it’s what makes a Python extension of this size fast enough to be worth shipping, and it’s also exactly what runs out of memory under emulation later in this post.

One wheel, then, is only valid for one exact combination of operating system, architecture, Python minor version, and polars version. Cross any of those and either the wheel doesn’t load (wrong platform tag) or it loads and produces the wrong answer silently (hash happens to still match on an unrelated axis, plan types don’t). The matrix at the bottom of this post is what “cross any of those” costs in practice.

Project layout with maturin

The Cargo side compiles to a cdylib, not a normal Rust library:

[lib]
name = "_rtlf"
crate-type = ["cdylib"]

(Cargo.toml:6-8.) pyproject.toml tells maturin where the Python-facing half of the package lives and what to call the compiled module once it’s built:

[tool.maturin]
# The Rust extension module is placed inside the Python package directory.
# Python source lives under python/ so maturin knows where to find it.
python-source = "python"
module-name = "rtlf._rtlf"
features = ["pyo3/extension-module"]

(pyproject.toml:15-20.) So the compiled extension lands as rtlf/_rtlf (an underscore-prefixed private module), and the public package rtlf is three lines of plain Python re-exporting the two classes the extension provides:

from rtlf._rtlf import PyCompiledRealtimeLazyFrame, PyRealtimeLazyFrame

__all__ = ["PyRealtimeLazyFrame", "PyCompiledRealtimeLazyFrame"]

(python/rtlf/__init__.py.) It’s a small thing, but it’s the difference between users writing from rtlf._rtlf import PyRealtimeLazyFrame (reaching into a module named with an underscore, a module whose name is an implementation detail of maturin’s layout), and writing from rtlf import PyRealtimeLazyFrame. The #[pymodule] on the Rust side matches: _rtlf registers exactly those two #[pyclass] types and nothing else (src/lib.rs:9-14).

pyo3 in practice: four small lessons

The Rust-Python boundary in rtlf is thin (src/python/mod.rs is 91 lines), but it’s dense with pyo3 idioms worth having seen once.

Schema from a Python dict. read_placeholder takes a schema as a plain dict[str, pl.DataType] from Python and has to turn it into a Polars Schema:

fn extract_schema(ob: &Bound<'_, PyAny>) -> PyResult<Schema> {
    let dict = ob.cast::<PyDict>()?;
    dict.iter()
        .map(|(k, v)| {
            let name = k.extract::<PyBackedStr>()?;
            let dtype = v.extract::<PyDataType>()?.0;
            Ok(Field::new((&*name).into(), dtype))
        })
        .collect::<PyResult<Schema>>()
}

(src/python/mod.rs:15-24.) PyBackedStr borrows the Python string’s underlying buffer instead of copying it into a Rust String; PyDataType is pyo3-polars’s wrapper that knows how to read a polars.DataType object back into the Rust DataType enum. The collect::<PyResult<Schema>>() at the end is the usual trick of collecting an iterator of Results into a Result of a collection: the whole dict is either extracted cleanly or the first bad entry short-circuits the loop with a PyErr.

Zero-copy frames. Every PyDataFrame and PyLazyFrame crossing the boundary is a pyo3-polars newtype around the real DataFrame / LazyFrame. unwrap_inputs (src/python/mod.rs:26-28) just unwraps the HashMap<String, PyDataFrame> Python handed over into the HashMap<String, DataFrame> the Rust side wants. No serialization, no Arrow IPC round-trip: it’s the same underlying Arrow buffers, wearing a different Rust type on either side of the call.

Releasing the GIL. Both collect methods wrap the actual work in py.detach:

fn collect(&self, py: Python<'_>, inputs: HashMap<String, PyDataFrame>) -> PyResult<PyDataFrame> {
    let rust_inputs = unwrap_inputs(inputs);
    py.detach(|| {
        self.inner
            .collect(rust_inputs)
            .map(PyDataFrame)
            .map_err(|e| pyo3::PyErr::from(PyRtlfErr::from(e)))
    })
}

(src/python/mod.rs:61-69, and identically at 82-90 for the compiled path.) Without py.detach, the Global Interpreter Lock stays held for the entire Rust-side execution, which, for CompiledRealtimeLazyFrame, is a multithreaded polars-stream query running underneath. Every other Python thread in the process would simply wait. py.detach releases the GIL for the closure’s duration and re-acquires it before returning, so a collect() call from one thread doesn’t stall the interpreter for everyone else.

The orphan rule. This is the one worth pausing on, because it’s a Rust language constraint dressed up as a design choice. rtlf wants PolarsErrors to become Python exceptions automatically via ?, which means it wants:

impl From<PolarsError> for PyErr

Rust won’t allow it. The orphan rule says an impl Trait for Type is only legal in the crate that owns Trait or the crate that owns Type, and rtlf owns neither From/PyErr (that’s pyo3’s) nor PolarsError (that’s Polars’). The fix is the textbook one: wrap the foreign type in a newtype you do own, and implement the conversions on that instead. The whole file is 23 lines:

use polars_core::error::PolarsError;
use pyo3::PyErr;

/// Newtype so we can impl From<PolarsError> for PyErr (orphan rule).
pub struct PyRtlfErr(pub PolarsError);

impl From<PolarsError> for PyRtlfErr {
    fn from(e: PolarsError) -> Self {
        Self(e)
    }
}

impl From<PyRtlfErr> for PyErr {
    fn from(e: PyRtlfErr) -> PyErr {
        use PolarsError::*;
        match e.0 {
            ColumnNotFound(msg) => pyo3::exceptions::PyKeyError::new_err(msg.to_string()),
            ComputeError(msg) => pyo3::exceptions::PyRuntimeError::new_err(msg.to_string()),
            IO { error, .. } => pyo3::exceptions::PyIOError::new_err(error.to_string()),
            other => pyo3::exceptions::PyRuntimeError::new_err(other.to_string()),
        }
    }
}

(src/error.rs, in full.) Two small From impls, and every fallible call site in src/python/mod.rs can write .map_err(PyRtlfErr::from)? or the two-step .map_err(|e| pyo3::PyErr::from(PyRtlfErr::from(e)))? seen above, and get a real KeyError for a missing column or a real IOError for a filesystem problem, rather than one undifferentiated RuntimeError for everything. It’s a small file, but it’s a clean answer to a question every pyo3 project that touches a third-party error type eventually has to ask.

ci/configure.py: a build system that edits its own manifests

Here’s the part that’s unusual even by Rust-packaging standards. There is no single Cargo.toml for rtlf: there’s a Cargo.toml that gets rewritten before every build, by a 129-line Python script, from a table that maps each supported polars release to everything else that has to change alongside it:

# Each entry maps a polars Python release to:
#   rtlf        - the package version published to PyPI
#   crate       - the polars-* crate semver (changes with minor polars releases)
#   nightly     - the Rust nightly that matches the DSL_SCHEMA_HASH for this polars build
#   pyo3        - pyo3 crate version used by this polars build
#   pyo3_polars - pyo3-polars crate version used by this polars build
#
# Versioning scheme: rtlf 0.POLARS_MINOR.POLARS_PATCH
# e.g. polars 1.38.1 → rtlf 0.38.1

(ci/versions.toml:1-9.) Eleven such entries exist as of this post; the CI matrix below builds six of them. ci/configure.py 1.41.1 looks up that row and patches three files in place: Cargo.toml’s crate versions and git tag, pyproject.toml’s package version and polars== pin, and rust-toolchain.toml’s channel: three sources of truth, generated from one table, so nobody hand-edits a version number in the wrong file and gets a wheel that silently expects a nightly that isn’t installed.

The regex that patches Cargo.toml’s dependency versions is anchored to the start of the line on purpose:

# Update crate version numbers in [dependencies]
# Anchored to line start so "pyo3-polars" (contains "polars") is not matched.
# Matches: polars-foo = { version = "0.52", ... }
text = re.sub(
    r'(?m)^(polars[-\w]* = \{ version = )"[^"]+"',
    lambda m: f'{m.group(1)}"{crate_ver}"',
    text,
)

(ci/configure.py:51-58.) pyo3-polars legitimately contains the substring polars, so an unanchored pattern would match it and overwrite its version with the polars crate’s version number instead of its own: wrong, and wrong in a way that would compile fine and fail at runtime. Anchoring to ^ (with re.MULTILINE) means the match has to start a line, and pyo3-polars = {...} starts with pyo3, not polars, so it’s skipped by construction. pyo3 and pyo3-polars each get their own separate, correctly-targeted substitution three lines later.

The other detail worth keeping is the feature rename table:

# Feature renames keyed by the polars version they were introduced in.
# Each entry: (since_polars, old_name, new_name)
# configure.py normalises to old_name for versions before the cutoff and
# new_name for versions at or after, so it is safe to run in either direction.
_FEATURE_RENAMES: list[tuple[str, str, str]] = [
    ("1.41.0", "new_streaming", "streaming"),
]

(ci/configure.py:36-42.) Polars renamed its new_streaming Cargo feature to streaming at 1.41.0. rtlf’s Cargo.toml requests it under one name or the other depending on which polars version this run targets, applied idempotently regardless of which name is currently in the file: running configure.py twice in a row, or running it for an older version after a newer one, converges to the same text either time.

The 72-job matrix, and two battle scars

The workflow’s strategy.matrix crosses four platforms, three Python versions and six polars versions: 4 × 3 × 6 = 72 build-and-test jobs, each running python3 ci/configure.py for its polars version before maturin-action ever sees the manifests. Two of those platform legs earned real comments in the workflow file, and they’re worth quoting exactly as written rather than paraphrased:

env:
  # manylinux2014 cross-toolchain's old gcc doesn't predefine __aarch64__
  # when compiling .S files, so ring's asm_base.h fails its __ARM_ARCH check.
  # Defining it explicitly via CFLAGS is the standard workaround.
  CFLAGS_aarch64_unknown_linux_gnu: ${{ matrix.platform.qemu && '-D__ARM_ARCH=8' || '' }}
  # fat LTO + QEMU emulation OOMs the 7GB runner; thin LTO uses far less memory.
  CARGO_PROFILE_RELEASE_LTO: ${{ matrix.platform.qemu && 'thin' || 'fat' }}

(.github/workflows/ci.yml:56-62.) The first is a cross-compilation quirk one layer removed from rtlf entirely: ring (a transitive dependency, pulled in for TLS) checks __ARM_ARCH while assembling its hand-written .S files, and the manylinux2014 cross-toolchain’s older gcc doesn’t predefine __aarch64__ when it assembles them, so the check fails unless the value is forced explicitly via CFLAGS. The second is the [profile.release] block from earlier catching up with reality: lto = "fat" plus codegen-units = 1 is expensive to link even natively, and under QEMU’s aarch64 emulation on a 7 GB GitHub-hosted runner, that link step runs out of memory outright. The fix isn’t to weaken the release profile everywhere (the fast, native legs keep fat LTO); it’s to drop to thin LTO on exactly the one leg where fat LTO can’t fit in the runner’s memory.

The job-level settings are just as deliberate:

jobs:
  build-and-test:
    runs-on: ${{ matrix.platform.os }}
    continue-on-error: true
    strategy:
      fail-fast: false

(.github/workflows/ci.yml:10-15.) fail-fast: false means one failing cell doesn’t cancel the other 71: useful on its own with a matrix this wide, since a single transient runner problem shouldn’t blank out the whole build. continue-on-error: true goes further: the job itself is allowed to fail without failing the workflow. Some platform × Python × polars combinations are expected not to work, and the point of the matrix isn’t to demand all 72 pass: it’s to find out which ones do. The actions/upload-artifact step has no if: always() on it, so it’s only reached after the test step succeeds:

      # Only reached if tests pass
      - uses: actions/upload-artifact@v4

(.github/workflows/ci.yml:80-81.) That one-line comment is doing real work: it means the publish job, which downloads every uploaded artifact and pushes it to PyPI, can only ever see wheels that were built and imported and passed the test suite on their actual target platform. A wheel that fails to build never gets an artifact. A wheel that builds but fails pytest tests/ never gets an artifact either. continue-on-error lets that failure show up as a red job in the Actions UI without blocking the other 71, but it doesn’t smuggle a broken wheel past the gate.

Publishing itself uses OIDC trusted publishing rather than a stored token:

  publish:
    needs: build-and-test
    environment: pypi
    permissions:
      id-token: write
    steps:
      - uses: pypa/gh-action-pypi-publish@release/v1
        with:
          skip-existing: true

(.github/workflows/ci.yml:86-102, abridged.) No PYPI_API_TOKEN secret to rotate or leak: GitHub Actions mints a short-lived OIDC token scoped to the pypi environment, and PyPI trusts it because the project registered this exact repository and workflow as a publisher. skip-existing: true makes re-runs safe: if 68 of the 72 legs already published successfully and four flaked, re-running the workflow only uploads the four that are missing.

What it costs to depend on someone’s internals

None of this (the patched crates, the version table, the per-leg LTO override, the nightly pin) would exist if rtlf called a stable, public Polars API. It exists because the fastest path to “skip the optimizer” ran directly through Polars’ own internal IR and executor types, which was never designed to be depended on and comes with no compatibility promise attached. The README says as much in its first line: this is an experimental workaround for a specific upstream issue, tested on one polars version, and the whole CI apparatus in this post is the tax for supporting more than that one version without hand-testing all eleven.

That tax is worth paying when the thing you’re building genuinely can’t be built any other way, when the sanctioned API is measurably slower (as the previous post’s ExecutionState-cache detour found out), or doesn’t exist at all, and when you’re honest, loudly and in the first line of the README, about exactly how narrow the ground you’re standing on is. It stops being worth it the moment the internals move and nobody’s watching: a library like this needs the matrix below re-run on every upstream release, forever, or it quietly stops being safe to use. Seventy-two wheels is what “supporting six Polars releases across four platforms” costs when your one Rust crate has to match a hash baked into someone else’s compiled binary. It’s a lot of CI for 480 lines of Rust. Every line of that CI is there because the alternative was either “don’t ship this” or “ship something that only works on my laptop.”

Build Matrix Explorer

ci/configure.py is deterministic: give it a polars version and it tells you the rtlf version, the crate semver, the required nightly, and the pyo3 pins. Cross that with the platform axis from the workflow’s env: block and you get, for any one of the 72 cells below, the exact Cargo.toml, pyproject.toml and rust-toolchain.toml fragments that leg would build with, and, on the QEMU leg, the two comments above, attached to the cell they actually apply to.

With JavaScript enabled this becomes a clickable 4×3×6 grid (platform by Python by polars version) where every cell shows the resolved rtlf version, the Rust nightly, the pyo3 pins, the git tag, and the exact Cargo.toml / pyproject.toml / rust-toolchain.toml fragments ci/configure.py would generate for that build, with a copy button on each. Filters narrow the grid by platform, Python or polars version; the emulated aarch64 row is marked and explains, in the workflow’s own words, why its build settings differ from the other three.