Skip to content

Lazy frames

Every node so far has returned a materialized pl.DataFrame — and for most apps that is exactly right, because the registry is the memo cache: a stored frame is reused for free on every interaction that doesn't touch it. But a chain of transform nodes pays a cost eager frames can't avoid: each node boundary materializes an intermediate frame, even when the next node immediately filters most of it away.

A node may instead return a pl.LazyFrame. The plan flows downstream unmaterialized, Polars fuses the whole chain at the point it is finally collected, and its optimizer — predicate pushdown, projection pushdown, slice pushdown — works across your node boundaries, not just within one node's body.

Keep the source eager, make the transforms lazy, and let the view collect:

import polars as pl
from golit import App, create_app, slider

app = App(title="Lazy pipeline")


@app.source
def data() -> pl.DataFrame:
    # Eager and memoized: loaded once, cached in the registry across interactions.
    return pl.read_parquet("sales.parquet")


@app.reactive
def filtered(data: pl.DataFrame, threshold: int = slider(0, 100, default=20)) -> pl.LazyFrame:
    # A plan, not a frame — built in microseconds, nothing materializes here.
    return data.lazy().filter(pl.col("revenue") > threshold)


@app.reactive
def by_region(filtered: pl.LazyFrame) -> pl.LazyFrame:
    # Extends the upstream plan; still nothing materializes.
    return filtered.group_by("region").agg(pl.col("revenue").sum())


@app.view
def table(by_region: pl.LazyFrame) -> pl.LazyFrame:
    return by_region  # Golit collects at the render boundary

On a slider move, the dirty subgraph re-runs filtered and by_region — which only rebuild the plan, near-free — and the single real execution happens once, fused and optimized, when the view renders. The source never reloads: it's clean, so its cached frame is reused, and the lazy plan reads it from memory.

A view returning a LazyFrame renders as a table, and the collection is slice-aware: the table shows at most 50 rows, so Golit collects head(51) — slice pushdown makes that cheap — and runs a projection-pruned count query only when the frame proves to extend past the table. ui.table(...) accepts a LazyFrame the same way. Any other use (a chart, a custom component) collects explicitly in the view: frame.collect().

What the framework does for you

  • Plan hashing. hash_value fingerprints a LazyFrame by its plan, never by collecting it. That is the right semantics inside the graph: a lazy value changes either because an upstream changed (caught by epochs) or because the plan itself changed (caught by the plan hash).
  • Fan-out materialization. A lazy plan is executed by whoever collects it — so if one lazy node fed three views, each view's collect would re-execute that node's work. Golit knows each node's consumer count from the graph, and materializes a LazyFrame once at its producer whenever more than one node consumes it. The registry becomes the shared memo again, exactly as if the node had returned an eager frame. With a single consumer the plan stays lazy and fuses downstream — the optimal case — and you never have to think about which of the two happens.
  • Pollers materialize. An @app.poll source exists to detect content change in external data, and a plan hides content — the same plan over a mutated file hashes identically. A poller that returns a LazyFrame is collected (off the event loop) before hashing, and downstream nodes receive the materialized frame.

Larger-than-memory data: the streaming engine

Because collection happens at well-defined boundaries, Polars' streaming engine — which executes a plan in batches instead of all at once — drops straight in. Golit collects with engine="auto", which respects Polars' own global affinity setting:

import polars as pl

pl.Config.set_engine_affinity("streaming")   # or: export POLARS_ENGINE_AFFINITY=streaming

Pair that with a scanned source and the full pipeline streams:

@app.source
def data() -> pl.LazyFrame:
    return pl.scan_parquet("events/*.parquet")   # nothing loads at startup

A lazy source like this trades the in-memory memo for out-of-core execution: every downstream collect re-scans the files (with pushdown, so a filtered table reads only what it needs). Reach for it when the data genuinely doesn't fit in memory; otherwise the eager-source shape above is faster.

When to stay eager

  • The data is small. Plan optimization has overhead too; for a few thousand rows the eager chain is already sub-millisecond and simpler to debug.
  • The node's output is expensive and shared. The fan-out rule already materializes shared lazy outputs for you, but if you want the memo and explicit control, just return the eager frame — that's the classic Golit shape.
  • Schema-dependent operations. A pivot can't run lazily without a declared schema — Polars requires a static schema before collection. Collect first, or keep that node eager.