Mapping the Grammar to Xorq

To illustrate the grammar and expression of the grammar in plain Python, here is how github.py could look, in six lines:

def star_velocity_30d(con, repo):          # TEMPLATE: bind this sentence to any repo
    cutoff = datetime.now() - timedelta(days=30)
    t = con.table("raw_github_events")      # NOUN: a lazy pointer, no computation yet
    return (
        t.filter([                          # VERB
            t.repo_name == repo,
            t.type == "WatchEvent",
            t.created_at > cutoff,
        ])
        .mutate(week=t.created_at.truncate("W"))   # VERB
        .group_by("week")                          # VERB
        .agg(stars=t.id.count())                   # VERB
        .order_by("week")                          # VERB
    )

Every one of these functions and the Makefile lets us run the grammar as steps of the grammatical grammar, building a sentence like this. I added this for illustration, but as an overview, if we map the commands to a xorq call, we can see the connection from the grammar of data to the xorq function:

The Data Engineering Digest: What We Found

If we run the demo project with the 90-day windows (PyPI max provides this window without storing data ourselves), we get a couple of interesting insights that this demo project produces from digesting the full Data Engineering ecosystem. The digest ranks tools and terms of data engineering by their momentum with PyPI download growth and GitHub data, and enriches each tool with its Bluesky chatter on socials¹.

This is how it looks with make digest:

package          growth_pct  recent_daily  total_downloads   buzz
xorq                  130.9           968          125,104     10
ibis-framework         49.5        90,384       13,655,147     15
sqlglot                44.7     2,484,432      379,638,408     17
duckdb                 42.5     1,712,837      263,396,944  3,200
dagster                39.7       283,339       43,951,132    275
polars                 35.8     2,160,110      339,334,602     61
pydantic               30.0    35,567,354    5,688,585,163     90
sqlmesh                28.6        17,964        2,885,574     13
pyiceberg              24.4     1,300,103      211,933,200    640
prefect                19.7       441,434       73,236,617     38
dbt-core                9.9     3,527,875      609,587,211    369
apache-airflow          8.6       702,582      122,123,125    113
dlt                   -13.8       237,792       46,500,226    178

Interesting to see that we get a rise of the dataframe & query engines: the fastest-growing DE packages over the last 90 days are all query/dataframe engines:

• ibis-framework +49%, sqlglot +45%, DuckDB +42%, Polars +36%.

One caveat: I included xorq. It has the biggest growth, but it’s also the smallest package overall and still early, so the growth can have more spikes (we went from ~400 to ~968). And it’s worth mentioning that xorq, the tool we use for the grammar series, uses and is built on ibis, its great expression layer, as xorq builds on a rising dataframe for a composable, in-process engine.

We also see that sqlmesh keeps climbing even after the Fivetran acquisition and dbt Labs joining Fivetran:

• sqlmesh grew +28.6% over 90 days, while dbt-core grew the slowest of the pack (+9.9%).

Not surprising, DuckDB wins social media attention. On Bluesky, its buzz score is 3200, 5× more than the next tool (pyiceberg 640, dbt 369). DuckDB is the tool that is both growing fast and the “loudest”.

Number 1 by raw downloads is pydantic. By absolute volume, pydantic tops everything at 5.64B downloads (~9× dbt-core’s 610M):

── Naive leaderboard: raw PyPI downloads (all-time) ──
package            downloads
pydantic       5,639,574,035
dbt-core         603,522,561
sqlglot          376,710,633
polars           335,901,439
duckdb           261,307,529

This is probably because Pydantic is powering half of PyData while it doesn’t really have a lot of hype, but is a Data Engineering Toolkit for data validation and settings management using Python type annotations, used by any data engineer. It’s a part of the grammar that makes sure re-runs run deterministically.

The Catalog and ML Capabilities

The project has added a catalog to retrieve versioned entries of our metric and created artifacts. With make catalog, this project with xorq-catalog registers a curated set of expressions as versioned, content-addressed entries in a local, git-backed catalog at ./catalog. The catalog.yaml manifest is the shelf where entries are addressed by content hash, and aliases are the human-readable handles:

entries:                     # content hashes (a new hash = a new version)
  - 5adcf6bccba9             # dbt-star-velocity
  - a4f38c87079a             # dbt-download-trend
  - a636a49497fb             # dbt-momentum
  # …
aliases:
  - dbt-star-velocity
  - dbt-download-trend
  - dbt-momentum
  - duckdb-buzz
  - dbt-mentions
  - dbt-health
  - rising-tools

Each entry holds its own metadata sidecar with the expression + metadata + cached result, addressed by hash. Here is dbt-download-trend, the whole metric captured declaratively (kind, output schema, the compiled SQL over the source table, and the cache key):

md5sum: 5a7d75db1e0cf46dc59a713a6ffa9573
backends: [xorq_datafusion]
expr_metadata:
  kind: expr
  schema_out:
    date: timestamp(9)
    downloads: int64
    package: string
  cache_keys:
    key: xorq_cache-snapshot-4788a41e541c0526df207922813c0377
    relative_path: parquet
  sql_queries:
    - - main
      - xorq_datafusion
      - |-
        SELECT "t0"."date", "t0"."downloads", "t0"."package"
        FROM "raw_pypi_downloads" AS "t0"
        WHERE "t0"."package" = 'dbt-core'
          AND "t0"."category" = 'without_mirrors'
          AND "t0"."date" > DATE_TRUNC('DAY', '2026-05-14')
        ORDER BY "t0"."date" ASC

Because xorq bundles the source read at build time, an entry is self-contained: make catalog-run ALIAS=dbt-momentum re-executes it with no re-ingest. Edit an expression (say days=30 → 90) and its content hash changes, so it registers as a new version while the old one stays retrievable.

The project also ships a small ML task (build feature matrix, split, fit a sklearn LogisticRegression wrapped in a xorq Pipeline, predict an adoption label) to mimic prediction and training of a real-life project. It could be interesting to add more sophisticated logic once more data is downloaded, and potentially even more sources are added.

With that example, we use the grammar of data with xorq to get essentially needed steps as part of the Data Engineering Lifecycle. We can have full lineage, we get deterministic reruns, we get the metrics in the catalog. Plus, we get extensibility at each stage if we need it.

E.g. extend the metrics from ‘catalog -> Boring Semantic Layer’, or use dlt for ingestion as I did in this project to load data incrementally into a staging area.

The Lineage

For instance, make lineage shows everything xorq knows about a sentence before it touches a single row: its source nouns, output schema, bound engine, and the verbs compiled to SQL. This is how it looks:

LINEAGE — what xorq knows before any data is read:

>> expr.op().find(DatabaseTable)   — the source nouns this expression reads:
     ['raw_github_events']

>> expr.schema()                   — the output columns (resolved at build time):
     ibis.Schema { week: timestamp; stars: int64 }

>> expr.ls.backends                — the engine(s) bound to it (MODIFIER):
     ['Backend']

>> ibis.to_sql(expr)               — the VERB chain, compiled to SQL:
SELECT "t1"."week", COUNT("t1"."id") AS "stars"
FROM ( SELECT DATE_TRUNC('WEEK', "created_at") AS "week", "id"
       FROM "raw_github_events"
       WHERE "repo_name" = 'dbt-labs/dbt-core' AND "type" = 'WatchEvent' )
GROUP BY "t1"."week" ORDER BY "t1"."week"

If you use the xorq Desktop app, this is integrated into a nice UI like this:

Lineage view in xorq’s upcoming Desktop app

Machine Learning as Another Modifier

The tool-adoption model reuses the same four parts of speech: fit(...) attaches a modifier - the fitted model rides along as metadata (xorq tracks a training_hash) without changing what the expression computes, which is precisely Part 1’s definition of a modifier. predict(...) is just a verb returning an Ibis table expression. And because predict is an expression, xorq build can manifest the inference pipeline too. Deploying a model collapses into the same write → manifest → execute cycle as deploying a metric.

Quick Note on dlt Ingestion

All four sources use idempotent upsert, so re-running only adds new rows / updates existing ones (never duplicates):

• RSS + Bluesky → dlt write_disposition="merge" on primary_key="id" (article URL / post URI)
• PyPI → INSERT OR REPLACE on (package, date, category)
• GitHub → INSERT OR IGNORE on event id

Loading 1 Year on Snowflake

To run a full year or more, just use Snowflake, for example by installing xorq[snowflake] and ingesting the data into Snowflake. Re-running is always safe (idempotent merge), so just pull more and re-run. Each source has its own ceiling:

# 1. More Bluesky history — edit src/de_ecosystem/ingest/bluesky.py
MAX_PAGES = 200            # 40 → 200, pages much further back in time

# 2. More GitHub events — download extra hours into data/raw/gharchive/ (gitignored)
for h in $(seq 0 23); do
  wget -nc -P data/raw/gharchive "https://data.gharchive.org/2026-08-05-$h.json.gz"
done

# 3. Re-ingest (safe, merges) and re-run the whole sentence
make ingest
make run-sentence

The one limit is that the live pypistats.org API only serves ~180 days of downloads, so PyPI momentum caps at a 90-day-vs-prior-90-day window.

A full-year digest needs a source that actually stores that history, and that is where Snowflake (or BigQuery’s public bigquery-public-data.pypi.file_downloads) comes in, both holding years of daily download stats.

Because the grammar separates what from where, you don’t rewrite the metric. You just point the noun at the warehouse and run the same sentence:

uv sync                       # the snowflake driver ships as a base dependency
# put SNOWFLAKE_ACCOUNT / USER / PASSWORD / ROLE / DATABASE / WAREHOUSE / SCHEMA
# in .env (or export them in your shell)
make engines                  # runs star_velocity on DuckDB, DataFusion AND Snowflake

And with the same verb and only the engine binding changed:

con = settings.backend("snowflake")                             # xo.snowflake.connect_env()
momentum = download_momentum(con, "duckdb", window_days=365)    # a full year

The catalog code itself has no engine awareness. settings.backend("duckdb" | "datafusion" | "snowflake") is the only thing that changes. Define once, represent it everywhere: your laptop for a 90-day pulse, a warehouse for the year-in-review.

The one-time grants (de_digest must exist and DLT_LOADER_ROLE needs CREATE TABLE and CREATE STAGE, since the loader stages the tables before copying them in):

-- run once as ACCOUNTADMIN; the demo materialises the 4 raw tables into de_digest.PUBLIC
USE ROLE ACCOUNTADMIN;

CREATE DATABASE IF NOT EXISTS de_digest;   -- PUBLIC schema is created automatically

GRANT USAGE          ON DATABASE  de_digest        TO ROLE DLT_LOADER_ROLE;
GRANT USAGE          ON SCHEMA    de_digest.PUBLIC TO ROLE DLT_LOADER_ROLE;
GRANT CREATE TABLE   ON SCHEMA    de_digest.PUBLIC TO ROLE DLT_LOADER_ROLE;
GRANT CREATE STAGE   ON SCHEMA    de_digest.PUBLIC TO ROLE DLT_LOADER_ROLE;  -- adbc/pandas bulk load
GRANT USAGE, OPERATE ON WAREHOUSE COMPUTE_WH       TO ROLE DLT_LOADER_ROLE;

GRANT ROLE DLT_LOADER_ROLE TO USER loader;   -- if not already

And the tables it created in Snowflake if everything works correctly:

If you got interested and want to know more about how xorq works, check out the docs, or the open source repo on GitHub.

Also check out the upcoming xorq Desktop app (join the waitlist), which targets data analysts from the top down. It’s a desktop app on macOS and a trusted harness. It has additional features that do a verification check and more.