Apache Arrow

Arrow is the columnar in-memory format that ties the engines below together. A RecordBatch produced by Postgres has the same physical layout as one produced by DuckDB or Snowflake, so xorq can move table-shaped values between engines without per-engine deserialization or a pandas round-trip. arrow.apache.org

Arrow shows up in three places in xorq:

• into_backend — table values move between engines as streamed RecordBatches.
• Flight transport — FlightServer serves expressions as Arrow batches over gRPC, consumable from any Arrow-aware client.
• DataFusion + UDFs — Python UDFs, UDAFs, and UDWFs receive pyarrow.Array inputs and return Arrow outputs directly.

→ xorq.flight — Flight server/client built on Arrow IPC

PostgreSQL

Open-source relational database with SQL, transactions, and a mature ecosystem. postgresql.org

→ examples/postgres_caching.py → examples/multi_engine.py

Snowflake

Cloud data warehouse with separated compute and storage. snowflake.com

→ xorq.backends.snowflake → Snowflake Cortex Code: A Shallow Dive

Databricks

Lakehouse platform built on Apache Spark, with Unity Catalog and Databricks SQL on top. databricks.com

→ examples/databricks_into_backend_example.py

DuckDB

In-process analytical SQL engine for columnar data. duckdb.org

→ examples/duckdb_flight_example.py → Augmenting Trino with DuckDB’s AsOf Join

Apache DataFusion

Arrow-native SQL query engine written in Rust; xorq’s default in-process executor. datafusion.apache.org

→ xorq.backends.xorq_datafusion → Harlequin DataFusion adapter

Trino

Distributed SQL query engine for federated queries across heterogeneous sources. trino.io

→ xorq.backends.trino → Augmenting Trino with DuckDB’s AsOf Join and PyTorch

Apache Iceberg

Open table format for large analytic datasets with schema evolution and time travel. iceberg.apache.org

→ examples/pyiceberg_backend_simple.py

SQLite

Embedded SQL database stored in a single file. sqlite.org

→ examples/sqlite_example.py

pandas

Python DataFrame library for in-memory tabular data. pandas.pydata.org

→ examples/pandas_example.py

Mixing engines

into_backend moves table-shaped values between any two of the engines above over Arrow, so a single expression can span Postgres, DuckDB, Snowflake, and friends.

→ examples/multi_engine.py → examples/into_backend_example.py