After years of struggling with scaling compute that worked in notebooks but failed in production, we decided to do something about it. Data has standards like Iceberg and Delta. But compute is still a mess—trapped in notebooks, duplicated effort across teams, or baked into custom Airflow DAGs. We think of Xorq as the missing analog to Apache Iceberg, but for compute.
We’ve spent the last year building Xorq, an *compute catalog* that helps teams *reuse, ship, and observe* transformations, features, models, and pipelines across engines.
Xorq is built on:
- *Arrow Flight* (`do_exchange`) for high-speed data transport - *Ibis* for cross-engine expression trees, serialized to YAML - A portable UDF engine that compiles pipelines to SQL or Python - `uv` to make Python environments fully reproducible
Xorq features:
- pandas-style declarative transformations, backed by Ibis - Multi-engine execution (e.g., DuckDB, Snowflake) - UDFs as portable Flight endpoints - Serveable transforms by way of flight_udxf operator - Built-in caching and lineage tracking - Diff-able YAML artifacts, great for CI/CD
Xorq use cases:
Since our last major release, it’s been exciting to see the first Xorq use-cases show up in the wild. All with *Python simplicity and SQL-scale performance*.
- Feature Stores (https://www.xorq.dev/blog/featurestore-to-featurehouse) - Semantic Layers (e.g. https://github.com/boringdata/boring-semantic-layer) - MCP + ML Integration (https://docs.xorq.dev/vignettes/mcp_flight_server)
We’re open source and learning fast. Would love feedback on what’s useful or missing. Thanks in advance for trying it out!
Check out the demo of the Xorq CLI tool in action: https://asciinema.org/a/730484
---
Get Started
- Github: https://github.com/xorq-labs/xorq - Xorq docs: https://docs.xorq.dev/ ---
Sneak peak - Xorq Compute Catalog UI Console:
Check out this interactive Claude demo showing how the Xorq compute catalog can be visualized to accelerate composition, reuse, and troubleshooting of AI compute: https://claude.ai/public/artifacts/d2f00d2a-a3f9-4032-884e-d...
