Skip to main content

Welcome to DataFusion Nexus

DataFusion Nexus is an experimental GPU query engine built on Apache DataFusion. It evaluates relational physical-plan candidates and selectively executes supported workloads on cuDF, while preserving DataFusion execution as the baseline when GPU execution is not selected.

Nexus also exposes specialized GPU algorithms through SQL table functions, allowing results from cuGraph, cuVS to be composed with standard relational operations such as joins, filters, and aggregations. The two families have different execution readiness (see Current limitations). Data can be read directly from local Parquet files or remote Iceberg tables.

Designed to integrate into existing architectures, Nexus provides structured, machine-readable planning decisions, capabilities, and errors, enabling both applications and AI agents to reason about queries before and after execution.

To learn why we built DataFusion Nexus, read DataFusion Nexus: Specialized GPU Algorithms, Now in SQL.

Architecture at a glance​

Nexus selects supported whole relational candidates for in-process cuDF execution, while specialized SQL algorithm calls enter explicit GPU islands; a non-selected relational candidate falls back to DataFusion CPU execution under the same Arrow result contract. KvikIO streams Parquet from local disk or S3-compatible object storage into the GPU, and Iceberg metadata comes from REST or Glue catalogs. Arrow batches return through the caller's original interface.

Features​

  • DataFusion-native GPU execution. Nexus lowers supported physical-plan candidates to a DataFusion-independent cuDF IR. Rewrites normally replace a complete candidate; explicitly supported mixed plans can keep a DataFusion CPU boundary. Unsupported or cost-rejected candidates retain their executable DataFusion plan, and callers can require a final plan with no CPU execution.
  • Graph SQL and GPU function discovery. cugraph_* table functions run PageRank, BFS, Louvain, ForceAtlas2, and other algorithms over relations and return ordinary rows. The shared GPU function catalog lists the cuGraph and cuVS families installed in a session, describes their contracts, validates calls, and reports whether each function is executable.
  • Parquet and Iceberg sources. Native scans read local and S3-compatible Parquet, including Iceberg tables resolved through REST and Glue catalogs. Remote reads use cuDF's KvikIO-backed S3 data source; unsupported native Iceberg scans keep their executable CPU delegate. Remote catalogs are read-only, with an optional mutable local workspace for views and scratch tables.
  • Structured planning diagnostics. Planning reports record candidate outcomes and the completed-plan disposition. nexus_explain_coverage and Flight SQL EXPLAIN GPU expose the same coverage evidence and remedies. Errors retain typed identities at the DataFusion boundary.
  • Bounded per-device execution. Every native attempt requires an immutable GPU memory grant and capability proof. Shapes that cannot establish the required bound are rejected before GPU work begins.

Current limitations​

  • Capability-gated native SQL. Native cuDF execution supports a subset of DataFusion operators, expressions, and Arrow types. Unsupported or cost-rejected candidates retain their DataFusion plan, but failures after GPU execution starts do not replay on CPU. See Native SQL Support and Data Type Support.
  • Single-device query execution. Independent queries can be scheduled across configured GPUs, but each query runs on one GPU; there is no cross-GPU exchange.
  • No disk or host-memory spilling. Native execution runs entirely inside an attempt's immutable device-memory grant (Admission & Memory Governance); there is no spilling of execution state to host memory or disk in the style of cuCascade. Work that cannot fit its grant fails with a structured error instead of degrading.
  • Specialized GPU coverage is uneven. cuGraph is executable. cuVS SQL calls can be discovered, validated, and planned, but bounded execution is rejected until the lower-level peak-memory preflight exists.
  • Native file scans are Parquet-only. Native scans support local and S3-compatible Parquet. Iceberg position deletes are supported when their scan is otherwise admissible; equality deletes, delete vectors, non-Parquet files, and other source types remain on the DataFusion path.

Repository layout​

  • datafusion-nexus — the DataFusion-facing adapter: optimizer rule, table functions, Iceberg integration, and reports.
  • nexus-query-engine — the DataFusion-free native plan and GPU executor.
  • nexus-workloads — adapter-neutral TPC-H, TPC-DS, and ClickBench catalogs, fixtures, dataset discovery, and DataFusion registration.
  • nexus-iceberg — DataFusion-independent Iceberg resolution and scan-fact types shared by the adapter and source integrations.
  • nexus-bench — benchmark, report, triage, and stress binaries.
  • nexus-tools — developer and operator tools, including local end-to-end runners.
  • nexus-examples — runnable embedded-backend examples.
  • nexus-server — the Arrow Flight SQL service and server configuration.

The maintained Rust integration layers are first-class components of this repository: components/cudf is the Rust/libcudf boundary, components/cuvs provides vector-search bindings, and components/cugraph is the Rust/libcugraph boundary behind the cugraph_* SQL functions. Their native sources are the root-pinned submodules under the same directories.

Next steps​

  • Discover & Validate GPU Functions — inspect the installed function families and their current execution capabilities.
  • Design — the integration surfaces, execution boundary, and diagnostic contracts.
  • Integrating Nexus — embedded Rust setup, Flight SQL server startup, client connection, and server configuration.
  • cuGraph SQL API — discover and call the cugraph_* table functions.
  • Build & Test — source builds, Docker packaging, CUDA/compiler prerequisites, environment, upstream native libraries, and the verified test surface.
  • Reference — support boundaries, benchmark evidence, and native execution contracts.
  • Guides — local end-to-end operational flows.