Error Handling Design

Nexus keeps an error typed at the layer that owns the failure and converts it only when crossing a real boundary. Applications should make decisions from typed fields, never from formatted error text.

The typical native-execution path is:

cuDF / cuGraph / cuVS
  -> internal native execution error
  -> datafusion_nexus::error::Error
  -> DataFusionError::External
  -> embedded application or Flight SQL status

Normal native-planning declines do not enter this path. NotSupported and NotSelectedByCost are candidate outcomes in a PlanningReport, not error codes or DataFusionErrors, and the original DataFusion plan remains executable. A malformed plan, an unsatisfied final-plan requirement, or a runtime failure is an error.

Ownership by layer

Layer	Error type	Responsibility
`nexus-query-engine`	`QueryError`	DataFusion-independent IR, admission, execution, GPU resource, and native-library failures.
Root adapter	`datafusion_nexus::error::Error`	DataFusion planning/execution context and projection of native errors into the public adapter taxonomy.
DataFusion traits	`DataFusionError`	Framework boundary only. Nexus errors cross it as a typed `External` source.
`nexus-server`	`ServerError` / `tonic::Status`	Server lifecycle, client-safe messages, gRPC codes, metadata, and retained diagnostics.

Fix a missing classification at its owner. If a cuDF or cuGraph error lacks the typed information Nexus needs, extend the owning component crate instead of parsing its message in the adapter.

One identity, several projections

An adapter Error carries:

Field	Use
`code`	Most specific stable machine identity.
`kind`	Coarse grouping derived from `code`; useful for protocol and aggregate policy.
`status`	Coarse source condition derived from `code`: `Permanent`, `Temporary`, or `Persistent`.
`operation`	Stable dotted identifier for the operation that added this boundary context.
`message`	Human diagnostic text.
`facts`	Structured diagnostic context. Raw by default.
`source`	Original typed cause chain.

kind and status are mappings, not independent classifications. Add a new code first, then update the exhaustive mappings and contract tests.

Do not parse Display or Debug. Adapter Error::Display intentionally emits only its human message. Native QueryError formatting is for trusted in-process diagnostics only and can contain raw facts and source text.

Crossing DataFusion

From<Error> for DataFusionError performs one conversion:

DataFusionError::External(Box::new(nexus_error))

DataFusion may wrap that value in context, diagnostics, shared errors, or error collections. Use DataFusionErrorExt::find_nexus_error() rather than matching one External layer:

use datafusion::common::DataFusionError;
use datafusion_nexus::error::DataFusionErrorExt;

fn record_failure(error: &DataFusionError) {
    if let Some(nexus) = error.find_nexus_error() {
        tracing::warn!(
            code = nexus.code().as_str(),
            kind = nexus.kind().as_str(),
            status = nexus.status().as_str(),
            operation = nexus.operation(),
            "Nexus query failed"
        );
        return;
    }

    tracing::warn!(error = %error, "DataFusion query failed");
}

The search is deterministic and bounded, follows standard source chains, and visits every DataFusionError::Collection branch. A None result means the error has no Nexus identity; keep the application's existing DataFusion policy. There is deliberately no blanket conversion in the other direction.

Creating and wrapping errors

Create the error where the failed contract is known:

Use QueryError for DataFusion-independent engine behavior.
Use adapter Error for DataFusion-specific planning, wrappers, and public integration behavior.
Use Error::from_native when a QueryError reaches the adapter. It maps the native code and preserves native code, kind, status, operation, message, non-colliding facts, and source.
Use the existing typed lower-layer constructors and provenance at cuDF, cuGraph, or cuVS boundaries. Their mappings use enums and status, not message text.

Operations should be stable dotted identifiers. Messages explain the failure to a human. Facts carry machine-readable context needed for policy or diagnosis, and the source preserves the original cause. Do not create a second error taxonomy in a feature module or transport adapter.

Use the result type owned by the current layer: engine Result<T> before the adapter, adapter Result<T> before DataFusion, and DataFusion's result type in DataFusion traits. Let ? perform the single adapter-to-DataFusion conversion.

Once native GPU execution starts, a failure closes the attempt; Nexus does not replay the retained CPU plan. Native-owned failures stay typed, while a DataFusion-owned boundary failure keeps its DataFusion identity.

Retry and recovery are evidence-based

ErrorStatus describes the source condition; it is not permission to retry:

Status	Meaning
`Temporary`	The condition may clear, but the error alone does not prove when or how to retry.
`Persistent`	A dependency failure is known or observed to continue; stop automatic retry and escalate.
`Permanent`	Do not automatically retry the same request. It normally needs different input, configuration, capability, or code; cancellation is also terminal.

There is no adapter Error::is_retryable(). Engine-internal bounded retries are owned by the exact execution operator and recognize only their explicit resource cases.

For a failure returned while admission is still being attempted, use QueryAdmissionOutcome::from_error (or from_error_with_queue_bounds). After a handle has been published, use from_execution_error; a post-admission OOM is an execution failure, not queue capacity that can be retroactively waited on. This projection is policy-neutral: the application still decides whether to queue, shed, or return a protocol error.

For an attempted query, the authoritative recovery signal is the sealed QueryAttemptReport::retry_advice. It is derived only after terminal outcome, GPU cleanup, capacity disposition, device health, native-fault isolation, and resource evidence are committed. Queue evidence can advise backoff; an exact reservation or allocation limit can require configuration change; proven device-local quarantine can allow another eligible device; unknown isolation can require backend recovery. Missing evidence remains indeterminate.

The engine never retries from this advice. A caller must satisfy its paired prerequisite and submit a new attempt. Cancellation, drop, and panic do not become retries, and transport delivery failure does not rewrite the sealed attempt outcome.

Diagnostics and disclosure

Treat raw facts and source chains as trusted in-process diagnostics. They may contain query literals, object paths, identifiers, backend messages, or credentials.

Adapter with_fact stays hidden from log and retained-diagnostic projections. with_log_safe_fact is an explicit opt-in for bounded operational classifications; sanitizing and truncating remain defense in depth.
QueryError::to_json() includes raw native facts and source text. Do not use it as a client response.
Sealed attempt reports project a bounded allowlist from the primary failure; they do not copy arbitrary messages or facts.
Preserve the typed source for internal diagnosis unless the owning boundary deliberately omits it because the foreign text can expose a sensitive location.

Flight SQL boundary

The built-in server first looks for a typed adapter error. Ordinary DataFusion errors receive a server-private classification for gRPC presentation, but that does not give them a public Nexus identity for embedded callers.

The server maps cancellation, auth, not-found, invalid/planning, unsupported, and resource kinds to their matching gRPC codes. Execution, dependency, invariant, and internal failures use INTERNAL. Client messages are narrowed for source, dependency, invariant, and internal failures; logs and retained diagnostics keep the separate structured fields and bounded safe projection.

Every status created from an adapter Error includes these metadata fields:

x-datafusion-nexus-error-code
x-datafusion-nexus-error-kind
x-datafusion-nexus-error-status
x-datafusion-nexus-error-operation

The server also projects only unsupported_column, unsupported_column_type, unsupported_column_role, predicate_role, reason, required_option, and native_message when present. Query and correlation identifiers are added by the statement boundary. The source chain is retained locally, not emitted as metadata. Client-visible fact selection is independent of with_log_safe_fact; a producer of native_message, for example, must treat it as transport-visible data.

Strict unsupported-type rejection

With the NoDataFusionCpu final-plan requirement, an unsupported native column cannot remain on the executable DataFusion plan. The outer adapter error code is final_plan_requirement_unsatisfied, and Flight returns InvalidArgument. For an unsupported list projection, the status message itself identifies the first blocking field in schema order, its exact diagnostic Arrow type, and its stable role:

GPU execution does not support projected column `countDetail` with Arrow type `List(Decimal128(10, 2))`

That status also carries the matching structured metadata:

Metadata key	Example value
`x-datafusion-nexus-reason`	`unsupported_type`
`x-datafusion-nexus-unsupported-column`	`countDetail`
`x-datafusion-nexus-unsupported-column-type`	`List(Decimal128(10, 2))`
`x-datafusion-nexus-unsupported-column-role`	`projected`

The terminal GetQueryStats structured error retains the same facts as fact.reason, fact.unsupported_column, fact.unsupported_column_type, and fact.unsupported_column_role. Flight metadata and query stats are additional machine-readable projections; they do not replace the actionable status message. The existing allowlist and masking rules still apply, so raw facts and the source chain are not exposed.

Protocol policy outside the built-in server belongs to the application. The embedded REST example intentionally maps typed errors before applying its ordinary DataFusion fallback, and does not expose raw facts wholesale.

Extending the contract

When adding an error:

Add the most specific code at the owning layer.
Update code-to-kind and code-to-status mappings; do not infer either from the variant name.
Map native codes exhaustively at the adapter boundary while preserving the original native classification.
Decide explicitly whether any fact is raw, log-safe, attempt-report-safe, or Flight-visible. These are separate disclosure choices.
Extend the existing error, admission, attempt-report, or Flight contract tests. A useful regression test must fail when the mapping or boundary behavior is reverted.

The canonical implementation points are src/error.rs, src/error/code.rs, crates/nexus-query-engine/src/error.rs, crates/nexus-query-engine/src/observability/attempt_report.rs, and crates/nexus-server/src/error.rs.

Ownership by layer​

One identity, several projections​

Crossing DataFusion​

Creating and wrapping errors​

Retry and recovery are evidence-based​

Diagnostics and disclosure​

Flight SQL boundary​

Strict unsupported-type rejection​

Extending the contract​