Demo Dataset: Citation Network

The per-algorithm Examples in this section all run against one dataset: the DBLP / AMiner Citation Network V12 — 4,894,081 papers connected by 45,564,149 directed citation edges (src cites dst). It is small enough to fit on one GPU and detailed enough to support substantive queries: every vertex is a paper with a title, year, venue, authors, and fields of study, so every algorithm result can be joined back to human-readable metadata.

Dataset vintage

The V12 snapshot was compiled on 2020-04-09, so this corpus is old by design: it stops in early 2020, misses everything published since, and its citation counts reflect that moment rather than today. Treat every example result as a property of this fixed snapshot, not a statement about the current literature.

The examples were re-run against the local Iceberg REST catalog backed by RustFS on a single NVIDIA RTX PRO 6000 (96 GB VRAM). That machine is the validation host, not the minimum requirement: this citation-network demo also runs on an 8 GB VRAM GPU. The same SQL can still read direct local Parquet tables, but the primary demo path is the Iceberg namespace used below.

Tables

The primary demo source is the local Iceberg REST namespace lake.citation_network. It contains five Iceberg tables loaded from the Parquet files generated by fixture/graph/dblp_ingest.py (see fixture/README.md in the repository):

Table	Rows	One row is
citation_edges	45.6M	a citation edge (src, dst, weight=1.0), clustered by src
citation_edges_by_dst	45.6M	the same edges, clustered by dst for reverse traversal
papers	4.9M	a paper: paper_id, title, year, venue, n_citation, n_references, primary_fos, …
paper_authors	14.9M	a paper–author link
paper_fos	45.0M	a paper–field-of-study link

Load the namespace with the local REST/RustFS stack:

docker compose -f fixture/iceberg-local/docker-compose.yml up -d
fixture/fixture.sh iceberg rest load \
  --workload citation_network \
  --load-mode add-files

add-files stages the generated Parquet files into RustFS and registers them with Iceberg in place. Do not delete the staging prefix while the REST fixture metadata is live. The local REST fixture stores catalog metadata in ephemeral SQLite, so rerun the loader after every fresh docker compose up.

To serve those tables through the Flight SQL server, use the citation demo server recipe. The server recipe enables cuGraph SQL, binds to localhost, and installs the workspace overlay that makes these tables available as unqualified names.

Keep example DDL in the local workspace

The workspace overlay makes unqualified names such as citation_edges and papers resolve through to lake.citation_network.*, while interactive DDL stays local to datafusion.public. Prefer CREATE VIEW for SQL-defined subgraphs. Use CREATE TABLE ... AS only when an example explicitly needs a stable local snapshot of non-deterministic algorithm labels; it does not write back to the Iceberg source catalog.

For direct-Parquet debugging without the REST catalog, start the server with Iceberg disabled and register the same logical names manually:

CREATE EXTERNAL TABLE citation_edges STORED AS PARQUET LOCATION '<data>/parquet/edges_by_src.parquet';
CREATE EXTERNAL TABLE citation_edges_by_dst STORED AS PARQUET LOCATION '<data>/parquet/edges_by_dst.parquet';
CREATE EXTERNAL TABLE papers STORED AS PARQUET LOCATION '<data>/parquet/profiles_by_paper_id.parquet';
CREATE EXTERNAL TABLE paper_authors STORED AS PARQUET LOCATION '<data>/parquet/paper_authors.parquet';
CREATE EXTERNAL TABLE paper_fos STORED AS PARQUET LOCATION '<data>/parquet/paper_fos.parquet';

AWS Glue can serve the same table layout for remote demos, but it requires an AWS account, a warehouse bucket, and credential setup. The local REST catalog is the reproducible path for the website examples.

Int32 renumbering for cuGraph C API algorithms

Two cuGraph C API algorithms (cugraph_spectral_modularity_maximization, cugraph_minimum_spanning_tree) require Int32 vertex columns. AMiner paper ids do not fit: 42% of them exceed 2,147,483,647 (the maximum id is 3,009,038,462), so a plain CAST overflows. The examples renumber in SQL instead — materialize a dense mapping once, then join it onto the edge view:

CREATE TABLE my_vertex_ids AS
SELECT paper_id, CAST(ROW_NUMBER() OVER (ORDER BY paper_id) AS INT) AS vid
FROM (SELECT src AS paper_id FROM my_edges UNION SELECT dst FROM my_edges);

Materializing with CREATE TABLE ... AS (not a view) matters: it keeps the window function out of the cuGraph table-function input, and the mapping table joins results back to papers afterwards.

The two-table pattern

cuGraph functions consume an edge relation and return vertex ids. The examples follow one pattern throughout:

1. SQL defines the graph the GPU sees — a CREATE VIEW with joins and WHERE clauses builds the subgraph (an era, a field, an ego network), and the view name is passed directly to the cugraph_* function.
2. SQL turns vertex ids back into results — the (vertex, value) output is joined to papers, windowed, aggregated, or anti-joined like any table.

Seed papers used in the examples

paper_id	Paper
2963403868	Attention Is All You Need (2017)
2896457183	BERT (2018)
2163605009	AlexNet (2012)
1686810756	VGG (2014)
2194775991	ResNet (2016)
2064675550	Long short-term memory (1997)
2066636486	The anatomy of a large-scale hypertextual Web search engine — the PageRank paper (1998)

Two practical caveats

Snapshot non-deterministic labels before self-joining

Component and community ids (label, partition) are assigned per execution. A view over a cugraph_* call re-runs the algorithm on every scan, so a query that scans such a view twice (for example a CTE joined back to it) can join mismatched labels. If one statement cannot keep the algorithm output single-use, materialize once into the mutable datafusion.public workspace with CREATE TABLE <name> AS SELECT …, then analyze the snapshot. That snapshot is local session/workspace state, not an Iceberg table write.

n_citation counts the whole world, in-degree counts this corpus

papers.n_citation is AMiner's global citation count; an in-graph degree only counts edges present in this dataset. The in-degree example measures exactly that gap.