Iceberg Integration

SQE is built on the iceberg-rust library (vendored fork) and speaks the Iceberg REST Catalog protocol natively. Iceberg is the only table format SQE supports.

Iceberg version

• iceberg-rust: apache/iceberg-rust main, with the RisingWave fork’s dev_rebase_main_20260807 branch merged on top for the row-level write stack apache does not ship yet: ReplaceFilesAction for Copy-on-Write DELETE and UPDATE, position-delete and deletion-vector writers for Merge-on-Read. Vendored at vendor/iceberg-rust/.
• DataFusion integration: apache/datafusion-iceberg, vendored at vendor/datafusion-iceberg/. SQE’s own scan does not depend on it.
• DataFusion: 55
• Arrow: 59
• Parquet: 59
• Iceberg table format: V2 and V3. V3 is the default for new tables. V3 features verified end-to-end: TIMESTAMP_NS, column defaults, equality-delete UPDATE on identifier-fields, partition evolution, deletion vectors, row lineage, VARIANT, and top-level unknown columns.

The matrix score against icebergmatrix.org is 147/162 (90.7%), rescored in October 2026 against the current rubric, which scores only the cells that apply to each format version. Scores from the older rubric (167/189) are not comparable. See getsqe.com/compare/iceberg for the per-cell breakdown and the side-by-side V2/V3 comparison against every other engine on the public scoreboard.

Architecture

graph TB
    subgraph SQE
        SC["SessionCatalog<br/>per-user token"]
        TP["IcebergTableProvider<br/>DataFusion TableProvider"]
        WR["Writer<br/>Parquet output"]
    end

    SC -->|REST API + bearer or SigV4| CAT["Catalog backend<br/>Polaris, Nessie, Glue REST,<br/>S3 Tables, Unity, HMS, JDBC, Hadoop"]
    CAT -->|table metadata| SC
    CAT -->|S3 credentials<br/>credential vending| SC

    TP -->|read Parquet| OS["Object storage<br/>AWS S3, GCS, ADLS Gen2, R2,<br/>Ceph, MinIO, rustfs"]
    WR -->|write Parquet| OS

    SC -->|commit| CAT

Supported catalogs

SQE keeps the catalog choice as a runtime configuration concern. Every catalog below ships compiled in by default; pick one in [catalog.backend]. See Catalog backends for the full per-backend recipe with TOML examples and verification queries. For where the data files live (S3, R2, GCS, ADLS Gen2, HTTPS, hf://), see Storage backends.

There are two ways into AWS-managed Iceberg. The REST path (type = "rest" with the Glue REST or S3 Tables REST endpoint as catalog_url) speaks the Iceberg REST protocol that AWS exposes on top of these services. The native SDK path (type = "glue" or type = "s3tables") goes directly through aws-sdk-glue or aws-sdk-s3tables without REST. Both work; the SDK path is currently more maintained upstream and avoids the SigV4 signing patches.

The five non-Hadoop backends share one dispatch path through the upstream iceberg-catalog-loader crate. SQE’s for_session_other_backend translates the typed CatalogBackend enum into a uniform (catalog_type, name, props) tuple and hands it to iceberg_catalog_loader::load(type). The loader picks the right CatalogBuilder (REST / Glue / S3 Tables / HMS / SQL) and returns an Arc<dyn iceberg::Catalog>. Hadoop is the lone outlier because it is filesystem-only; its dispatch stays in sqe-catalog/src/backends/hadoop.rs.

Two SQE-only patches sit on top of the vendored loader. The first feature-gates each backend so a slim build does not transitively pull every backend’s AWS SDK / Thrift / sqlx weight. The second adds Send + Sync to BoxedCatalogBuilder so the boxed builder can cross await points in async contexts. Both are documented in vendor/iceberg-rust/README.md and forward-compatible with upstream.

Live integration tests for HMS, Nessie, JDBC Postgres, AWS Glue, AWS S3 Tables, and Unity OSS live in sqe-catalog/tests/backends_integration.rs. Each is #[ignore] and runs against a docker-compose overlay or a real cloud account configured via .env. Glue and S3 Tables are also live-verified (2026-05-05) against AWS account 123456789012 in eu-example-1 and eu-example-2.

The AWS REST endpoints share the OSS Iceberg REST code path. Phase P added an aws-sigv4 cargo feature to the vendored iceberg-catalog-rest crate that swaps the OAuth/Bearer authenticator for an AWS SigV4 signer when rest.sigv4-enabled=true lands in the catalog properties (or in the server’s /v1/config defaults). The signer reads credentials from the standard AWS provider chain.

Catalog REST surface

For Polaris, Nessie, Unity OSS, AWS Glue, and AWS S3 Tables, SQE talks to the catalog via the Iceberg REST API. Key interactions:

Every request includes the user’s bearer token (Polaris, Unity, Nessie, anything OIDC) or is signed with AWS SigV4 (Glue, S3 Tables). The catalog enforces access control.

Credential vending

When SQE loads a table, the catalog returns the table metadata and scoped storage credentials for accessing the data files:

sequenceDiagram
    participant SQE
    participant Catalog
    participant Storage as S3 / Object store

    SQE->>Catalog: Load table (user token / SigV4)
    Catalog-->>SQE: Table metadata<br/>+ scoped storage credentials<br/>(STS / table-scoped key)
    Note over SQE: Credentials are per-user,<br/>per-table, time-limited
    SQE->>Storage: Read Parquet (scoped credentials)
    Storage-->>SQE: Data (allowed prefix only)

This means:

• No service account with broad storage access.
• Each user’s storage access is scoped to exactly the tables they are querying.
• Credentials are short-lived (STS or equivalent).

Read path

graph LR
    PLAN["LogicalPlan<br/>TableScan"] --> META["Load table metadata<br/>from catalog"]
    META --> PRUNE["Partition pruning<br/>(manifest filtering)"]
    PRUNE --> FILES["Data file list"]
    FILES --> READ["Read Parquet<br/>(columnar, predicate pushdown,<br/>row-group skipping, RowFilter)"]
    READ --> BATCH["Arrow RecordBatches"]

Read-side optimizations:

• Partition pruning: Iceberg manifest stats skip whole partitions that cannot match the query predicate.
• Column projection: only requested columns leave Parquet.
• Predicate pushdown: filters land at the row group level, the page-index level, and the Parquet RowFilter.
• Runtime filter pushdown: Phase P shipped a DynamicPredicate API that absorbs DataFusion 54 hash-join build-side runtime filters into the same pruning surface. SF10 TPC-H lineitem-heavy queries saw q06 -51%, q07 -31%, q14 -33%. Engineering log at Runtime filter pushdown.
• Bloom filter consultation: write.parquet.bloom-filter-columns lands bloom offsets in the file footer; DataFusion consults them automatically for literal equality predicates at scan time.
• 5-layer caching: REST catalog cache, table metadata cache, manifest cache, SessionContext cache, OAuth token cache. Warm queries hit sub-millisecond planning.

Write path

graph LR
    SQL["CTAS / INSERT / DELETE / UPDATE / MERGE"] --> EXEC["Execute SELECT"]
    EXEC --> BATCH["RecordBatches"]
    BATCH --> WRITE["Write Parquet<br/>(WriterProperties from table props)"]
    WRITE --> COMMIT["Commit to Iceberg<br/>(append / row-delta / rewrite)"]

Supported DML, both V2 and V3 verified:

• CREATE TABLE AS SELECT: Apache Iceberg V2 and V3, including TIMESTAMP_NS columns and DEFAULT literals.
• INSERT INTO: streaming, with proper schema validation against the catalog.
• DELETE FROM: Copy-on-Write via RewriteFilesAction, or Merge-on-Read via PositionDeleteFileWriter when write.delete.mode=merge-on-read.
• UPDATE: CoW or MoR (equality deletes when the table declares an identifier-field-id).
• MERGE INTO: full WHEN MATCHED / WHEN NOT MATCHED semantics, dispatching to CoW or MoR based on the table’s write.merge.mode (default copy-on-write).
• ALTER TABLE: ADD/DROP/RENAME COLUMN, SET/DROP NOT NULL, type promotion, ADD/DROP/REPLACE PARTITION FIELD (partition evolution), CREATE/DROP BRANCH/TAG, SET WRITE BRANCH.

DELETE reads write.delete.mode, UPDATE reads write.update.mode, and MERGE reads write.merge.mode. Each defaults to copy-on-write. CTAS and INSERT stream RecordBatches straight to Parquet at constant memory. Partitioned writes use a bounded fanout writer by default. It caps the number of open per-partition writers and the buffered bytes, both derived from the memory pool unless fanout_max_open_writers or fanout_buffer_budget is set, and flushes the least-recently-written writer when a limit is hit.

The writer respects write.parquet.bloom-filter-columns and write.parquet.bloom-filter-fpp for any column the schema knows about. The footer-inspection test in sqe-catalog/src/parquet_writer_config.rs proves bloom offsets land in the resulting Parquet file.

Maintenance procedures

SQE runs table maintenance through CALL system.* procedures, each committing through the vendored iceberg-rust transaction actions:

• rewrite_data_files: bin-packs small files into larger ones, with target_file_size_bytes, min_input_files, and max_concurrent_file_group_rewrites options.
• expire_snapshots: drops snapshots older than a cutoff, honouring older_than and retain_last.
• remove_orphan_files: deletes data files no snapshot references.
• rewrite_manifests: compacts manifest files.
• register_table and drop_table: catalog-pointer operations for migration, disaster recovery, and the golden-dataset workflow. drop_table(purge => true) also deletes the table’s files with the caller’s credentials, and refuses when another table shares the location (see CALL procedures).
• set_current_snapshot and rollback_to_snapshot: move the table’s main branch to another snapshot.

V3 features verified

• TIMESTAMP_NS / TIMESTAMPTZ_NS: V3 nanosecond timestamps round-trip end-to-end.
• Variant: VARIANT columns store unshredded variant values written from JSON text, survive CTAS and INSERT … SELECT, and read back as JSON text (details). On a V3 table a JSON column is a variant too, and the json* functions read it. Shredded files written by other engines read the same as unshredded ones; SQE writes unshredded variants and has no variant path functions yet.
• Column defaults: CREATE TABLE ... DEFAULT <literal> applies write_default; ALTER TABLE ADD COLUMN ... DEFAULT applies initial_default.
• Deletion vectors (V3): MoR DELETE on a V3 table writes one deletion vector per data file, never a position-delete file. A second DELETE on the same file merges its existing deletes into a new vector and retires the old one in the same commit, so a data file never carries two. The engine refuses any commit that would add a position-delete file to a V3 table.
• Row lineage (V3): new rows get a fresh _row_id at commit. CoW DELETE / UPDATE / MERGE, MoR UPDATE / MERGE and rewrite_data_files carry _row_id and _last_updated_sequence_number into the files they write.
• Unknown columns (V3): a top-level UNKNOWN or VOID column holds only NULLs, is never written to data files, and is promoted later with ALTER COLUMN ... SET DATA TYPE. A Spark-written VOID column reads as NULLs. Nested unknown fields are refused.
• Equality deletes (V3): UPDATE with a declared identifier-field-id commits a single RowDelta with new data file plus equality-delete row.
• Partition evolution (V3): ALTER TABLE ADD/DROP/REPLACE PARTITION FIELD evolves the spec on V3 tables, including with day(ts) on TIMESTAMP_NS columns.
• Time travel (V3): FOR SYSTEM_TIME AS OF and FOR VERSION AS OF work against V3 tables through the same snapshot walk as V2.
• Schema evolution (V3): ADD COLUMN, DROP COLUMN, RENAME COLUMN, SET DATA TYPE all work on V3.

V3 features not available yet:

• Geometry / geography: no iceberg-rust support, and pending DataFusion UDT #12644.
• Multi-arg partition transforms: PARTITIONED BY accepts single-source transforms only.
• Variant path functions: no variant_get yet.

The deferred list is tracked in docs/evidence/iceberg-matrix-state.json under caveats for each cell.