CALL procedures

Iceberg maintenance operations exposed as SQL CALL statements. SQE wraps the vendored iceberg-rust action APIs (RewriteFilesAction, RemoveSnapshotAction, RewriteManifestsAction) and adds an SQE-specific bloom-filter suggestion procedure that walks recent query history.

All procedures use Iceberg’s named-argument syntax: CALL system.<proc>(name => value, ...). Unknown argument names raise a parse error so typos fail fast.

Source: sqe-sql/src/procedures.rs. Handlers in crates/sqe-coordinator/src/maintenance.rs.

Reference

Comparison to other engines

The Spark and SQE shapes are aligned: Spark uses t.system.<proc> (table-qualified), SQE uses system.<proc>(table => 'ns.t') (named arg). Both are explicit. Trino prefers EXECUTE-as-DDL syntax which is harder to script.

Examples

Compact a partitioned fact table

CALL system.rewrite_data_files(
    table => 'analytics.events',
    target_file_size_bytes => 268435456,    -- 256 MiB
    min_input_files => 8
);

Returns one summary row:

+----------------------+----------------------+----------------------+
| files_rewritten      | bytes_rewritten      | snapshot_id          |
+----------------------+----------------------+----------------------+
| 142                  | 39283744832          | 8472810294831234567  |
+----------------------+----------------------+----------------------+

Sort-compact for read pruning

Load fast (unsorted), then compact into sorted files once. The sort strategy gathers a whole partition into one stream, orders it by sort_order through a spillable DataFusion sort, and rolls the output at target_file_size_bytes. Sorting the partition as a single stream is what makes the result prunable: the output files come out with disjoint key ranges instead of each file spanning the full domain. The sort spills to disk, so the rewrite stays memory-bounded even when a partition is larger than RAM. Unlike bin-pack, the sort strategy also rewrites files already at or above the target size, because they still have to be re-laid-out to join the sorted layout.

-- Lexicographic sort on one or more columns.
CALL system.rewrite_data_files(
    table => 'analytics.events',
    strategy => 'sort',
    sort_order => 'event_date ASC, user_id ASC'
);

-- Z-order clustering for multi-dimensional locality.
CALL system.rewrite_data_files(
    table => 'analytics.events',
    strategy => 'sort',
    sort_order => 'zorder(user_id, device_id)'
);

Sorted files give the reader tight min/max stats per file, so predicate pruning skips more files. Z-order clusters several columns at once, which helps when queries filter on different subsets of those columns. Iceberg’s sort-order metadata cannot express z-order, so none is stamped for the z-order case (matches Spark).

Verify the layout with table_files: after a sort compaction the lower_bounds / upper_bounds of the output files should not overlap on the sort column.

SELECT file_path, lower_bounds, upper_bounds
FROM table_files('analytics', 'events')
ORDER BY lower_bounds;

Clean up delete-heavy Merge-on-Read files

On a Merge-on-Read table, repeated DELETE/UPDATE/MERGE accumulate delete files. A data file with many deletes is slow to read (every delete file has to be applied on scan). delete_file_threshold rewrites any data file with at least that many delete files applying to it, even when the file is already at or above the target size, so bin-pack would otherwise leave it alone.

CALL system.rewrite_data_files(
    table => 'analytics.events',
    delete_file_threshold => 10
);

The count includes every delete file the scan attaches to the data file, both position and equality deletes. A low threshold on an equality-heavy table therefore rewrites broadly, since one equality delete can apply to many files. The option is off by default and is a no-op under strategy => 'sort', which already rewrites the whole partition.

On a format-version 3 table (the default for new tables) position deletes are deletion vectors: each data file has at most one, and every DELETE replaces it. A threshold above 1 is therefore never reached by SQE’s own deletes, however many rows they remove; only equality deletes from other writers still add up. On V3, use delete_ratio_threshold instead. It rewrites a data file once its deleted rows reach that share of its rows:

CALL system.rewrite_data_files(
    table => 'analytics.events',
    delete_ratio_threshold => 0.3
);

The share is the deletion vector’s cardinality over the data file’s row count, so a file with 3 of its 10 rows deleted qualifies at 0.3. It also counts V2 position delete files written for a single data file. When a file has both kinds, only the deletion vector counts, because it already holds the older deletes. Equality deletes and position delete files that span several data files are not counted, because their row count does not say how many of this file’s rows they remove; delete_file_threshold still sees them. The value must be greater than 0 and at most 1. 0.3 is a reasonable start; Spark’s delete-ratio-threshold uses the same default. Both thresholds can be given in one call, and a file meeting either one is rewritten. Like the count, the ratio is off by default and is a no-op under strategy => 'sort'.

rewrite_all => true, or a count threshold of 1, still rewrites every data file that has any deletes.

Override the distribution mode for one call

distributed => 'auto'|'local'|'require' overrides [maintenance.distribution] mode (see Configuration) for this one CALL, without touching the coordinator’s config file. 'require' fails the call immediately if fewer than min_workers workers are currently healthy, rather than silently falling back to a coordinator-local rewrite:

CALL system.rewrite_data_files(
    table => 'analytics.events',
    distributed => 'require'
);

'local' forces a coordinator-local rewrite even with a healthy fleet present, useful for a one-off run you want to keep off the workers (a small table, or a maintenance window where the fleet is busy with query traffic). Omitting distributed entirely uses the configured [maintenance.distribution] mode. See Distributed compaction for how a distributed call plans, dispatches, and commits.

max_concurrent_file_group_rewrites only bounds concurrency on the coordinator-local path; a distributed rewrite is instead bounded by [maintenance.distribution] max_inflight_groups_per_worker (per-worker, not global), configured in Configuration.

Force a full rewrite

rewrite_all => true rewrites every data file, including files already at or above the target size and partitions below min_input_files. It applies all deletes and re-encodes at the target size. Use it to force a clean pass after a schema or partition-spec change, or to apply accumulated deletes across a whole table in one commit.

CALL system.rewrite_data_files(
    table => 'analytics.events',
    rewrite_all => true
);

Because it re-encodes everything, it costs a full read and write of the table. It is off by default and subsumed by strategy => 'sort', which already rewrites the whole partition. rewrite_all is supported on both the coordinator-local and distributed paths: it forces every file into the group plan and bypasses the min_input_files floor on either path.

Check compaction debt before deciding whether to run a rewrite

table_health is read-only: it reuses the same file-collection and bin-pack logic rewrite_data_files uses to plan a rewrite, but never writes a file or commits a snapshot. It bypasses the write-privilege gate entirely, so a SELECT-only session can run it.

CALL system.table_health(table => 'analytics.events');

Returns one summary row:

+-----------------+-------------+----------------+----------------+--------------+---------------------+------------------+--------------------+-----------------------------+----------------------+
| live_data_files | small_files | avg_file_bytes | p50_file_bytes | delete_files | delete_heavy_files  | eligible_groups  | est_rewrite_bytes  | last_compaction_snapshot_ms| maintenance_enabled  |
+-----------------+-------------+----------------+----------------+--------------+---------------------+------------------+--------------------+-----------------------------+----------------------+
| 1842            | 611         | 41943040       | 33554432       | 96           | 12                  | 7                | 2248146944         | NULL                       | true                 |
+-----------------+-------------+----------------+----------------+--------------+---------------------+------------------+--------------------+-----------------------------+----------------------+

Column notes:

• small_files counts live data files below [maintenance.compaction].target_file_size_bytes (default 512 MiB).
• eligible_groups / est_rewrite_bytes report pure bin-pack debt: groups that meet min_input_files on file count alone. delete_heavy_files is a separate signal, files with at least delete_file_threshold delete files applying to them, or a deleted-row share of at least [maintenance.compaction] delete_ratio_threshold. table_health reads both thresholds from [maintenance.compaction], not from a CALL argument. A later rewrite_data_files(delete_file_threshold => N) call rewrites the union of both sets, so treat the two counts as additive, not eligible_groups already including delete-heavy files.
• last_compaction_snapshot_ms is always NULL. Active-mode compactions do stamp sqe.maintenance.job-id/principal/trigger onto the snapshot they commit (see the system.rewrite_data_files note above), but table_health does not yet read that snapshot property back; check the table’s snapshot history directly for compaction attribution until a later phase wires this column up.
• maintenance_enabled reflects the sqe.maintenance.enabled table property, i.e. whether the advisory/active scheduler would even consider this table. It does not mean a rewrite ran: advisory mode never mutates, and active mode may still find no eligible compaction debt on a given tick.

Drop snapshots older than 30 days, keeping the last 10

CALL system.expire_snapshots(
    table => 'analytics.events',
    older_than => TIMESTAMP '2026-04-08 00:00:00',
    retain_last => 10
);

The retain_last floor is enforced even when older_than would clear more. Useful for keeping a rollback budget while clamping storage growth.

Bloom filter suggestion before a tuning pass

CALL system.suggest_bloom_filter_columns(
    table => 'analytics.events',
    history_limit => 5000
);

Returns one row per column with a positive equality-predicate count, ranked descending:

+----------+-------------------+------------------+
| column   | equality_pred_hits | recommendation  |
+----------+-------------------+------------------+
| user_id  | 4823              | strongly suggested |
| event_id | 1241              | suggested         |
| device   | 312               | weak             |
+----------+-------------------+------------------+

Apply with:

ALTER TABLE analytics.events SET TBLPROPERTIES (
    'write.parquet.bloom-filter-columns' = 'user_id,event_id'
);

The next write picks up the new property; existing files are unaffected until rewritten.

Combined maintenance run

-- Once a week, in this order:
CALL system.expire_snapshots(table => 'analytics.events',
    older_than => TIMESTAMP '2026-04-08 00:00:00', retain_last => 30);
CALL system.remove_orphan_files(table => 'analytics.events',
    older_than => TIMESTAMP '2026-04-08 00:00:00');
CALL system.rewrite_manifests(table => 'analytics.events');
CALL system.rewrite_data_files(table => 'analytics.events');

Order matters: expire snapshots before removing orphan files (otherwise files referenced by snapshots about to expire look orphaned), and rewrite manifests before rewriting data files (so the rewrite plan reads compact manifests).

Permissions

Procedures are gated in three tiers, by blast radius. Polaris grants on the target table still apply underneath every tier.

The admin tier exists because these procedures delete data files or history without a catalog commit Polaris could refuse, or move the table’s main branch for everyone. They are administrative in Spark and Trino as well.

A denied call fails with 403 Forbidden and is recorded in the audit log as AdminGateDenied.

Safety notes

• remove_orphan_files with no older_than uses the 3-day default, which is conservative against compaction or COPY jobs in flight. Override with older_than only after confirming no concurrent writers.
• expire_snapshots is destructive for time-travel queries. Once a snapshot is expired, FOR VERSION AS OF <id> for that snapshot fails. Document a retention window your team agrees on, and stick to it.
• rewrite_data_files rewrites entire data files, not row groups. Two consecutive calls can churn the same files; rely on the min_input_files floor (default 5) to keep churn bounded.
• rewrite_data_files is delete-aware on Merge-on-Read tables. It reads each file group through the Iceberg scan, so position and equality deletes are applied during the rewrite and deleted rows never reappear. The compacted output is pinned to the sequence number of the snapshot it read, so an equality delete another writer commits mid-compaction still applies to the compacted files. Fully-covered position delete files are dropped in the same commit; equality deletes are left to age out via expire_snapshots. It groups files per partition, so partitioned tables consolidate within each partition.
• rewrite_data_files retries on conflict. A concurrent writer that commits between the read and the commit produces a retryable conflict; the procedure re-reads and retries with backoff a bounded number of times before surfacing the conflict.
• drop_table(purge => true) cannot be undone. It refuses, before dropping anything, when the table sets gc.enabled=false; when its location is a bucket root; when a write.data.path, write.metadata.path or similar property, or the current metadata file, points outside the location; or when any other table in the catalog uses the same location, a location inside it, or one containing it (for example a second identifier created with register_table). Drop the other table first, or drop without purge. SQE does not use the REST catalog’s own purge flag, which in Polaris needs DROP_WITH_PURGE_ENABLED and deletes later with the catalog’s credentials. If the catalog drop succeeds but the file delete fails, the error names the prefix to delete by hand.
• Run procedures in a quiet window. A concurrent writer that commits mid-run can cause rewrite_data_files to return a retryable error. The other procedures tolerate concurrency and reconcile against the live snapshot.

Commit failures

Every procedure commits through the same REST catalog path that CTAS and INSERT use, so commit failures surface as SqeError::Execution and fall into two buckets:

• Retryable. The message contains conflict or retry. rewrite_data_files already re-reads and retries a bounded number of times internally; a retryable error surfaced to the caller means those attempts were exhausted, so schedule another run after a back-off. The other procedures surface the conflict directly.
• Permanent. Everything else. Check the message for the upstream cause.

What is not exposed

The vendored iceberg-rust crate has more transaction actions than SQE wires up. Notable omissions:

• expire_snapshots_by_id (drop a specific snapshot rather than by age). easy to add if needed.
• rewrite_position_deletes (compact MoR delete files). not yet wrapped; on the V13 backlog.
• cherrypick_snapshot (apply a non-current snapshot’s changes to the head). out of scope for now; rare use case.

File an issue if you hit one of these in production.