Security & Policy

SQE enforces fine-grained security through LogicalPlan rewriting, injecting row filters and column masks into the query plan before DataFusion’s optimizer runs.

Status: The plan-rewriting policy enforcer is implemented and pluggable, and it is off by default. The default policy engine is passthrough (PassthroughEnforcer), which returns plans unmodified, so enforcement is opt-in. The one enforcement backend is Apache Ranger, with an in-memory store for dev and tests. A default open-source deployment runs without any of this. See GRANT and REVOKE for the SQL surface and the Chameleon / SBP note.

Design Principle

Security enforcement happens at the logical plan level, not at the data level:

graph TB
    SQL["SQL: SELECT * FROM sales"] --> PARSE["Parse"]
    PARSE --> PLAN["LogicalPlan<br/>Projection → TableScan(sales)"]
    PLAN --> POLICY["Policy Enforcer<br/>inject row filter + column mask"]
    POLICY --> SECURED["Secured LogicalPlan<br/>Projection → Filter(region='EU') → TableScan(sales)<br/>+ mask(ssn)"]
    SECURED --> OPT["DataFusion Optimizer"]
    OPT --> EXEC["Execute"]

    style POLICY fill:#f96,stroke:#333

This approach means:

• Row filters are transparent. The user doesn’t know they exist
• Column masks block predicate pushdown on raw values. You can’t WHERE ssn = '123-45-6789' to probe masked data
• Denied columns are invisible. They don’t appear in SELECT *, not as errors
• The optimizer can push user predicates through row filters but not through column masks

Policy Enforcer Trait

#![allow(unused)]
fn main() {
#[async_trait]
pub trait PolicyEnforcer: Send + Sync {
    async fn evaluate(
        &self,
        user: &SessionUser,
        plan: LogicalPlan,
    ) -> Result<LogicalPlan>;
}
}

Implementations:

• PassthroughEnforcer: returns plan unchanged (default; enforcement opt-in)
• Ranger: reads row-filter and column-mask policies from Apache Ranger and feeds the plan rewriter (shipped, wired)
• InMemory: grants stored in a hash map for dev and tests (shipped, wired)

The Ranger enforcer reads the same hive service-def that Apache Spark reads through its Kyuubi authorization plugin, so one policy written once in Ranger enforces byte-identically in SQE and in Spark. See Fine-grained access control for the how-to and Spark / Ranger Parity for the validated result.

SQL Extensions

-- Grant row filter
GRANT SELECT ON sales TO ROLE analyst
  ROWS WHERE region = 'EU';

-- Grant column mask
GRANT SELECT ON customers TO ROLE support
  MASKED WITH (ssn AS '***-**-' || RIGHT(ssn, 4));

-- View effective grants
SHOW EFFECTIVE GRANTS FOR USER "alice";

-- View grants
SHOW GRANTS ON sales;

No Information Leakage

Following the PostgreSQL RLS model:

Runtime Security Controls

SQE includes several runtime security mechanisms that are active by default or can be enabled via configuration.

Rate Limiting

Throttles query submission to prevent abuse or runaway clients. Uses a token-bucket algorithm (via the governor crate).

[rate_limit]
enabled = true
per_user_queries_per_minute = 60
global_queries_per_minute = 1000

When a limit is exceeded, the client receives a RESOURCE_EXHAUSTED Flight error. Rate limiting is disabled by default.

Query Timeouts

Every query is subject to an execution timeout. If the query exceeds the limit, it is cancelled and the client receives an error.

[query]
timeout_secs = 300              # Default: 5 minutes

[query.role_overrides]
admin = 3600                    # Admins get 1 hour
analyst = 600                   # Analysts get 10 minutes

Role overrides allow different timeout limits per role. The user’s longest-matching role timeout wins.

Session Lifecycle

Sessions have both idle and absolute timeouts. A background sweeper runs every 60 seconds to clean up expired sessions.

[session]
idle_timeout_secs = 900         # 15 min idle timeout
absolute_timeout_secs = 28800   # 8 hour hard cap

• Idle timeout: sessions with no query activity for this long are expired
• Absolute timeout: sessions older than this are expired regardless of activity

Query Cancellation

SQE supports Arrow Flight’s native cancellation mechanism. When a client cancels a query (or disconnects), the CancellationToken is triggered and propagated to workers, stopping execution promptly.

Error Sanitization

In production mode (debug = false, the default), error messages returned to clients are sanitized:

• Internal details (stack traces, file paths, internal error types) are stripped
• Clients receive a short error message and a request ID for correlation
• Full details are logged server-side for debugging

Enable debug = true during development to see full error details:

[coordinator]
debug = true

TLS Encryption

Flight SQL connections can be encrypted with TLS. Optional mTLS adds client certificate verification.

[coordinator.tls]
cert_file = "/etc/sqe/server.crt"
key_file  = "/etc/sqe/server.key"
ca_file   = "/etc/sqe/ca.crt"    # Optional: mTLS

See Configuration for details.