Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Kubernetes & Helm

SQE includes a Helm chart for production Kubernetes deployment.

Architecture on K8s

graph TB
    subgraph "Kubernetes Cluster"
        subgraph "Coordinator Deployment"
            C1["sqe-server<br/>--mode coordinator"]
        end

        subgraph "Worker Deployment (optional)"
            W1["sqe-server<br/>--mode worker"]
            W2["sqe-server<br/>--mode worker"]
        end

        SVC["Service: sqe-coordinator<br/>ClusterIP"]
        CM["ConfigMap: sqe-config<br/>(sqe.toml)"]
        SEC["Secret: sqe-secrets<br/>(credentials)"]
        SM["ServiceMonitor<br/>(optional)"]

        SVC --> C1
        CM --> C1
        CM --> W1
        CM --> W2
        SEC --> C1
        SEC --> W1
        SEC --> W2
        C1 --> W1
        C1 --> W2
    end

    CLIENT["Clients"] --> SVC
    PROM["Prometheus"] --> SM

Install with Helm

Single-Node (small environments)

helm install sqe deploy/helm/sqe/ \
  --set config.auth.keycloak_url=https://keycloak.example.com \
  --set config.catalog.catalog_url=http://polaris:8181/api/catalog \
  --set secrets.SQE_AUTH__CLIENT_SECRET=my-secret \
  --set secrets.SQE_STORAGE__S3_ACCESS_KEY=minioadmin \
  --set secrets.SQE_STORAGE__S3_SECRET_KEY=minioadmin

Workers are disabled by default. The coordinator runs queries locally.

values.yaml is the easy dev path (production_mode: false). For production, overlay values-production.yaml: it turns on production_mode, rate limiting, catalog.require_vended_credentials, a durable audit PVC, and ServiceMonitor. You must still supply existing Secrets, TLS/ingress, and workerSecret when workers are on. Templating fails if production_mode and worker.enabled are both true with an empty worker secret. The engine also refuses to start if a REST catalog still has require_vended_credentials = false.

production_mode also refuses the two governance defaults: policy.engine = passthrough (no row filters, masks or column restrictions; only catalog-level authorization applies) and access_control.backend = none (GRANT and REVOKE rejected). The overlay selects the Ranger pairing for both; fill in config.policy.ranger.url and provide the admin password as the SQE_POLICY__RANGER__ADMIN_PASSWORD environment override. A deployment that deliberately runs without a policy engine or grant backend says so with config.security.allow_no_policy_engine / config.security.allow_no_grant_backend; the choice is then visible in the values diff, and the coordinator logs the disabled enforcement at warn level on every start.

helm install sqe deploy/helm/sqe/ \
  -f deploy/helm/sqe/values-production.yaml \
  --set existingSecret=sqe-credentials

Distributed (production)

helm install sqe deploy/helm/sqe/ \
  --set worker.enabled=true \
  --set worker.replicas=4 \
  --set coordinator.resources.limits.memory=4Gi \
  --set worker.resources.limits.memory=16Gi \
  --set worker.resources.limits.cpu=8 \
  --set config.auth.keycloak_url=https://keycloak.example.com \
  --set config.catalog.catalog_url=http://polaris:8181/api/catalog \
  --set existingSecret=sqe-credentials

Using an Existing Secret

Create the secret separately (e.g., via sealed-secrets or external-secrets-operator):

apiVersion: v1
kind: Secret
metadata:
  name: sqe-credentials
type: Opaque
stringData:
  SQE_AUTH__CLIENT_SECRET: "my-secret"
  SQE_STORAGE__S3_ACCESS_KEY: "AKIA..."
  SQE_STORAGE__S3_SECRET_KEY: "wJalrXUtnFEMI/K7MDENG..."

Then reference it:

helm install sqe deploy/helm/sqe/ --set existingSecret=sqe-credentials

Values Reference

Image

image:
  repository: sqe
  tag: latest           # Defaults to Chart.appVersion
  pullPolicy: IfNotPresent
imagePullSecrets: []

Coordinator

The coordinator runs as a single replica and is a single point of failure: a restart drops in-flight queries and process-local session state. Coordinator high availability is on the roadmap.

coordinator:
  replicas: 1
  resources:
    requests: { memory: "512Mi", cpu: "500m" }
    limits:   { memory: "2Gi",   cpu: "2" }
  nodeSelector: {}
  tolerations: []
  affinity: {}
  podAnnotations: {}

Workers

worker:
  enabled: false         # Enable for distributed execution
  replicas: 2
  resources:
    requests: { memory: "1Gi", cpu: "1" }
    limits:   { memory: "8Gi", cpu: "4" }
  nodeSelector: {}
  tolerations: []
  affinity: {}
  podAnnotations: {}

Service

service:
  type: ClusterIP
  flightSqlPort: 50051
  trinoHttpPort: 8080
  metricsPort: 9090
  mcpPort: 8098

The Service publishes Flight SQL, Trino HTTP, and metrics, plus mcpPort when MCP is on. healthPort (9091) is a container port used by the probes and is NOT in the Service, so the web UI served there reaches no cluster client through it.

MCP is off by default. Set config.mcp.enabled: true to render [mcp] into sqe.toml and add an mcp container port (config.mcp.port) and Service port (service.mcpPort), both 8098 by default. With ingress.enabled the chart also routes /mcp on the same host, and refuses to render unless ingress.tls is set. config.mcp.auth.mode must be oauth (which needs config.mcp.authorization_servers) or static; the chart refuses anonymous. Set config.mcp.resource_uri to the HTTPS URL clients actually call, because it is the token audience.

Health Probes

healthPort: 9091
livenessProbe:
  initialDelaySeconds: 5
  periodSeconds: 10
readinessProbe:
  initialDelaySeconds: 5
  periodSeconds: 5

Monitoring

serviceMonitor:
  enabled: false
  interval: 30s
  labels: {}            # e.g., { release: prometheus }

Operations

Scaling Workers

kubectl scale deployment sqe-worker --replicas=8
# or
helm upgrade sqe deploy/helm/sqe/ --set worker.replicas=8

Rolling Update

Config changes trigger automatic rolling restarts (via checksum annotation on the ConfigMap):

helm upgrade sqe deploy/helm/sqe/ --set config.catalog.metadata_cache_ttl_secs=60

Interactive SQL

kubectl exec -it deploy/sqe-coordinator -- sqe-cli

Port Forwarding

# Flight SQL
kubectl port-forward svc/sqe-coordinator 50051:50051

# Trino HTTP (for dashboards)
kubectl port-forward svc/sqe-coordinator 8080:8080

# Metrics
kubectl port-forward svc/sqe-coordinator 9090:9090

Logs

kubectl logs deploy/sqe-coordinator -f
kubectl logs deploy/sqe-worker -f

Logs are structured JSON. Pipe to jq for readability:

kubectl logs deploy/sqe-coordinator | jq .

Worker Secret (distributed mode)

In distributed mode the coordinator and every worker share a secret that authenticates worker registration and credential push. The engine refuses to start when coordinator_url / worker_urls is set with an empty worker_secret, so a distributed install without a secret crashloops. The chart renders coordinator_url only when worker.enabled=true, so a single-node install needs no secret.

Provide the secret one of two ways. The chart injects the value under both SQE_COORDINATOR__WORKER_SECRET and SQE_WORKER__WORKER_SECRET from one Secret key, so the guards on both pods are satisfied with matching values.

# Preferred: reference a Secret you manage, naming the key that holds the value.
kubectl create secret generic sqe-worker-secret \
  --from-literal=SQE_WORKER_SECRET="$(openssl rand -hex 32)"

helm install sqe deploy/helm/sqe/ \
  --set worker.enabled=true \
  --set workerSecret.existingSecret=sqe-worker-secret \
  --set workerSecret.key=SQE_WORKER_SECRET

# Dev/test: inline value. The chart creates the Secret for you.
helm install sqe deploy/helm/sqe/ \
  --set worker.enabled=true \
  --set workerSecret.value=dev-only-shared-secret

Availability and Disruption Budgets

The chart ships PodDisruptionBudgets and default pod anti-affinity so a node drain or rolling upgrade cannot take the cluster down at once.

  • Coordinator PDB: minAvailable: 1. The coordinator is single-replica today, so this blocks an unforced eviction of the only coordinator. A node drain that targets it will not proceed until you act (cordon and delete the pod, or scale the deployment to 0 first). The budget protects a single point of failure; it does not provide HA.
  • Worker PDB: maxUnavailable: 1. A drain rolls through one node at a time while the rest keep serving fragments. Rendered only when worker.enabled.
  • Anti-affinity: when a component’s affinity is empty, the chart applies a preferred podAntiAffinity by hostname so replicas spread across nodes. Workers always get it; the coordinator gets it only at replicas > 1. Set affinity to override the default entirely, or defaultAntiAffinity: false to render none.

Toggle the budgets with podDisruptionBudget.enabled (default true) and tune podDisruptionBudget.coordinator.minAvailable / podDisruptionBudget.worker.maxUnavailable.

The coordinator is a single point of failure

Service level we ship: restartable, not HA. One active coordinator. A restart kills in-flight queries and live sessions. Clients retry. Queries do not survive the process. That is the production contract (issue #405). Do not advertise multiple coordinators as HA until there is an external session and query store, plus fencing.

The coordinator runs as a single replica. Sessions, the query tracker, cancellation, caches, CREATE SECRET and ATTACH are process-local. Session-file restore is incomplete: tokens are omitted, so a restored session is not a live one. There is no shared store. A coordinator restart drops every in-flight query and invalidates client sessions, so connected clients must re-authenticate and re-run. A node drain that moves the coordinator pod is a brief outage, not a transparent failover.

The Helm PDB minAvailable: 1 only blocks an unforced eviction. Two replicas behind a generic load balancer are unsafe: they do not share sessions or the registry, so a client would land on a coordinator that never saw its session. Open transactions are process-local too. A Trino transaction id that reaches another coordinator, or the same one after a restart, fails with TRANSACTION_CONFLICT instead of starting an empty transaction, and a graceful shutdown rolls back every open transaction. Keep coordinator.replicas: 1. Full coordinator HA is a later design.