Kubernetes Deployment

This guide covers deploying OpenSandbox on Kubernetes: the cluster foundation (CRDs), the controller, the lifecycle server, and the optional components (ingress gateway, node agent, fast-sandbox runtime).

Charts are versioned sources in the OpenSandbox repository. They are installed from a checkout of a release-X.Y.Z tag — standalone chart packages are not published.

Prerequisites

• Kubernetes 1.21.1+
• Helm 3.x
• kubectl configured for your cluster

Deployment Order

Install the charts in this order:

base → opensandbox-controller → fast-sandbox* → ingress-gateway → opensandbox-server → optional components

* fast-sandbox is optional, but when used it must be installed before the ingress gateway and the server. ingress-gateway is required for Kubernetes deployments — sandbox Pods are ClusterIP-only, so client traffic routes through the gateway. It is installed before the server so the server announces it from the first install.

Step	Chart	Why it comes here
1	base	Owns the sandbox.opensandbox.io CRDs, the fast-sandbox CRDs (sandbox.fast.io), and their RBAC. Everything else depends on these objects existing. Install once per cluster.
2	opensandbox-controller	Reconciles BatchSandbox, Pool, and SandboxSnapshot objects created by you and by the server. Requires the CRDs from base.
3	fast-sandbox (optional)	Firecracker runtime (sandbox.fast.io). Must be installed before the server: the server's [runtime]/fsb configuration points at the FastPath gRPC endpoint (fast-sandbox-fastpath...svc:9090) and watches sandbox.fast.io objects at startup. Also consumes the ServiceAccounts from base — keep the namespace values in sync with it. Skip if you only use the default Kubernetes runtime.
4	ingress-gateway	Required for Kubernetes deployments: sandbox Pods are ClusterIP-only and client traffic routes through the gateway. Installed before the server so the announcement is configured from the first install. When serving sandboxes through the fsb runtime, point gateway.fastpathEndpoint at the FastPath service from the previous step.
5	opensandbox-server	The lifecycle REST API that creates and deletes sandboxes. Requires the CRDs from base, a running controller, and — when serving sandboxes through the fsb runtime — the FastPath endpoint from the fast-sandbox release. Announces the ingress gateway through server.gateway.*.
6	opensandbox-node-agent (optional)	Node-level sandbox data collection. Order-independent.

If you use the umbrella chart, this order is handled for you in a single release.

Install

Check out the version you want to deploy:

git clone https://github.com/opensandbox-group/OpenSandbox.git
cd OpenSandbox
git checkout release-1.1.0   # or main for development

Option 1: Umbrella chart (recommended)

One release installs every component with the correct ordering:

cd manifests/charts

# Package the sub-charts (charts/ is git-ignored, rebuilt every time)
helm dependency build opensandbox

helm install opensandbox opensandbox \
  --namespace opensandbox-system \
  --create-namespace

Optional components default to off. For Kubernetes deployments, enable the ingress gateway — sandbox Pods are ClusterIP-only and client traffic routes through it:

helm install opensandbox opensandbox \
  --namespace opensandbox-system \
  --create-namespace \
  --set ingress-gateway.enabled=true

Option 2: Per-component releases

# 1. Cluster foundation: CRDs + RBAC
helm install base manifests/charts/base

# 2. Controller
helm install opensandbox-controller manifests/charts/controller \
  --namespace opensandbox-system \
  --create-namespace

# 3. Ingress gateway (required on Kubernetes; see Deployment Order)
helm install ingress-gateway manifests/charts/ingress-gateway \
  --namespace opensandbox-system

# 4. Lifecycle server
helm install opensandbox-server manifests/charts/server \
  --namespace opensandbox-system \
  --create-namespace

Configure API authentication

By default, the server refuses to start without an API key in a non-interactive container. Create both the control-plane namespace and the default sandbox workload namespace, then store the key in a Kubernetes Secret:

kubectl create namespace opensandbox-system --dry-run=client -o yaml | kubectl apply -f -
kubectl create namespace opensandbox --dry-run=client -o yaml | kubectl apply -f -

read -s OPENSANDBOX_API_KEY
kubectl create secret generic opensandbox-api-key \
  --namespace opensandbox-system \
  --from-literal=api-key="${OPENSANDBOX_API_KEY}" \
  --dry-run=client -o yaml | kubectl apply -f -
unset OPENSANDBOX_API_KEY

Reference the Secret from a values file:

# values-server.yaml
server:
  replicaCount: 1
  env:
    - name: OPENSANDBOX_SERVER_API_KEY
      valueFrom:
        secretKeyRef:
          name: opensandbox-api-key
          key: api-key

Use an external secret manager instead of creating the Secret manually in production environments.

The chart installs the server into opensandbox-system, while the default configToml creates sandbox and pool resources in opensandbox. If you change [kubernetes].namespace in configToml, create that namespace instead of opensandbox before submitting workloads.

Single-active Server default

The chart defaults to server.replicaCount: 1. Keep one active Lifecycle Server unless you deliberately use the PostgreSQL-backed Kubernetes public snapshot topology documented below. That exception coordinates public snapshots only; it does not provide general multi-replica Server HA. The Server Deployment uses the Recreate strategy so an upgrade stops the active Server before starting its replacement; expect a brief API interruption during upgrades.

Use PostgreSQL for server persistence

Create a Secret containing the PostgreSQL connection string:

read -s OPENSANDBOX_POSTGRESQL_DSN
kubectl create secret generic opensandbox-postgresql \
  --namespace opensandbox-system \
  --from-literal=dsn="${OPENSANDBOX_POSTGRESQL_DSN}" \
  --dry-run=client -o yaml | kubectl apply -f -
unset OPENSANDBOX_POSTGRESQL_DSN

In values-server.yaml, keep the default server.replicaCount at 1, add the Secret-backed environment variable below, and add the shown [store] tables to the complete configToml value:

server:
  replicaCount: 1
  env:
    - name: OPENSANDBOX_STORE_POSTGRESQL_DSN
      valueFrom:
        secretKeyRef:
          name: opensandbox-postgresql
          key: dsn

configToml: |
  # Keep the rest of the chart's complete server configuration here.
  [store]
  type = "postgresql"

  [store.postgresql]
  min_pool_size = 1
  max_pool_size = 10
  snapshot_recovery_interval_seconds = 15

INFO

The chart default remains one Server replica. You may explicitly set server.replicaCount: 2 for multi-active public snapshot handling only when both replicas use the same PostgreSQL database and the Kubernetes runtime. SQLite and Docker snapshot execution do not support this multi-active topology.

Install the server with the API key and verify

Install the server referencing your values file:

helm install opensandbox-server manifests/charts/server \
  --namespace opensandbox-system \
  --create-namespace \
  --values values-server.yaml

Wait for the Deployment and verify the API health endpoint:

kubectl rollout status deployment/opensandbox-server \
  --namespace opensandbox-system \
  --timeout=180s

kubectl port-forward \
  --namespace opensandbox-system \
  service/opensandbox-server 8080:80

In another terminal:

curl --fail http://127.0.0.1:8080/health

Important values

Value	Purpose	Notes
server.image.repository	Server image registry and repository	Override for a private mirror or custom build.
server.image.tag	Server image version	Defaults to the release-<appVersion> image published with the chart version; override only for custom builds.
server.replicaCount	Number of server Pods	Defaults to 1; general multi-replica Server HA is not supported yet.
server.env	Additional container environment variables	Use it with secretKeyRef for OPENSANDBOX_SERVER_API_KEY.
configToml	Complete server configuration	Mounted at /etc/opensandbox/config.toml; overriding it replaces the complete default TOML, including the workload namespace.
server.gateway.enabled	Announce an ingress gateway to clients	Defaults to false. The gateway itself is deployed by the ingress-gateway chart.
server.service.type	Service type for the server	Defaults to ClusterIP. Use NodePort or LoadBalancer for access from outside the cluster; pin the port with server.service.nodePort.
namespaceOverride	Namespace used by chart resources	Defaults to opensandbox-system.

The server container and its Service use port 80. Keep [server].port = 80 when replacing configToml unless the chart templates are also updated to use a different port. The Service is ClusterIP by default; set server.service.type to reach the server from outside the cluster.

Configure egress sidecar resources

When a create request includes networkPolicy, the lifecycle server adds an egress sidecar to each non-pooled sandbox Pod. Namespace LimitRange defaults apply to this container when it does not declare resources, which can reserve substantially more capacity than basic DNS/nft enforcement needs.

Add optional resource settings to the [egress] section of configToml:

[egress]
image = "opensandbox/egress:release-1.1.0"
requests = { cpu = "25m", memory = "64Mi" }
limits = { cpu = "250m", memory = "256Mi" }

You can omit either requests or limits. Treat these values as a starting point and tune them from observed usage; Credential Vault and transparent mitmproxy generally need more headroom than basic DNS/nft enforcement.

Ingress gateway (required on Kubernetes)

Sandbox Pods on Kubernetes are ClusterIP-only — client traffic reaches sandboxes through the ingress gateway ([ingress] mode = "gateway", the Kubernetes-oriented mode; the Docker runtime uses direct and does not need the gateway). Install the gateway before the server, then include the announcement in the server install:

helm install ingress-gateway manifests/charts/ingress-gateway \
  --namespace opensandbox-system \
  --set gateway.fastpathEndpoint=fast-sandbox-fastpath.opensandbox-system.svc:9090

Install the server with the announcement enabled (or helm upgrade an existing server release with the same flags):

helm install opensandbox-server manifests/charts/server \
  --namespace opensandbox-system \
  --set server.gateway.enabled=true \
  --set server.gateway.host=gateway.example.com \
  --values values-server.yaml

Keep server.gateway.gatewayRouteMode in sync with gateway.gatewayRouteMode of the gateway chart. For signed, expiring sandbox routes, configure the shared secure-access key ring on both charts — see secure-access keys.

Optional: fast-sandbox runtime

The fast-sandbox chart adds the Firecracker (sandbox.fast.io) runtime. Install it before the ingress gateway and the lifecycle server (see Deployment Order): the server's [runtime]/fsb configuration points at the FastPath gRPC endpoint this chart creates. It also requires base first, KVM-capable nodes, and companion images built from a pinned upstream commit — see the fast-sandbox runtime deployment guide.

Upgrade

Upgrade the umbrella release from a newer checkout:

git fetch --tags
git checkout release-1.2.0

cd manifests/charts
helm dependency build opensandbox
helm upgrade opensandbox opensandbox --namespace opensandbox-system

For a per-component server release:

git checkout release-1.2.0
helm upgrade opensandbox-server manifests/charts/server \
  --namespace opensandbox-system \
  --values values-server.yaml

For the complete values reference and local development installation, see the opensandbox-server chart README.

Operator Metrics

The operator (controller-manager) exposes standard controller-runtime Prometheus metrics — reconcile rate and latency (controller_runtime_reconcile_*), work-queue depth, client-go request counts, and Go runtime stats. The endpoint is disabled by default (--metrics-bind-address=0).

Enable it through the opensandbox-controller chart values:

Value	Default	Purpose
controller.metrics.enabled	false	Expose the /metrics endpoint (sets --metrics-bind-address)
controller.metrics.port	8080	Port for the metrics endpoint
controller.metrics.secure	false	Serve over HTTPS with authn/authz (--metrics-secure); set false for plain HTTP scraping

controller:
  metrics:
    enabled: true
    port: 8080
    secure: false   # plain HTTP, e.g. for a PodMonitoring/ServiceMonitor scrape

• With secure: false the endpoint is plain HTTP and can be scraped directly (no TLS or bearer token).
• With secure: true the controller-runtime filter authenticates and authorizes each scrape via TokenReview/SubjectAccessReview. The chart then provisions two ClusterRoles automatically:
  • opensandbox-metrics-auth-role (bound to the manager) — lets the controller run the auth checks.
  • opensandbox-metrics-reader (not bound by the chart) — grants get on the /metrics non-resource URL. Bind it to your scraper's ServiceAccount (e.g. Prometheus) and have the scraper present that account's bearer token.

Point your Prometheus stack at the metrics container port (for example via a ServiceMonitor or PodMonitoring).

Business capacity metrics

The elected controller also exports low-cardinality business capacity metrics over OTLP/HTTP when OTEL_EXPORTER_OTLP_METRICS_ENDPOINT or OTEL_EXPORTER_OTLP_ENDPOINT is set. This is independent of the controller-runtime Prometheus endpoint and remains disabled when neither variable is configured.

extraEnv:
  - name: OTEL_EXPORTER_OTLP_METRICS_ENDPOINT
    value: http://otel-collector.observability:4318/v1/metrics

Metric	Unit	Attributes	Description
controller.pool.pods	{pod}	namespace, pool_name, state	Current Pool Pods, where state is total, allocated, available, or updated
controller.pool.cpu.requested	{cpu}	namespace, pool_name, state	Scheduler-equivalent CPU requests represented by total, allocated, or available Pool Pods
controller.pool.memory.requested	By	namespace, pool_name, state	Scheduler-equivalent memory requests represented by total, allocated, or available Pool Pods
controller.batchsandbox.count	{batchsandbox}	namespace, phase, allocation_mode	Current BatchSandbox objects by lifecycle phase and pool/direct mode
controller.batchsandbox.pods	{pod}	namespace, state, allocation_mode	Desired, current, allocated, and ready BatchSandbox Pod counts
controller.capacity.collect.duration	s	None	Time spent reading cached objects and collecting one capacity snapshot

The metrics deliberately omit sandbox, BatchSandbox, and Pod identifiers. Only the leader exports them, so multiple controller replicas do not duplicate cluster totals. An unset initial BatchSandbox phase is exported as Unknown. Derive Pool utilization from allocated / total and calculate peak, valley, or percentile capacity in the telemetry backend. Actual CPU and memory usage remains available from kubelet/cAdvisor rather than being duplicated here.

Each collection reads all Pools and BatchSandboxes from the controller manager's informer cache, then performs one cached, owner-UID-indexed Pod list for every non-deleting Pool. Collection CPU and memory therefore grow linearly with the number of cached Pools, BatchSandboxes, and Pool-owned Pods, without issuing one API-server list request per Pool. The OpenTelemetry periodic reader exports every 60 seconds by default; OTEL_METRIC_EXPORT_INTERVAL can change the interval in milliseconds. Monitor controller.capacity.collect.duration and validate the target cluster scale before shortening that interval. If OTLP setup fails after an endpoint is configured, the controller continues reconciling and logs the failed setup stage together with the endpoint environment variable and a credential-stripped endpoint.

Configure the Server for Kubernetes

Generate a Kubernetes-oriented server config:

opensandbox-server init-config ~/.sandbox.toml --example k8s

Key Kubernetes-specific configuration sections:

Section	Purpose
[kubernetes]	Workload provider, BatchSandbox template file
[agent_sandbox]	Agent sandbox settings
[ingress]	Ingress gateway for sandbox traffic routing
[secure_runtime]	Secure container runtime (gVisor, Kata)

See Configuration for the full reference.

Components on Kubernetes

Component	Deployment	Purpose
CRDs + RBAC (base)	Cluster-scoped	BatchSandbox, Pool, SandboxSnapshot and sandbox.fast.io API types
Server	Deployment	Lifecycle control plane
Controller (operator)	Deployment	Manages BatchSandbox/Pool CRDs
Ingress gateway	Deployment	Routes traffic to sandboxes; required on Kubernetes deployments
Egress	Sidecar	Per-sandbox egress policy enforcement
Execd	Built into sandbox images	In-sandbox execution
Node agent	DaemonSet	Optional node-level sandbox data collection
fast-sandbox	Deployment + DaemonSet	Optional Firecracker control plane and node runtime

Uninstall

helm uninstall opensandbox -n opensandbox-system

CRDs carry the helm.sh/resource-policy: keep annotation and are retained across uninstalls; the opensandbox-dataplane namespace (created by base for fast-sandbox) is kept as well. Delete them manually once their data is no longer needed:

kubectl delete crd batchsandboxes.sandbox.opensandbox.io \
  pools.sandbox.opensandbox.io \
  sandboxsnapshots.sandbox.opensandbox.io
kubectl delete namespace opensandbox-dataplane

Related

• Kubernetes Overview — Operator features and CRDs
• Pause & Resume — Snapshot-based pause/resume on Kubernetes
• Secure Container — gVisor and Kata on Kubernetes
• Network Isolation — Egress policy design for Kubernetes
• Helm charts source — Chart sources and values reference