Node Agent

Sandboxes are ephemeral by design: when a sandbox is deleted, everything it printed to stdout and stderr disappears with it. Node Agent is an optional DaemonSet that fixes this — one collector per Linux node watches every sandbox pod on that node and ships their output to durable storage, so records outlive the sandbox that produced them. Collection is entirely node-side and invisible to the workload.

Experimental capacity

Collection and delivery are implemented, but production resource defaults still await benchmark and soak results (OSEP-0019). Treat the chart's resource values as starting points.

Capabilities at a glance

Capability	What you get
Log collection	stdout/stderr of every sandbox pod on the node (pooled warm pods are excluded — they are not allocations)
Syscall observation	An optional kernel-level source recording each syscall a sandbox makes, via an embedded eBPF program
Durable targets	Alibaba Cloud OSS or durable local files — one target per Agent; a best-effort stdout mode exists for debugging
Delivery	At-least-once, crash-recovering, with explicit completeness markers per stream
Organization	Records laid out per cluster / namespace / sandbox / pod, ready for downstream consumption
Safety	Backpressure instead of silent drops; refuses unsafe bucket configurations
Observability	Liveness vs readiness split; metrics free of sensitive labels

How collection works

The Agent is a pipeline: sources produce records, bounded per-sandbox queues carry them, and one sink writes them out. Three design choices shape the whole system:

• Sources and sinks are compile-time registries, not plugins. A build embeds the sources it supports; configuration selects among them, and an unknown name leaves the Agent unready rather than starting a partial pipeline. No code is loaded at runtime.
• Queues are bounded, and the drop policy is explicit. The default block policy applies backpressure to the source when a sink slows down — collection slows, nothing is lost. The alternative drop policy bounds memory harder and accounts for whatever it discards in the stream's completeness marker. A global memory budget, a per-sandbox queue budget, and an optional per-sandbox rate limit keep one noisy sandbox from starving the node.
• Fail closed, never crash-loop. Invalid configuration, damaged or mismatched recovery state, or an unusable target keeps the process alive but not ready, reporting a named reason. It never silently resets and re-collects, and it never advances progress onto an unverified destination.

Pod selection is identity-based, not label-guesswork: a pod is collected when it carries sandbox identity (the sandbox ID label, the sandbox container, its restart count) and is not allocated from a pool (pods labeled with their pool are warm capacity, not sandboxes).

Sources

container-logs (default) tails the kubelet's log files for sandbox pods on the node and streams each container stream as container-log records. An inotify hot path backed by low-frequency reconciliation follows rotation, compression, and file reuse; partial CRI lines are reassembled before delivery. Log files are identified by fingerprint, so a reclaimed or replaced file becomes an accounted gap instead of silently spliced data. A stream is finalized only after its pod has terminated and its reads have stayed at end-of-file.

syscalls records what sandboxes actually do at the kernel level: an embedded eBPF program is attached to each sandbox pod's cgroup and emits one record per syscall — syscall number, host PID/TID, command name, timestamps — as syscall records. Kernel ring-buffer overflows, agent-restart intervals, and late attachment are all tracked per stream and reflected in the markers, and container restarts rebind the tracer behind a drain barrier so events cannot be misattributed across generations. This source is Linux-only (kernel 5.11+, cgroup v2) and requires BPF and PERFMON capabilities.

Each source owns its record kind and storage format; both feed the same marker and recovery machinery.

Where records land

Each stream becomes an object family under a layout that mirrors cluster reality. Every record kind has its own family (container-log, syscall), each with its own encoding and marker set:

<cluster>/<namespace>/<sandbox_id>/<pod_uid>/
  sandbox.log
  sandbox.<generation>.log
  sandbox.finalized.<revision>.json

Non-log record kinds are isolated in a separate <cluster>/_streams/<kind>/… tree under the same rules.

The finalized marker is the stream's closing statement — a cumulative, immutable snapshot listing every closed object with its size and checksum, which consumers verify. It answers the only question that matters after an incident: can I trust these logs?

Marker status	Meaning
complete	No intentional drop or known gap. At-least-once delivery may still contain duplicates.
complete-with-drops	At least one record was intentionally dropped, with the affected interval accounted for.
incomplete	At least one known interval could not be read, or its coverage cannot be proved.

The honesty is deliberate. If the Agent restarts while a stream is open, the recovered data is still delivered — but the next marker is incomplete, because nothing can prove that a short-lived rotation didn't vanish during the outage. Consumers get truth, not optimism.

Delivery guarantees

• At-least-once, never silent loss: a record's cursor advances only after the sink has durably accepted it and the source has acknowledged it; finalization itself is a two-phase, crash-recoverable transaction. Duplicates are possible; unaccounted loss is not. Anything questionable shows up in the marker.
• Strict per-stream ordering: a stream has at most one in-flight batch and one finalizing revision at a time — the next revision starts only after the previous marker is durable.
• State bound to the destination: local recovery state is keyed to a digest of the configured target. A damaged or mismatched state makes the Agent unready — it never silently resets and re-collects from scratch.
• Bounded footprint: local files rotate under size, count, and total-volume caps; retention-driven cleanup is crash-resumable — it moves a finished family into a staging area and removes it only after the move is durable, and it never deletes one generation to make room for another.
• Retention is the operator's job: the Agent collects; it does not make retention policy. OSS deletion always happens through a separate offline cleanup tool that requires explicit delete credentials — collection credentials never have them. The stdout debug mode is best-effort: it publishes no markers and makes no durability claim.

Safety model

• Least-privilege credentials: OSS collection credentials need append, read, and write permissions — never delete.
• Refuses unsafe targets: bucket versioning or WORM, or an overlapping lifecycle rule, is a permanent startup error — configurations that would break the Agent's integrity assumptions are rejected, not worked around.
• Host reserves are validated at startup: file-descriptor and inotify limits, disk headroom for state and output, and the queue budget against the cgroup memory limit are checked before collection begins — problems Helm's resource schema cannot express fail startup instead of failing collection later.
• Readiness means trustworthy: /readyz aggregates a set of named, low-cardinality reasons (invalid config, target unreachable, retry loop active, …) and passes only when none apply. A not-ready Agent is visible and diagnosable, not silently dropping data.

Observability

Liveness (/healthz) and readiness (/readyz) are deliberately separate: the process can be alive while collection is blocked, and readiness surfaces why using low-cardinality reason codes — never credentials, paths, or bucket names. When an OTLP endpoint is configured, the Agent reports record and byte counts, intentional drops, retries, and queue depth, with no sandbox or pod identifiers as labels.

Enabling it

Node Agent ships in the node-agent Helm chart and is disabled by default in the umbrella chart. See the node-agent chart for values — target type, OSS credentials, retention, and cluster ID — and the component source for implementation details.