← Back to the article: How to read GPU logs in Kubernetes

GPU Log Watch

Logging demo for GPU clusters on Kubernetes · Fluent Bit → Loki → Grafana
10:00:00

A simulated GPU cluster is writing logs right now: a training job, two vLLM inference servers, the GPU Operator, the kubelet and the Linux kernel. Run a LogQL query, or trigger a failure on the right and find the line that explains it. Companion to GPU Cluster Watch, which shows the same failures as metrics.

How the logs get here

Sources on each GPU nodekernel ring buffer, container stdout, kubelet journal
→
Fluent BitDaemonSet tails files, adds node, namespace, pod labels
→
Lokiindexes labels only, stores lines compressed
→
Grafana · Loki rulerLogQL queries and log-based alerts (this page)

Labels in this demo: node, namespace, pod, container, source (kernel, container or kubelet). Kernel lines have no namespace or pod: they belong to the node. Demo clock runs in real time.

Explore · LogQL

0 matching lines in the last 2 minutes errorwarninginfo
−2m−1mnow

Newest first, up to 200 lines. Click a line to see its labels and what to check next.

Trigger a failure

Log alerts

Selected line

Click any log line to inspect it.