# litek8s
> A whole Kubernetes in one Rust binary: a local cluster in about a second,
> spare machines as one cluster, a control plane that idles at 23 MiB. It
> serves the real Kubernetes API (kubectl, Helm, Argo CD work unchanged), plus
> a Docker Engine API, built-in metrics/logs/alerts, a terminal UI and an MCP
> server (`litek8s mcp`).
Agents: prefer `litek8s mcp` (tools for dev clusters, get/apply/logs/PromQL,
diagnose_pod) and its resources `litek8s://skill/...` and `litek8s://docs/...`;
`litek8s skill` prints the same guide. Servers authenticate clients and
enforce RBAC since 0.2.12: a 401 or 403 on a cluster made with an earlier
release is fixed as in "Authentication and RBAC" below. GitOps with Argo CD's
manifests in one small pod: litecd ("GitOps with litecd" below).
---
# litek8s: a lightweight Kubernetes in one binary
**A lightweight Kubernetes in one binary**, for local development, home
labs and the edge. It serves the real Kubernetes API, so `kubectl`, Helm
and Argo CD work unchanged, but the whole control plane, scheduler,
controllers, DNS, service proxy and node agent are one Rust process that
idles at 23 MiB on Linux (on macOS, plus the small VM it runs in).
```bash
litek8s dev up # a local cluster: ~0.2 s on Linux, ~1.3 s on macOS
kubectl get nodes # context litek8s-dev, merged into ~/.kube/config
litek8s ui # terminal UI for the cluster, containers and images
```
> **Status: early preview (v0.x).** Built and tested by one person; expect
> rough edges. Servers authenticate clients and enforce RBAC
> ([Authentication and RBAC](auth.md)); see [Limitations](limitations.md).
> Feedback and issues are very welcome.
## What's in the binary
- **Dev clusters** (`litek8s dev`): kind/k3d/minikube without Docker. As
many clusters as you like, each with as many nodes, in one rootless
sandbox (Linux) or one small Virtualization.framework VM (macOS) that
grows its memory on demand. NodePorts on `localhost`, your home directory mounted
at the same path, `litek8s dev load` for local images. [dev.md](dev.md)
- **Docker Engine API** on `~/.litek8s/docker.sock`: `docker`,
`docker compose`, `docker build`, Dev Containers and Testcontainers run
on the dev cluster, next to (not instead of) Docker or OrbStack.
[docker.md](docker.md)
- **Terminal UI** (`litek8s ui`): cluster state, a k9s-style Kubernetes
view, containers by compose project with logs and shells, images,
volumes, disk cleanup; keyboard-driven, works over SSH. [ui.md](ui.md)
- **Built-in observability**: an embedded time-series store with a
Prometheus-compatible API (Grafana works), log search across every pod
and node, alert rules to a webhook, ntfy or Telegram; `kubectl top`.
[observability.md](observability.md)
- **Real servers and nodes**: `litek8s apiserver` on a box, more machines
join with `litek8s agent --join ... --token ...` (k3s-style token that
pins the CA), pods talk across nodes over a WireGuard mesh litek8s sets
up itself. Pods run on crun directly: no containerd, no pause containers.
[Multi-node](multi-node.md)
- **HA control plane**: 3 or 5 API servers share the store through Raft;
no etcd. [ha.md](ha.md)
- **For LLM agents**: `litek8s mcp` is an MCP server (dev clusters, objects,
pod diagnosis, logs, PromQL, alerts) with guardrails, and `litek8s skill`
installs an agent skill: `claude mcp add litek8s -- litek8s mcp`.
[agents.md](agents.md)
- **CRDs done properly**: OpenAPI schema validation, CEL rules type-checked
and cost-estimated, ratcheting, server-side apply with managedFields.
[crds.md](crds.md)
Use case: **one LLM across all your machines**. A model too big for any
one of them runs on several with llama.cpp's RPC backend (a DaemonSet of
workers, one OpenAI-compatible server):
[examples/llm-rpc](../examples/llm-rpc/README.md), [llm-cluster.md](llm-cluster.md).
## How it works
One process does what a Kubernetes distribution splits across etcd, the
API server, the controller manager, the scheduler, CoreDNS, kube-proxy,
the kubelet and containerd, plus a Docker-compatible API and a metrics,
logs and alerts stack:
```mermaid
flowchart TB
k8s["kubectl · Helm · Argo CD · litek8s ui · MCP agents"]
dk["docker · compose · Dev Containers · Testcontainers"]
subgraph bin["litek8s · one process"]
api["Kubernetes API server CRDs + CEL · server-side apply · watch"]
dockerapi["Docker Engine API"]
store[("object store WAL, or Raft in HA")]
ctrl["controllers + scheduler"]
net["DNS + service proxy"]
obs["metrics · logs alerts"]
agent["node agent"]
api --- store
api <--> ctrl & net & obs & agent
end
k8s -->|":6443"| api
dk -->|"docker.sock"| dockerapi
agent --> crun["crun: containers no containerd, no pause containers"]
dockerapi --> crun
```
## License
Apache-2.0, see [LICENSE](../LICENSE) and [NOTICE](../NOTICE). Third-party
components: [THIRD-PARTY.md](../THIRD-PARTY.md).
---
# litek8s next to the Kubernetes you already run
litek8s does not replace your clusters or your tools. It adds clusters to
the kubeconfig you already have, and everything that reads that file
(kubectl, Helm, k9s, Lens, Argo CD, `litek8s ui`) sees them next to your
EKS, GKE or Civo contexts. Switch with `kubectl config use-context` or
kubectx, as you do today.
```mermaid
flowchart LR
subgraph tools["your tools"]
direction TB
kubectl ~~~ helm["Helm"] ~~~ k9s["k9s · Lens"] ~~~ argo["Argo CD"] ~~~ ui["litek8s ui"]
end
kc[("~/.kube/config kubectl config use-context · kubectx")]
tools --> kc
subgraph lk["litek8s clusters"]
direction TB
dev["litek8s-dev"] ~~~ ci["litek8s-ci"] ~~~ home["home: a server and its nodes"]
end
subgraph cloud["clusters you already run"]
direction TB
eks["EKS"] ~~~ gke["GKE"] ~~~ civo["Civo"]
end
kc --> lk
kc --> cloud
```
## No lock-in, by construction
- **It only adds contexts.** `litek8s dev up` writes one context,
`litek8s-dev` (`--name ci` adds `litek8s-ci`), into `~/.kube/config`
and makes it current; `--no-switch` adds it without switching. A server
installed with `litek8s install server --ssh` or `litek8s cluster apply`
adds its own context the same way. Your other contexts are left as they
are.
- **It is the Kubernetes API.** litek8s serves Kubernetes v1.33
(`kubectl version` shows `v1.33.0+litek8s`): 119/119 on the compatibility
corpus (kubectl workflows, CRDs, workloads, networking, 47 Helm charts)
and 413/419 on the official conformance suite, where k3s scores 416/419
([compat/RESULTS.md](../compat/RESULTS.md),
[compat/CONFORMANCE.md](../compat/CONFORMANCE.md)).
- **Same manifests, same charts, both ways.** What you apply to litek8s
applies to any other cluster, and the other way round. There is no
litek8s manifest format, no annotations you have to add, no SDK.
- **Going back is deleting it.** `litek8s dev delete` stops a dev cluster
and removes its data and its context (`--name ` for each other one;
`litek8s dev list` shows them), then `brew uninstall litek8s`. On a
server, `sudo litek8s uninstall --purge`. Nothing to migrate: your
manifests and charts never lived in litek8s.
## kubectl
litek8s does not ship kubectl; use the one you have, or install it:
```bash
brew install kubectl # macOS, or Linux with Homebrew
```
or the official packages from pkgs.k8s.io
([Install tools](https://kubernetes.io/docs/tasks/tools/)). Keep it within
one minor version of the server, as Kubernetes recommends for any cluster:
**1.32 to 1.34** for litek8s's 1.33. Helm, k9s, Lens, kubectx and Argo CD
need nothing special either.
## `litek8s ui` on your other clusters
The terminal UI works on any conformant cluster through the same
kubeconfig, credentials included: tokens, client certificates and exec
plugins such as `gke-gcloud-auth-plugin`, `aws eks get-token` or
`kubelogin` ([ui.md](ui.md#read-only-clusters)).
- **Read-only by default** for every context that is not a litek8s
cluster: list, describe, YAML, logs, the object tree and port-forwards
work; deletes, edits, applies, scaling, shells and the like are refused.
To allow changes on one, list it in `ui.yaml` (`writableContexts:
[staging]`); a per-context switch in the UI itself is being added.
- **litek8s-only:** Observe (built-in metrics, log search and alerts)
reads the litek8s API server's own metrics, and Home shows the dev VM or
sandbox. On other clusters you get the Kubernetes view; the Docker views
(containers, images, volumes) follow your Docker context, not the
kube context.
## What's different from upstream
litek8s is an early preview with a few gaps compared with upstream
Kubernetes, among them simpler authentication and RBAC (no OIDC, no Node
authorizer): see [Limitations](limitations.md).
---
# Install
macOS (Apple silicon) and Linux (x86_64, aarch64), with Homebrew:
```bash
brew install jairoFernandez/tap/litek8s
```
or with the install script:
```bash
curl -fsSL https://raw.githubusercontent.com/jairoFernandez/litek8s/main/install.sh | sh
```
The script downloads the latest release for your OS and architecture, checks its
SHA-256, and installs `litek8s` to `~/.local/bin` (on macOS also the dev
VM's guest kernel and root image to `~/.local/share/litek8s/vm`). Options:
`LITEK8S_VERSION=v0.1.0`, `LITEK8S_INSTALL_DIR=/usr/local/bin`. Or
download an archive from the [releases page](https://github.com/jairoFernandez/litek8s/releases)
and check it against `SHA256SUMS`.
On Linux, dev clusters run rootless and need `crun`, `uidmap`, `passt`
(or `slirp4netns`), `iproute2` and `iptables`: `litek8s dev setup` checks
everything and says what is missing ([dev.md](dev.md#what-it-needs)), and
`sudo "$(command -v litek8s)" dev setup` installs the missing packages (the
full path because sudo's PATH leaves out `~/.local/bin`).
macOS needs nothing else: no Docker Desktop, no Lima, no sudo.
Running it as a service on Linux servers (`litek8s install server|agent`,
also over SSH) is in [clusters.md](clusters.md); private registries and
mirrors for their nodes (`--registries registries.yaml`) in
[registries.md](registries.md).
Shell completion: Homebrew installs it; otherwise `litek8s completion zsh`
(or bash, fish, powershell, elvish) prints the script, e.g. `source <(litek8s
completion zsh)` in `~/.zshrc`.
From source: `cargo install litek8s` (crates.io) or `cargo install --git
https://github.com/jairoFernandez/litek8s` (Rust ≥ 1.90; macOS also needs the
guest, see [dev.md](dev.md)).
---
# Quickstart
```bash
litek8s dev up
kubectl create deployment web --image=nginx
kubectl expose deployment web --type=NodePort --port 80
curl localhost:$(kubectl get svc web -o jsonpath='{.spec.ports[0].nodePort}')
litek8s ui # browse it all
export DOCKER_HOST=unix://$HOME/.litek8s/docker.sock
docker run --rm hello-world # the same cluster speaks the Docker API
litek8s dev up --name ci --nodes 3 # another cluster, three nodes
litek8s dev down # stop; objects and images are kept
litek8s dev delete # remove everything
```
Every cluster, and every node of a multi-node cluster, runs in one small
VM on macOS or one rootless sandbox on Linux, started with the first
cluster and stopped with the last. Each cluster gets its own API port,
kubectl context and Docker socket:
```mermaid
flowchart LR
subgraph host["your machine"]
you["kubectl · docker · litek8s ui"]
subgraph env["one VM / sandbox"]
subgraph dev["cluster dev"]
d1["node"]
end
subgraph ci["cluster ci, 3 nodes"]
c1["node 1 API server"]
c2["node 2"]
c3["node 3"]
c2 & c3 -->|join| c1
end
end
end
you -->|"127.0.0.1:6443 context litek8s-dev"| d1
you -->|"127.0.0.1:port context litek8s-ci"| c1
you -.->|"docker.sock"| d1
```
## Your own image, no registry
The Docker socket and the cluster share one image store, so an image you
build is a pod's `image:` right away:
```bash
export DOCKER_HOST=unix://$HOME/.litek8s/docker.sock
DOCKER_BUILDKIT=0 docker build -t myapp:dev . # litek8s's builder
kubectl create deployment myapp --image=myapp:dev
```
`DOCKER_BUILDKIT=0` works with any `docker` CLI. To build with buildx (Docker
Desktop and OrbStack install it) use `docker build --load -t myapp:dev .`:
without `--load` buildx keeps the image in its own builder and the cluster
never sees it. See [Builds](docker.md#builds-two-builders).
---
# Numbers
Measured against k3s, kind and k3d with the scripts in this repository; the
linked files have the setup, raw data and caveats.
| | litek8s | reference | source |
|---|---|---|---|
| Whole control plane, idle, no pods (PSS, v0.2.1, 1 GiB x86_64 VM) | **23.2 MiB** | k3s minimal: 567 MiB | [bench/RESULTS.md](../bench/RESULTS.md#footprint-of-the-v021-release-2026-10-21-x86_64kvm) |
| Whole orchestrator, 5 nginx pods (same) | **33.5 MiB** | k3s minimal: 579 MiB | same |
| Argo CD + one synced app, orchestrator memory | **90 MiB** | k3s minimal: 900 MiB | [bench/RESULTS.md](../bench/RESULTS.md#argo-cd-on-k3s-and-on-litek8s-2026-10-19-x86_64kvm) |
| `dev up` → node Ready, Linux / macOS | **0.17 s / 1.28 s** | kind 31.6 s / 34.3 s, k3d 11.1 s / 10.4 s | [bench/RESULTS.md](../bench/RESULTS.md#local-dev-cluster-litek8s-dev-vs-kind-and-k3d-2026-10-18) |
| Dev cluster idle memory, Linux | **47 MiB** | kind 526 MiB, k3d 398 MiB | same |
| GitHub Actions job, Helm chart e2e (setup, install, test, upgrade) | **21 s**, cluster ready in 2.6 s | kind (helm/kind-action): 79 s, 44.7 s | [docs/ci.md](ci.md#numbers) |
| Metrics, logs and alerts, added memory | **+1.8 MiB** | kube-prometheus-stack + Loki: 1,371 MiB | [bench/RESULTS.md](../bench/RESULTS.md#observability-built-in-vs-kube-prometheus-stack--loki-2026-10-20-x86_64kvm) |
| Compatibility corpus (kubectl, CRDs, workloads, networking, 47 Helm charts) | **119/119** | k3s: 119/119 | [compat/RESULTS.md](../compat/RESULTS.md#without-docker-2026-10-19-119-cases) |
| Kubernetes v1.33 conformance (419 specs) | **413/419** | k3s: 416/419 | [compat/CONFORMANCE.md](../compat/CONFORMANCE.md) |
On macOS a dev cluster runs in a Linux VM, and Activity Monitor shows the
whole VM: 238 MiB fresh, ~380 MiB after pulling and running 5 nginx pods
(the cluster itself is 30 and 79 MiB of that). macOS keeps charging the VM
for every page its guest has touched until the VM stops
([docs/dev.md](dev.md#memory-on-macos)).
The 6 conformance failures: 5 are macOS-only (TCP DNS on port 53, a
hostPort binding) and pass on Linux; 1 was a watch fixed since. A 3-server
HA control plane stays linearizable under leader kills and partitions
([bench/HA.md](../bench/HA.md)); a 3-node cluster ran 21 h under churn with no
leak ([bench/SOAK.md](../bench/SOAK.md)).
---
# Platforms
| | dev clusters | server / node (`apiserver`, `agent`) | release binary |
|---|---|---|---|
| **macOS, Apple silicon** (13+) | yes, in a VM (Virtualization.framework) | for development only (Docker runtime) | `darwin-arm64`, VM guest included |
| **Linux x86_64 / aarch64, rootless** | yes, user-namespace sandbox (kernel ≥ 5.11, cgroup v2) | – | `linux-x86_64`, `linux-aarch64` (static) |
| **Linux, root with crun** | yes | yes: the intended server and node setup | same |
| macOS Intel, Windows | no (WSL2 untested) | no | – |
Tested on Ubuntu 24.04 (arm64 and x86_64) and recent macOS on Apple
silicon. Ubuntu ≥ 23.10 needs `sudo "$(command -v litek8s)" dev setup` once
(AppArmor and unprivileged user namespaces).
---
# Multi-node and HA
A server runs the whole control plane and its own pods; any Linux machine
(a Raspberry Pi, an old laptop, a cheap cloud VM) joins it as a node with
the token the server prints:
```mermaid
flowchart TB
laptop["your laptop · kubectl"] -->|":6443"| s
s["server · litek8s-server (systemd) API server, controllers, its own pods"]
subgraph nodes["nodes (litek8s-agent)"]
direction LR
pi["Raspberry Pi"] ~~~ old["old laptop"] ~~~ ec2["cheap cloud VM"]
end
nodes -->|"join token · HTTPS"| s
wg["pods talk across nodes over a WireGuard mesh 10.42.0.0/16, one /24 per node relayed by the server across NATs"]
nodes -.- wg
```
On Linux servers (systemd, root), one command per machine, like k3s's
installer:
```bash
# the server (pods run here too); prints the join command and the kubeconfig path
curl -fsSL https://raw.githubusercontent.com/jairoFernandez/litek8s/main/install.sh | sh -s -- server
# every other machine: the command the server printed
curl -fsSL https://raw.githubusercontent.com/jairoFernandez/litek8s/main/install.sh | sh -s -- \
agent --join https://192.168.1.10:6443 --token K10::
```
This installs `/usr/local/bin/litek8s` (and `crun`/`iptables` from the
package manager when missing), checks the machine (cgroup v2, kernel
WireGuard, free ports) and runs `litek8s install server|agent`: a systemd
unit (`litek8s-server` / `litek8s-agent`, `Restart=always`) with the token in
`/etc/litek8s/*.env`, kubeconfig in `/etc/litek8s/kubeconfig.yaml`. Pods keep
running across `systemctl restart` and upgrades, and come back after a
reboot. Extra flags go to `litek8s apiserver` / `litek8s agent` (e.g.
`--node-ip`); re-running replaces them. `sudo litek8s uninstall [--purge]`
removes it all.
Or do it all from your laptop over SSH (your `~/.ssh/config`, keys and
agent; sudo on the machines), for one machine or a whole cluster declared
in a file ([clusters.md](clusters.md)):
```bash
litek8s install server --ssh pi@192.168.1.10 # adds the kubeconfig context
litek8s install agent --ssh me@old-laptop --join https://192.168.1.10:6443 --token K10::
cat > cluster.yaml <<'EOF'
apiVersion: litek8s.dev/v1
kind: Cluster
metadata: {name: home}
servers: [{ssh: pi@192.168.1.10}] # 1, or 3 for HA
agents:
- {ssh: me@old-laptop, labels: {disk: ssd}}
- {ssh: ubuntu@ec2-host, nodeIp: 10.0.0.7}
EOF
litek8s cluster apply -f cluster.yaml # again: no-op; agent removed + --prune: drained, uninstalled
litek8s cluster status -f cluster.yaml
```
Without systemd, run the same processes yourself:
```bash
litek8s apiserver --listen 0.0.0.0:6443 --data-dir /var/lib/litek8s/apiserver --node-ip 192.168.1.10
litek8s agent --join https://192.168.1.10:6443 --token K10:: --node-ip 192.168.1.21
```
Every node gets a `/24` of `10.42.0.0/16`; Services and DNS work
cluster-wide. Nodes behind NAT need no VPN or port forwarding: two homes
reach each other directly when their routers allow it, and a pair with no
direct path goes through a relay, by default the server (a cloud VM with a
public address: `--node-external-ip`), with the data path in the kernel
([networking.md](networking.md)). A node that stops goes `NotReady` after 40 s and its pods are
rescheduled after their `tolerationSeconds`; laptops that sleep can join
with a longer `--node-grace-period`. Three servers make the control plane
highly available: `install.sh | sh -s -- server --cluster-init` on the first,
`... server --server https://:6443 --token ` on two more
([ha.md](ha.md)). Any one server can die without losing writes;
`litek8s lb` gives kubectl one address in front of all three:
```mermaid
flowchart TB
laptop["your laptop · kubectl"] --> lb["litek8s lb 127.0.0.1:6443"]
subgraph cp["control plane"]
direction LR
s1["server 1"] <-->|"Raft :6444"| s2["server 2"] <-->|"Raft :6444"| s3["server 3"]
end
lb --> cp
subgraph nodes["nodes"]
direction LR
n1["agent"] ~~~ n2["agent"] ~~~ n3["agent"]
end
nodes -->|"any server, fail over"| cp
```
---
# Limitations
Honest list; more in [skills/litek8s/references/differences.md](../skills/litek8s/references/differences.md).
- **Authentication and RBAC are simpler than upstream's.** `litek8s
apiserver` (and so `install server`, `cluster apply`) authenticates
clients and enforces RBAC by default (`--authorization-mode RBAC`): service
account tokens, client certificates signed by the cluster's client CA (CN
user, O groups; the kubeconfig it writes is `system:admin` in
`system:masters`), and the join token (an administrator's credential).
Anonymous requests reach only `/healthz`, `/livez`, `/readyz`, `/version`
and the OIDC discovery documents; everything else gets 401.
`--authorization-mode RBAC-audit` decides, logs (`RBAC-audit: DENY ...`,
and each user's first use of a verb and resource as `ALLOW`) and allows,
to find the RBAC a cluster's charts lack before enforcing;
`AlwaysAllow` is the old behavior. Dev clusters (`litek8s dev`) stay
AlwaysAllow on `127.0.0.1`. Missing next to upstream: OIDC, static token
and webhook authenticators; the Node and ABAC authorizers (the nodes'
agents talk to the server over their own authenticated connection);
client certificate revocation (certificates are valid for ten years; to
cut one off, rotate the client CA by replacing `/sa.key`, which
also invalidates every service account token); aggregated API servers
(metrics-server) get no front-proxy identity, so their delegated
authorization sees anonymous requests; the escalation check compares rules
one by one (a role whose verbs are covered by several of the author's
rules together may be refused). Sign your own users' certificates with
`/client-ca.{crt,key}` (an Ed25519 CA; `extendedKeyUsage =
clientAuth`). Keep the API port (and the Raft port, 6444) off untrusted
networks anyway. Upgrading a cluster made with an earlier release, and
fixing 401/403: [auth.md](auth.md).
- One person's project, weeks old: APIs, flags and on-disk formats may
change between 0.x releases without migration.
- Not every kubelet behavior: no `Unhealthy`/`BackOff` events, a memory
`emptyDir` is one tmpfs per container mount.
- NetworkPolicies are enforced on crun nodes (Linux) only; the Docker
runtime (macOS servers) stores them without effect. Traffic from a node to
its own pods (probes, NodePort and LoadBalancer relays, which run on the
node) is always allowed, and pods with `NET_ADMIN` can forge source
addresses on their node's bridge
([networking.md](networking.md#networkpolicies)).
- Pods' `imagePullSecrets` are not used: nodes pull with the credentials of
their registry configuration ([registries.md](registries.md)), and the
Docker API cannot `docker push` yet.
- Metrics cover containers, nodes and cluster state; application
`/metrics` endpoints are not scraped (point a Prometheus at them).
- The macOS VM never returns memory to macOS once touched (a
Virtualization.framework limit); litek8s keeps the guest small instead
([dev.md](dev.md#memory-on-macos)).
- macOS servers use Docker for pods and are meant for development; run
real servers and nodes on Linux.
---
# `litek8s dev`: a local Kubernetes cluster for development
`litek8s dev` is a local Kubernetes cluster for your laptop, in the role
kind, k3d, minikube or Docker Desktop's Kubernetes play, without Docker; as
many as you like, each with as many nodes as you like, all in one small VM
(macOS) or one rootless sandbox (Linux).
It is the same `litek8s apiserver` (API server, controllers, scheduler,
cluster DNS, service proxy) with a node agent on crun, packaged so that:
```bash
litek8s dev up # ~0.1 s on Linux, ~1.5 s on macOS; merges context litek8s-dev into ~/.kube/config
kubectl create deployment web --image=nginx
kubectl expose deployment web --type=NodePort --port 80
curl localhost:$(kubectl get svc web -o jsonpath='{.spec.ports[0].nodePort}')
litek8s dev load myapp:dev # an image of your local Docker/Podman, or a `docker save` archive
litek8s dev status # mode, server, pods, memory, disk
litek8s dev down # stop; objects and images are kept
litek8s dev delete # stop and delete everything, and the kube context (`reset` does the same)
litek8s dev up --name ci --nodes 3 # another cluster: context litek8s-ci, 3 nodes
litek8s dev list # every local cluster: state, API port, nodes, memory, age
```
Numbers against kind and k3d are in [bench/RESULTS.md](../bench/RESULTS.md#local-dev-cluster-litek8s-dev-vs-kind-and-k3d);
the corpus score in [compat/RESULTS.md](../compat/RESULTS.md).
## Linux: a rootless sandbox
The cluster runs in namespaces of its own, as your user:
| namespace | what it isolates |
|---|---|
| user | your uid keeps its number inside; your `/etc/subuid` range backs every other id, root included, so image layers keep their owners and files a container creates in a bind mount as your uid are yours on the host (`LITEK8S_DEV_IDMAP=root` makes you root inside instead, Docker-rootless style) |
| network | the `litek8s0` bridge, pod veths and **every** iptables rule (pod NAT, Service DNAT) |
| mount | a fresh `/run` (pod network namespaces, crun state), sysfs, a cgroup2 mount rooted at the cluster's cgroup |
| PID | PID 1 is a small init; stopping it stops every pod process |
| cgroup, UTS, IPC | the node's cgroup tree, hostname `litek8s-dev` |
Pods reach the Internet through [pasta](https://passt.top) (or
slirp4netns), a userspace network stack on the host side, the way rootless
Podman does; `LITEK8S_DEV_NETWORK=slirp4netns` picks slirp4netns where both
are installed (what [CI](ci.md#pasta-slirp4netns-and-githubs-runners) uses). The host side (`litek8s dev supervise`) binds the API port on
`127.0.0.1` before the sandbox exists, and binds NodePorts and LoadBalancer
ports on `127.0.0.1` when the cluster asks for them (it receives the
listening sockets as file descriptors), so `localhost:` works
without kind's `extraPortMappings`.
**What it never touches:** the host's network interfaces, routes and
iptables/nftables rules (Docker's or anyone's), `/run`, other users' files.
Everything it writes is under `~/.local/share/litek8s/dev`.
### What it needs
| need | why | how |
|---|---|---|
| `crun` | runs the containers | `apt install crun` / `dnf install crun` |
| `newuidmap`, `newgidmap` (setuid) | map your subordinate ids into the user namespace | package `uidmap` (Debian/Ubuntu) or `shadow-utils`; usually already installed |
| a subuid/subgid range of 65536 ids | owners inside images | `/etc/subuid`, `/etc/subgid`; `useradd` creates them on current distros |
| `pasta` or `slirp4netns` | pod egress | package `passt` (or `slirp4netns`) |
| `ip`, `iptables-restore` | used inside the sandbox only | `iproute2`, `iptables` (nft or legacy backend) |
| cgroup v2 + a systemd user session | a delegated cgroup for the cluster's resource limits | `dev up` starts the supervisor in a `systemd-run --user --scope -p Delegate=yes` scope |
| Linux ≥ 5.11 | overlayfs in a user namespace (`userxattr`) | |
| **Ubuntu ≥ 23.10 only:** `sudo "$(command -v litek8s)" dev setup`, once | AppArmor restricts unprivileged user namespaces; this installs `/etc/apparmor.d/litek8s` (an unconfined profile with `userns`, like Ubuntu's own profile for crun) for the litek8s binary's path | re-run if you move the binary |
`litek8s dev setup` (without sudo) checks all of this and says what is
missing. As root it also installs the missing packages with apt, dnf, yum,
zypper, apk or pacman (and the AppArmor profile where needed); give sudo the
binary's full path, since its PATH leaves out `~/.local/bin`. That is the whole list: no daemon, no group membership, no
capabilities on the binary, no setuid litek8s. Run as root, `dev up` uses the
same sandbox minus the user namespace (cgroup `/sys/fs/cgroup/litek8s-dev`).
## macOS: a small VM on Virtualization.framework
`dev up` boots a Linux VM with Apple's Virtualization.framework (the
`litek8s dev-vm` process; no Docker Desktop, OrbStack, Lima or vfkit). The
guest is Alpine's `linux-virt` kernel and a read-only root disk (a romfs
image of crun, iptables, iproute2, e2fsprogs and busybox around a static
litek8s that runs as PID 1; a 0.75 MiB initrd mounts it under a tmpfs
overlay and hands over); a sparse disk holds images and objects. The Mac
side:
- forwards `127.0.0.1:` to the API server (6443 when free);
- watches the Services and listens on `127.0.0.1:` for every
NodePort and on `127.0.0.1:` for LoadBalancer ports
(`EXTERNAL-IP` is `127.0.0.1`, as with Docker Desktop's Kubernetes);
- shares `$HOME` (or `--share `) over virtiofs at the **same path**, so
`hostPath: /Users/you/project` volumes work and edits show up live (also
`/private/tmp` and `/private/var/folders`, where `/tmp` and `$TMPDIR` are);
- passes file changes under bind-mounted directories into the VM (FSEvents
on the Mac, IN_ATTRIB in the guest), so inotify watchers see Mac edits;
- shares Rosetta when it is installed, so `linux/amd64` images run;
- serves the Docker API on `~/.litek8s/docker.sock` (see [docker.md](docker.md));
- uses the console as the control channel (the guest reports its address
and its memory, `dev down` powers it off cleanly);
- keeps the guest on as little of its memory as it needs, growing it on
demand ([Memory](#memory-on-macos)).
The framework requires the `com.apple.security.virtualization` entitlement.
Release binaries are ad-hoc signed with it; a binary that lacks it (a
`cargo build`), makes `dev up` run an ad-hoc signed copy
(`codesign --sign -`), so nothing needs `sudo` or an Apple developer
account. VM size: `--memory` (MiB,
default 4096, an upper bound the guest grows into, see below) and `--cpus`
(default half the Mac's).
The guest (kernel, root image, initrd) comes with the macOS release
archive, in `share/litek8s/vm` beside `bin/litek8s` (install.sh puts it in
`~/.local/share/litek8s/vm`); `dev up` looks there first, so a release
binary always boots the guest of the same release. From source, build the
guest once (needs a Docker API or any Linux box for the musl build; a running `litek8s dev` serves one, so litek8s can build its next
guest itself, see [develop-without-docker.md](develop-without-docker.md)):
```bash
dev/vm/build.sh # static litek8s, then kernel + root image + initrd from Alpine packages
```
### Memory on macOS
Virtualization.framework never gives a VM's memory back to macOS: every
page the guest has touched once stays charged to the VM's process (its
footprint, what Activity Monitor shows) until the VM stops, however much
the guest frees. Dropping 1.5 GB of guest page cache moved the footprint by
nothing; its balloon device does not help either (it offers no free page
reporting, and pages handed to it stay charged: inflating it by 3 GiB moved
the footprint by 0 MB, or by +540 MB with Alpine's `init_on_alloc`), and
saving and restoring the VM brings all of its memory back in (4.2 GB
footprint for a guest using 300 MB). Only stopping the VM returns memory;
OrbStack and Docker Desktop got dynamic memory by leaving the framework for
their own VMMs on Hypervisor.framework.
What litek8s does instead is keep the guest off memory it does not need, so
there is less to charge:
- **A ceiling that grows on demand.** The VM boots with 1 GiB (a quarter
of `--memory` if that is more) for the guest; the balloon takes the rest
before the guest has touched it, which costs nothing (the guest kernel
runs with `init_on_alloc=0`, so it does not zero what it hands over).
The guest reports its memory on the console every 2 s when it changes
(every 250 ms when little is left) and at once when a PSI trigger fires
(memory stalls of 100 ms in a second); `litek8s dev-vm` grows the ceiling
by half (at least 512 MiB) when MemAvailable drops under 256 MiB (1/16
of the VM) or memory stalls while free memory is short (stalls with
plenty free come from elsewhere, a pod at its own memory limit for
one), and to the whole VM when it is nearly out.
It shrinks again, by half the excess at a time, once the guest has been
calm for 2 minutes with more than twice that available: page cache then
stays within memory the guest already has instead of spreading over all
of it (an image pull no longer leaves the VM charged for gigabytes), and
the guest drops its oldest cache first (MGLRU); hot cache shows up as
stalls and grows the ceiling back.
- **No livelock under the ceiling.** With the root filesystem on a disk,
the code everything runs is reclaimable page cache; a burst of
allocations used to make the guest evict and re-read it for a minute
(the node agent lost its connection) instead of running out of memory.
The guest protects its working set of the last second
(`lru_gen/min_ttl_ms=1000`), so the kernel takes its OOM path at once,
and there the balloon deflates (the framework's balloon may deflate on
OOM) before anything is killed; the host grows the ceiling meanwhile.
- **A smaller fixed cost.** The root filesystem is a disk image the guest
reads as it needs it (an initramfs is unpacked into RAM and stays there:
56 MiB, plus the 25 MiB archive the framework loads at boot), and the
kernel skips its 64 MiB of bounce buffers (`swiotlb=noforce`: virtio
devices here reach all memory).
`LITEK8S_DEV_DYNAMIC_MEMORY=0 litek8s dev up` gives the guest all of
`--memory` from the start. The ceiling and what moved it are in the VM's
log (`litek8s dev-vm: memory ceiling 1024 -> 1536 MiB (little available;
...)`, `env.log` in the dev data directory), and `litek8s ui` shows it with
the memory breakdown. To hand back memory the VM once used (after a big
build, say), restart it: `litek8s dev down && litek8s dev up` (objects,
images and volumes are kept; pods restart).
What macOS charges the VM (`--cpus 4 --memory 4096`, fresh cluster each
time, `bench/vm-memory.sh`; before: the guest from before these changes,
an initramfs and no ceiling):
| | before | now |
|---|---|---|
| idle, nothing deployed | 384 MB | 295 MB |
| idle, a pod | 461 MB | 389 MB |
| Argo CD installed (7 pods) | 1847 MB | 1140 MB (the ceiling stayed at 1 GiB) |
| then a 1 GB image pulled | 3296 MB | 1200 MB |
| then 5 minutes idle | 3296 MB | 1210 MB |
| then 1 GiB allocated at once in a pod | 0.39 s, 4083 MB | 0.36 s, 1596 MB (ceiling 1 → 4 GiB); no OOM kill either way |
| 1 GiB written, read and deleted in a container | +1058 MB, for good | +733 MB, for good |
After the 1 GiB burst the ceiling came back down within 4 minutes (4096 →
2471 → 1662 → 1257 → 1052 MiB) while a 600 MB file read every 10 s stayed
in the guest's cache throughout (125 ms per read, no refaults). For
comparison, OrbStack's VM (its own VMM) grew by 2.3 GB for the same 1 GiB
container write and was back to where it started 30 s later: that, the
framework does not allow. Details and the experiments behind this: [bench/RESULTS.md](../bench/RESULTS.md#the-macos-vms-memory-a-ceiling-instead-of-a-ratchet-2026-10-19).
### Disk on macOS
The VM's data disk is a sparse file, `vm-state/disk.img` in the dev data
directory (64 GiB by default, `disk` in the settings; only what the guest
writes takes space). Unlike
its memory, the framework does give disk space back: the virtio disk
accepts discards (TRIM) and punches the discarded blocks out of the file.
So the guest trims its filesystem (FITRIM on `/var/lib/litek8s`) at boot
and then whenever the disk has been quiet for 30 s, at most once a minute;
ext4 only discards block groups freed since the last trim, so an idle
trim costs nothing. (Mounting with `discard` would discard on every
delete, in the middle of pulls and container churn.) `docker rmi` and
`docker image prune` free the image's layers at once (layers written in
the last 10 minutes wait for the node's image GC, in case a pull still
needs them), and the Mac has the space back within a minute or so:
| `disk.img` on the Mac (`du`) | before | now |
|---|---|---|
| a container writes 1 GiB, then `docker rm` | stays at 2.6 GB | back to 1.3 GB 55 s later |
| `docker rmi python:3.13` (1.1 GB unpacked) | stays at 1.54 GB | 1.29 → 0.78 GB within 3 min; 123 MB after the node's next image GC (9 min later) |
| boot with 362 MB deleted from an older guest | – | given back by the first trim |
`du` of `disk.img` is what counts; `ls` shows the image's 64 GiB size.
#### More disk (or CPUs, memory) for the VM
Images, volumes, container layers and every cluster's objects live on that
one data disk. Home in `litek8s ui` shows how full it is
(`disk 60.0/64.0 GiB · Mac 150 GB free · grow: 8`), and so do `litek8s
dev status` and `dev list`:
```
disk: 60.2 of 64 GiB (the VM's data disk, a sparse image, the Mac has 140.3 GiB free); more: `disk: ` in ~/.litek8s/config.yaml or `dev up --disk `, applied when the VM starts
```
To give it more, set the new size and restart the VM (it stops with the
last cluster):
```bash
litek8s dev down --all # the VM powers off with the last cluster
litek8s dev up --disk 128 # or `disk: 128` in ~/.litek8s/config.yaml (Settings, 8, in the UI)
```
`dev up` grows `disk.img` to the new size in place before the VM boots
(still sparse: the Mac's free space only goes down as the guest writes),
and the guest grows its ext4 filesystem to match right after mounting it
(online `resize2fs`, a second or so; the VM's log says `growing the data
disk's filesystem from 63.0 to 128.0 GiB`). Clusters, objects, images and
volumes stay where they are. A disk only grows: a `disk` smaller than the
image is ignored with a note (shrinking would cut the filesystem); to
start smaller, `litek8s dev delete` every cluster, stop the VM and remove
`vm-state/disk.img`. With clusters still running, `dev up` says the new
size waits (`note: the VM the clusters share is running with other disk;
they apply once every cluster is down`), and Settings says `Restart
needed for disk`.
Growing the VM disk does not help when the Mac itself is short: the
image can only take what the Mac's disk has free, so Home says `Mac
short` instead of `grow` then; free space on the Mac (or remove images:
Free space, 6, in the UI, `docker image prune`, see above). CPUs (`cpus`,
`--cpus`) and memory (`memory`, `--memory`, MiB) change the same way, at
the VM's next start, without the one-way rule.
### Data directory
The dev home is `dev/` in litek8s's data directory:
`~/Library/Application Support/dev.litek8s` on macOS,
`~/.local/share/litek8s` on Linux (`$LITEK8S_DEV_HOME` overrides the dev
home). The guest assets built from source go in `vm/` next to it
(`$LITEK8S_DEV_VM_ASSETS` overrides).
`LITEK8S_HOME=` puts all of litek8s's per-user files under ``
instead: the data directory becomes `/data` (so the VM, its disk and
the clusters are `/data/dev`, the guest `/data/vm`), the Docker
sockets and `config.yaml` are `/...` instead of `~/.litek8s/...`, the
kubeconfig is `/kubeconfig` (unless `KUBECONFIG` is set) and contexts
are `lkdev-` (`$LITEK8S_CONTEXT_PREFIX`). Two litek8s with
different homes run two VMs that know nothing of each other; that is how a
build from a checkout (`dev/lk`) runs next to an installed release
([develop-without-docker.md](develop-without-docker.md#using-litek8s-vs-developing-it)).
Builds before the rename used `~/Library/Application Support/dev.civojairo.litek8s`
on macOS. litek8s keeps using that directory as long as it exists and
`dev.litek8s` does not, so an existing VM, its disk and its clusters are
never moved behind your back. To move to the new name:
```bash
litek8s dev list # note the clusters that are up
litek8s dev down --all # every cluster; the VM powers off with the last
pgrep -fl 'litek8s dev-vm' # nothing left
mv ~/Library/Application\ Support/dev.civojairo.litek8s ~/Library/Application\ Support/dev.litek8s
litek8s dev up # same clusters, objects and images
```
## Settings
`dev up` takes its defaults from `~/.litek8s/config.yaml` (`$LITEK8S_CONFIG`),
which `litek8s ui` edits ([ui.md](ui.md)); flags win over it:
| setting | flag | macOS | Linux |
|---|---|---|---|
| `cpus` | `--cpus` | the VM's CPUs (default: half the Mac's, at most 8) | a CPU limit on the sandbox (default: none) |
| `memory` (MiB) | `--memory` | the VM's ceiling (default 4096) | a memory limit on the sandbox (default: none) |
| `disk` (GiB) | `--disk` | the VM's data disk (default 64, 16 to 4096): a bigger size grows it in place at the VM's next start, data kept; never shrinks ([More disk](#more-disk-or-cpus-memory-for-the-vm)) | - (the sandbox uses the host's disk) |
| `shares` | `--share` (repeatable) | directories shared at the same path (default `$HOME`) | - (the sandbox sees the host's files) |
| `port` | `--port` | API server port of the default cluster (default 6443, a free one if taken; other clusters keep the one they got) | same |
| `publish` | `--publish` | `localhost` (default) or `all`: where NodePorts, LoadBalancer ports and `docker run -p` ports listen | same |
The file is read at `up`; `env.json` records what the VM or sandbox started
with, each cluster's `dev.json` its own port and nodes. `cpus`, `memory`,
`disk`, `shares` and `publish` belong to the VM or sandbox all clusters
share: they apply when the first cluster starts it.
## Images
- `litek8s dev load ` exports the image from the local Docker (or
Podman's Docker API, via `DOCKER_HOST`) and imports it; `litek8s dev load
file.tar` takes `docker save` / `podman save` archives in Docker or OCI
layout, including Docker 25+'s multi-platform archives (only this host's
platform is used). Loaded images are never pulled over, whatever the pull
policy, so `myapp:latest` works without `imagePullPolicy: IfNotPresent`.
- A registry on your machine's `localhost` works as it is:
`image: localhost:5000/myapp:1` reaches the host's port 5000 from inside the
sandbox (pasta maps the gateway to the host's loopback) or the VM (the
framework's NAT gateway; the registry must listen on all interfaces, which
`docker run -p 5000:5000 registry:2` does).
Other plain-HTTP registries: `LITEK8S_INSECURE_REGISTRIES=host:port`.
- Private registries (a CA of your own, credentials, plain HTTP) and
mirrors: `~/.litek8s/registries.yaml`, k3s's format, which `dev up` gives
the cluster's nodes at each start ([registries.md](registries.md)).
## Several clusters
kind and k3d give every cluster its own node container (a whole systemd or
k3s with its own containerd, ~400-600 MB each); `litek8s dev` gives it a
few processes in the VM or sandbox that is already running:
```bash
litek8s dev up --name ci # context litek8s-ci, its own API port, data and Docker socket
litek8s dev up --name multi --nodes 3 # 3 nodes: litek8s-multi, -worker, -worker2
litek8s dev list
NAME STATE API NODES MEMORY AGE CONTEXT
dev running 127.0.0.1:6443 1 4.2 MiB 7m litek8s-dev
ci running 127.0.0.1:36097 1 3.9 MiB 1m litek8s-ci
multi running 127.0.0.1:41011 3 17.7 MiB 0s litek8s-multi
shared sandbox (rootless, 62.3 MiB in use)
litek8s dev down --name ci # stop one; the VM/sandbox stops with the last one
litek8s dev delete --name ci # and delete its data and context
litek8s dev down --all
```
`dev up|down|status|delete|reset|load` take `--name` (default `dev`, which
keeps the paths, port, context and socket litek8s always used, so an
existing cluster is not migrated). What each cluster gets:
| | default (`dev`) | `--name ` |
|---|---|---|
| kubectl context | `litek8s-dev` | `litek8s-` (`--no-switch` adds it without making it current) |
| API port | 6443, a free one if taken | the one it had last, else a free one |
| Docker socket | `~/.litek8s/docker.sock` | `~/.litek8s//docker.sock` (docker context `litek8s-`) |
| data (Linux) | `~/.local/share/litek8s/dev/cluster` | `~/.local/share/litek8s/dev/clusters//data` |
| data (macOS) | `/var/lib/litek8s/dev` in the VM's disk | `/var/lib/litek8s/` in the VM's disk |
| log | `dev.log` in the dev data directory | `clusters//dev.log` there (the VM's or sandbox's own: `env.log`) |
| node names | `litek8s-dev` | `litek8s-`, `litek8s--worker`, `-worker2`, ... |
| NodePorts from | 30000 | 30100-32700, by name (so two clusters rarely want the same `localhost` port) |
**How.** The first `dev up` starts the VM (macOS) or the sandbox (Linux);
each cluster is then a *cell* in it, made by the same code that makes the
Linux sandbox: its own network namespace (its `litek8s0` pod bridge,
iptables rules, DNS, kubelet and NodePort ports: every cluster uses the
usual 10.42.0.0/16 pods and 10.43.0.0/16 Services without conflict), mount
namespace (`/run`, crun state), PID namespace (stopping it stops all of its
processes), cgroup namespace (`/cells//`: a cluster's memory
is that directory's), and UTS and IPC namespaces. Cells reach each other and
the outside through a bridge in the VM or sandbox (`lkcells0`,
10.231.0.0/24, masqueraded). A small manager inside (`litek8s dev __cells`)
starts and stops cells; the VM process or supervisor outside binds each
cluster's API port and Docker socket on the host (Linux: handed to the cell
as descriptors, so connections are not copied; macOS: relayed over vsock into
the cell's network namespace) and forwards its NodePorts. When the last
cluster stops, the VM or sandbox goes too.
**Several nodes.** `--nodes N` (1-10, remembered) runs N-1 workers, each in
a cell of its own, joined with the real join flow (`litek8s agent --join
https://:6443 --token ...`, see
[multi-node](../compat/cases/M1-multinode.sh)) and connected by the same
WireGuard pod overlay real nodes use: test pod spreading, affinity,
DaemonSets, draining, or a node failing on your laptop:
```bash
litek8s dev down --name multi --node litek8s-multi-worker2 # its pods die; NotReady after the grace period (40 s)
litek8s dev up --name multi --node litek8s-multi-worker2 # back, Ready once it reconnects
```
On Linux the overlay needs
the `wireguard` kernel module loaded (`lsmod | grep wireguard`; a rootless
sandbox cannot load it; most distributions load it on first use by root,
e.g. `sudo modprobe wireguard` once).
**Isolation, honestly.** Cells isolate names, networks, files under `/run`,
processes and cgroups, as containers do; they do not isolate the kernel. All
clusters share:
- the VM's CPUs and memory (macOS) or the sandbox's limits (Linux,
`cpus`/`memory` in the settings): set for all of them, and one busy
cluster slows the others (no per-cluster limits yet);
- the VM's disk: one sparse image for every cluster; what they delete goes
back to the Mac within a minute or so (see [Disk on macOS](#disk-on-macos));
- on Linux, one user namespace: the same subordinate ids map to the same
host ids in every cluster;
- the host's ports: NodePorts, LoadBalancer ports and `docker run -p` ports
of every cluster listen on the same `localhost`; when two clusters want
one port, the first gets it and the other's log says "not exposed";
- the image store is per cluster: an image pulled or `dev load`ed in one
is not in the others (`dev load --name `).
A cluster deleted while the VM is down has its data deleted at the VM's next
boot.
### Memory: 1, 3 and 5 clusters
Each cluster one node, idle 60 s after `dev up` (or `kind create cluster`,
`k3d cluster create`), then with a 3-replica `nginx:alpine` deployment in each
(60 s after the rollout). A cluster's memory is counted the way `docker
stats` counts a container: its cgroup's usage less the page cache the
kernel would drop first. Measured 2026-10-19 with the release build.
**macOS** (M-series MacBook, 14 cores, 36 GB; litek8s VM at `--cpus 4
--memory 4096`; kind v0.33.0 and k3d v5.8.3 with their default images on
OrbStack, as installed):
| clusters | litek8s: whole VM¹ | litek8s: the clusters² | kind: node containers³ | k3d: server + LB containers³ |
|---|---|---|---|---|
| 1, idle | 375 MiB | 2.4 MiB | 564 MiB | 614 MiB |
| 3, idle | 378 MiB | 6.1 MiB | 1574 MiB | 1707 MiB |
| 5, idle | 386 MiB | 9.2 MiB | 2857 MiB | 2818 MiB |
| 1, 3 pods each | 452 MiB | 28 MiB | 671 MiB | 649 MiB |
| 3, 3 pods each | 697 MiB | 82 MiB | 2033 MiB | 1925 MiB |
| 5, 3 pods each | 898 MiB | 135 MiB | 3080 MiB | 3208 MiB |
¹ The VM's physical footprint as macOS counts it (what Activity Monitor
shows for it): the guest kernel, the initramfs (the litek8s binary, crun,
iptables) held in RAM, page cache, and every cluster. ² The sum of the
clusters' cells. ³ `docker stats`, without OrbStack's own VM, which the
kind and k3d columns do not include (it runs anyway for Docker). Creating
the clusters took 9/32/54 s with kind, 17/50/83 s with k3d, and 0.1-0.2 s
each with litek8s once the VM was up (its boot: about 1.5-2.5 s, once).
**Linux** (omarchy: x86_64, 16 threads, 32 GB, kernel 7.2, rootless; kind
and k3d are not installed there, so litek8s only):
| clusters | litek8s: whole sandbox⁴ | litek8s: the clusters² |
|---|---|---|
| 1, idle | 38 MiB | 4 MiB |
| 3, idle | 52 MiB | 13 MiB |
| 5, idle | 63 MiB | 20 MiB |
| 1, 3 pods each | 91 MiB | 57 MiB |
| 3, 3 pods each | 205 MiB | 166 MiB |
| 5, 3 pods each | 317 MiB | 273 MiB |
⁴ The sandbox's cgroup: the supervisor, pasta, the cell manager and every
cluster (and, on Linux, the page cache of the binary, charged to whoever
touched it first).
What it means: an idle cluster costs 2-4 MiB beyond the first and a kind
or k3d cluster about 550 MiB, so five idle litek8s clusters fit in the
memory of one kind node with room to spare. With pods the pods dominate:
nginx starts a worker per CPU the cell sees (4 in the VM, 16 on the Linux
machine). The VM's footprint grows faster than its clusters' sum with
workloads: each cluster pulls and unpacks its own images (the image store is
per cluster), and the guest's page cache stays in the footprint until macOS
asks for it back.
## Limits
- No authentication on the API (`--authorization-mode AlwaysAllow`: it
listens on 127.0.0.1 only; inside the VM or sandbox, on the cluster's
cell, and for multi-node clusters also on its cell address, which only
the other cells reach). RBAC objects are stored, not enforced; servers
installed with `litek8s install` or `cluster apply` enforce them.
- hostPorts are not exposed on the host's localhost (NodePorts are); rootless
cannot bind host ports below 1024.
- macOS: the first boot formats the data disk (~5 s); DHCP from the
framework's NAT occasionally takes 2–3 s when no other VM has run for a
while.
---
# The Docker API: litek8s instead of Docker Desktop / OrbStack
`litek8s dev up` also serves the Docker Engine API, so the real `docker` CLI,
`docker compose`, VS Code / Cursor Dev Containers and Testcontainers run
their containers on litek8s:
```bash
litek8s dev up
export DOCKER_HOST=unix://$HOME/.litek8s/docker.sock
docker run --rm -it alpine sh
docker compose up -d --build
```
**Building with buildx installed** (Docker Desktop, OrbStack and most
Linux packages ship it): `docker build` goes through buildx, so add `--load`
or the image does not show up in `docker images` (nor for pods). Or make
loading the default for this endpoint once, or use litek8s's own builder:
```bash
docker buildx create --use --name litek8s --driver docker-container --driver-opt default-load=true
DOCKER_BUILDKIT=0 docker build -t myapp:dev . # litek8s's builder, no buildx
```
`docker compose up --build` loads its images either way. Details in
[Builds](#builds-two-builders).
It runs **next to** an installed Docker or OrbStack, on its own socket:
nothing of theirs is stopped, changed or used. Leave `DOCKER_HOST` unset to
talk to them as before. To switch with `docker context` instead of an
environment variable (this only adds a CLI context):
```bash
docker context create litek8s --docker host=unix://$HOME/.litek8s/docker.sock
docker context use litek8s # docker context use default (or orbstack) to go back
```
With both engines up, check which one a shell talks to before trusting a
result: `docker version --format '{{.Server.Platform.Name}}'` says
`litek8s` here. Scripts and AI agents should put
`DOCKER_HOST=unix://...` on each command rather than rely on an earlier
`export`: an agent's shell tool often starts a new shell per call, so the
variable is gone and the command reaches the other engine, which looks
like litek8s misbehaving (a container stuck in `created` while "its" port
answers is the other engine's container holding that port). A port the
other engine already publishes fails here with `port is already
allocated`.
The socket is `$LITEK8S_DOCKER_SOCKET` if set. `litek8s dev down` stops the
containers with the cluster; `dev up` brings back the ones with a restart
policy (`always`, `unless-stopped`), as Docker does after a reboot.
## Where containers run
- **Linux:** in the rootless dev sandbox (user, network, mount, PID and
cgroup namespaces of their own, [dev.md](dev.md)), on crun, with no
daemon. The host's network, iptables rules and Docker are not touched.
- **macOS:** in the dev VM (Virtualization.framework), on crun. The Mac
side relays the socket over vsock, listens on `127.0.0.1` **and** `::1`
for every published port, shares `$HOME`, `/private/tmp` and
`/private/var/folders` at the same paths (so `-v $PWD:/app`,
`-v /tmp/x:/x` and Testcontainers' temp files work), passes file change
events into the VM, and shares Rosetta.
### Containers, not pods
A Docker container here is a runtime-level container, not a Kubernetes
Pod. Docker semantics do not fit Pods: a container is created and started
separately, attached to before it runs, stopped and started again **with
its files**, copied into while stopped (Testcontainers does that), lives
on networks of its own with aliases, and keeps its exit code until it is
removed. So the engine (`src/docker/`) keeps its own records and drives
crun through the same per-container shim the pod runtime uses, with the
writable layer kept between runs and an attach socket for live stdin and
stdout. What it shares with the cluster: the image store (an image you
`docker build` can be a pod's `image:` right away, and `kubectl` workloads
reuse pulled layers), the DNS server, the crun shim and bundle code.
`kubectl get pods` does not list Docker containers.
## What works
Tested by `compat/docker/run.sh` (17 cases, all passing on Linux x86_64
rootless and on the macOS VM; run it against Docker or OrbStack too, where
the litek8s-only `compose_budget` case is skipped):
| area | supported |
|---|---|
| `docker run` | `-d`, `--rm`, `-it` (pty, resize, detach keys), `-i` stdin, `-p`/`-P`, `-v`/`--mount` (bind, volume, tmpfs), `-e`/`--env-file`, `-w`, `-u`, `--name`, `--network`, `--network-alias`, `--entrypoint`, `--restart`, `--health-*`, `--add-host` (incl. `host-gateway`), `--platform`, `--cap-add/--cap-drop`, `--privileged`, `--read-only`, `--shm-size`, `--ulimit`, `--sysctl`, `--memory`, `--cpus`, `--pid`/`--network container:` |
| containers | `ps`, `logs` (`-f`, `--tail`, `--since`, `-t`), `exec` (`-it`, `-u`, `-e`, `-w`), `stop`/`start`/`restart`/`kill`/`pause`, `rm`, `inspect`, `rename`, `wait`, `cp` (both ways, also before start), `top`, `stats`, `port`, `prune`, `events` |
| images | `pull` (progress, `--platform`; `docker login` credentials are passed on; CAs, mirrors and credentials for every pull: [registries.md](registries.md)), `images`, `tag`, `rmi`, `inspect`, `history`, `save`, `load`, `prune` |
| build | `docker build` with the classic API: Dockerfile cache, multi-stage, `--target`, `--build-arg`, `.dockerignore`, `-f` outside the context; plus the BuildKit Dockerfile syntax (`RUN --mount=type=cache\|bind\|tmpfs`, `COPY --chmod`/`--link`, heredocs, `TARGETARCH` etc.) |
| networks | `network create/ls/inspect/rm/connect/disconnect/prune`; one bridge per network, names, aliases and short IDs resolve on user-defined networks, networks are isolated from each other; `host.docker.internal` reaches the host |
| volumes | named and anonymous volumes (filled from the image on first use, like Docker), `local` driver bind options (`o=bind,device=`), `volume ls/inspect/rm/prune` |
| compose | `up`/`down`/`ps`/`logs -f`/`exec`/`restart`/`build`/`pull`, project networks with service names and aliases, healthchecks and `depends_on: condition: service_healthy`, env files, named and anonymous volumes, bind mounts, published ports, restart policies, per-service limits (`mem_limit`, `cpus`, `pids_limit`, `deploy.resources.limits`), and a budget for the whole project (next section) |
| Dev Containers | the `devcontainer` CLI (what VS Code and Cursor run): features, `postCreateCommand`, `remoteUser`, `containerEnv`, `mounts`, exec, file ownership |
| Testcontainers | random ports (`-P`) through inspect, log and port wait strategies, exec, copy in/out, the Ryuk reaper (it mounts the Docker socket: any bind source named `docker.sock` is this engine's own socket inside containers) |
## Resource budgets for compose projects
Docker and OrbStack limit containers one by one. A compose stack on a
laptop is better thought of as one machine: litek8s puts every container
of a compose project (label `com.docker.compose.project`) in one cgroup,
`litek8s/docker/projects/`, and that cgroup can have a budget:
- **memory** (`memory.max`, no swap): the whole stack together. When it is
reached the kernel's OOM killer acts *inside the project*: it picks the
biggest process there (the runaway service), not something else in the
VM, and other projects and the cluster never notice. With
**oom_group** every container of the project is killed instead
(`memory.oom.group`), like a server that ran out of memory.
- **cpus** (`cpu.max`): the stack shares that many CPUs.
- **pids** (`pids.max`): a fork bomb stays in its project.
Per-container limits (`mem_limit`, `cpus`, ...) still apply inside the
budget. A budget comes from `~/.litek8s/config.yaml`:
```yaml
docker:
projects: # by compose project name
myapp: {memory: 2Gi, cpus: 2}
kafka-lab: {memory: 6Gi, cpus: 3, pids: 2000, oom_group: true}
default_project: # any project nothing else gives a budget
memory: 4Gi
defaults: # a container that sets no limit of its own (off when absent)
memory: 1Gi
cpus: 1
```
or from a label on any service of the project, so the budget can travel
with the compose file:
```yaml
services:
api:
labels:
dev.litek8s.project.memory: 2Gi
dev.litek8s.project.cpus: "2"
dev.litek8s.project.pids: "1000"
dev.litek8s.project.oom-group: "true"
```
Field by field, the config's entry for the project wins over labels
(your machine, your limits), labels win over `default_project`; when
several services' labels disagree the smallest wins. Sizes are `2Gi`,
`512Mi` or `1g` (binary, as Docker reads them; a bare number is MiB, like
the top-level `memory:`), CPUs `2`, `1.5` or `"500m"`. Labels that do not
parse fail `docker create` with the reason. `defaults` fill in only what
a container leaves unset (memory, cpus, pids); `docker inspect` still
shows what the client asked for.
`litek8s dev up` sends the `docker:` section to the cluster's Docker API,
which keeps it (`/docker/settings.json`) and applies it to running
projects at once; for a cluster that already runs, `litek8s dev up` again
(or saving the file from `litek8s ui`'s Settings) applies an edited file
without restarting anything. Labels apply when a container of the project
starts. `GET /litek8s/v1/docker/projects` on the Docker socket lists each
project's budget and use; `GET`/`PUT /litek8s/v1/docker/settings` read and
replace the settings.
What you see:
- `docker stats` shows the limit that really caps a container: the
smallest of its own, its project's budget and the cluster's (`MEM USAGE
/ LIMIT` is `… / 256MiB` for every service of a 256Mi project), and PIDs.
- An OOM-killed container exits with **137**, `docker inspect` says
`"OOMKilled": true`, and Docker's `oom` event carries which limit it
hit: `litek8s.oom=project` (with `litek8s.project.memory=256Mi`),
`container` (its own `mem_limit`) or `system`. The engine's log says it
in words: `docker: container shop-worker-1 (3f2a…) was OOM-killed: out
of compose project shop's budget (memory 256Mi)`.
- `litek8s ui`'s Containers view puts a memory meter against the budget,
the CPUs in use and the OOM kills on each project's row, marks killed
containers `OOMKilled` and says so in the status line
([ui.md](ui.md#containers-and-budgets)).
`compat/docker/run.sh compose_budget` runs it end to end: a 256Mi project
whose `tail /dev/zero` service is killed while its other service and
another project keep running, and the same with `oom-group`.
## Builds: two builders
The engine reports the classic builder (`Builder-Version: 1`). What runs
your build then depends on the `docker` CLI:
- **Without the buildx plugin**, or with `DOCKER_BUILDKIT=0`, `docker build`
uses **litek8s's builder**: every `RUN` is a crun container on an overlay
of the image's layers whose upper directory becomes the new layer, steps
are cached by a key over the instructions, build args and the content of
copied files (a fully cached rebuild gives the same image ID). Fastest
here; no extra container.
- **With buildx installed** (Docker Desktop, OrbStack and most Linux
packages ship it), `docker build` and compose's builds go through buildx,
whose default builder for an engine without BuildKit is a `moby/buildkit`
container (driver `docker-container`), which buildx starts on litek8s.
Full BuildKit (secrets, SSH mounts, cache export, multi-platform), and
compose loads the result into litek8s. Plain `docker build` needs `--load`
for the image to show up in `docker images`, or a builder that loads by
default:
```bash
docker buildx create --use --name litek8s --driver docker-container --driver-opt default-load=true
```
(builders are per endpoint: this does not change the builder of your
Docker or OrbStack context).
To use litek8s's builder instead: `DOCKER_BUILDKIT=0` for `docker build`,
`DOCKER_BUILDKIT=0 COMPOSE_BAKE=false` for compose (the CLI prints a
deprecation notice for the legacy builder; it still works).
A BuildKit container cannot use images that exist only in litek8s (it pulls
`FROM` images from registries). That matters for **Dev Containers on
Linux**: the CLI's step that changes the remote user's UID builds `FROM`
the image it just built. With buildx installed, run the devcontainer CLI
with the classic builder:
```bash
DOCKER_BUILDKIT=0 devcontainer up --buildkit never --workspace-folder .
```
or, since litek8s keeps your UID the same inside (below), set
`"updateRemoteUserUID": false` in `devcontainer.json` when your UID is the
image user's (usually 1000). On macOS the CLI skips that step.
## File ownership in bind mounts
- **Linux:** the rootless sandbox maps your own UID to itself and backs
every other UID, root included, with your `/etc/subuid` range. A file a
container creates in a bind mount as UID 1000 (a dev container's
`vscode` user, `docker run -u $(id -u)`) is yours on the host, as with
rootful Docker; one created by root belongs to a subordinate id (with
rootful Docker: to root). `LITEK8S_DEV_IDMAP=root litek8s dev up` maps
you to root instead (Docker rootless semantics: root's files are yours,
UID 1000's are not).
- **macOS:** files in shared directories always belong to you, whatever
UID wrote them.
## Hot reload on macOS
virtiofs shows the VM a file's new content right away but tells the VM's
kernel nothing when it changes on the Mac, so inotify watchers in
containers (uvicorn `--reload`, nodemon, Vite, webpack, Air) would miss
edits. The Mac side watches the directories running containers bind-mount
(FSEvents) and the guest re-applies each changed file's mode, which inotify
reports as `IN_ATTRIB`: an edit reaches a watcher in the container in
30-70 ms. A uvicorn app on a FastAPI + Postgres project reloads with the
change about 3.6 s after the save, nearly all of it the app restarting.
## linux/amd64 on Apple silicon
When Rosetta is installed (`softwareupdate --install-rosetta`), the VM
shares it and registers it for x86-64 binaries:
`docker run --platform linux/amd64 ...` and `platform: linux/amd64` in
compose run amd64 images. As with Docker's classic image store, a tag holds
one platform: pulling `alpine` for amd64 replaces the arm64 `alpine:latest`.
## Ports
Published ports listen on the host's `127.0.0.1` (and `::1` on macOS),
never on other interfaces: `docker ps` shows `127.0.0.1:8080->80/tcp`.
`-P` picks free ports. UDP ports are not published (they stay reachable on
the container's address inside the sandbox/VM). Rootless Linux cannot bind
host ports below 1024.
## Not supported (yet)
`docker push`, `commit`, `export`/`import`, `diff`, `docker swarm` and
services, plugins and other volume or network drivers (`overlay`,
`macvlan`, `ipvlan`), IPv6 on container networks, `--init`, `--device`,
GPU, published UDP ports, the BuildKit API itself (`/grpc`, `/session`:
buildx's `docker` driver), `docker stats` beyond CPU, memory and PIDs, `docker update` (accepted, changes nothing),
`checkpoint`. Containers do not survive `litek8s dev down` (restart
policies bring them back on `dev up`).
## Numbers
`bench/docker-bench.sh` (same protocol against any engine; images pulled
beforehand). Linux: Ubuntu 24.04 arm64 guest (qemu+HVF, 4 vCPU, 6 GiB),
litek8s rootless vs Docker 29.1 (Ubuntu's `docker.io`) in the same guest.
macOS: M3 Max, litek8s dev VM vs OrbStack (OrbStack's own containers and
VMs were running).
| | litek8s (Linux) | dockerd (Linux) | litek8s (macOS VM) | OrbStack |
|---|---|---|---|---|
| `docker run --rm alpine true` | **54 ms** | 179 ms | **89 ms** | 295 ms |
| nginx `run -d -p` until it answers | **52 ms** | 135 ms | **91 ms** | 175 ms |
| compose-basic up (postgres healthy, client ran) | **1.7 s** | 2.0 s | **2.8 s** | 3.1 s |
| build, no cache (python app) | **0.05 s**¹ | 0.37 s | 0.74 s¹ | **0.61 s** |
| build, all cached | **0.02 s**¹ | 0.15 s | 0.48 s¹ | **0.28 s** |
| memory, idle² | **35 MiB** | 113 MiB | | |
| memory, postgres + nginx running² | **64 MiB** | 149 MiB | | |
¹ litek8s's builder (`DOCKER_BUILDKIT=0`). Through buildx's BuildKit
container instead: Linux 0.88 / 0.29 s, macOS 11.5 / 0.76 s (the first
build also starts the BuildKit container; exporting the image to litek8s
dominates).
² cgroup `memory.current - inactive_file`: litek8s's whole sandbox (it
includes the Kubernetes API server, controllers and node agent),
dockerd + containerd + the containers' scopes. OrbStack's memory cannot be
isolated from its other machines and containers, so it is not compared.
Real projects run unchanged (in copies, with other host ports where the
user's Docker held them): a FastAPI + Postgres backend on macOS (build with
BuildKit, Alembic migrations, uvicorn hot reload of Mac edits), and on Linux
x86_64 a Prometheus + Grafana stack (provisioned datasource, scraping
`host.docker.internal`), MySQL + Redis with `restart: always`, and a Rails
app with MariaDB built from its Dockerfile (`bundle install` with native
gems and an asset build: 5 min 10 s with litek8s's builder, 6 min 28 s
through buildx's BuildKit container, exporting included).
---
# litek8s in CI
> **If your CI needs Kubernetes, use litek8s.** On GitHub's runners a
> cluster is Ready in **~2.5 s instead of ~45 s**, a Helm or multi-node job
> finishes **3–5× sooner** and the cluster takes **26–37× less memory**
> than kind. In a privileged container (GitLab's `docker:dind` setup) a
> Helm job is **1.8× faster** than kind in Docker in Docker, ready in 9 s
> instead of 40 s. Same kubectl, Helm and manifests: only the setup step
> changes.
A CI job that needs a Kubernetes cluster usually starts kind (a Docker
container per node, ~45 s on GitHub's runners before the nodes are Ready);
one that needs Docker uses the runner's dockerd. litek8s gives both in
about 2.5 s, download included, as the user running the job: `litek8s dev
up` is the same rootless sandbox as on a laptop (API server, controllers,
scheduler, cluster DNS, node agents on crun, a Docker Engine API), so
kubectl, Helm, `docker`, `docker compose` and Testcontainers work
unchanged.
## When it pays off
| the job needs | instead of | what changes | verdict |
|---|---|---|---|
| **a Kubernetes cluster** (kubectl, Helm, operators, e2e) on GitHub's runners | kind, k3d (Docker containers as nodes) | ready in ~2.5 s instead of ~45 s; **3–5× faster jobs, 26–37× less memory** | **switch** |
| **a Kubernetes cluster** in a privileged container (GitLab, runners in containers) | kind in Docker in Docker (`docker:dind`) | ready in 9 s instead of 40 s; **Helm job 39 s instead of 70 s** | **switch** |
| Docker only (build, compose, Testcontainers) on GitHub's runners | the runner's dockerd | ~10 s less per compose job; dockerd needs no setup | small gain |
| Docker only in a privileged container | Docker in Docker | the same: compose job 40 s against 38 s; dind's dockerd starts in ~1 s | no gain |
Numbers: [GitHub's runners](#numbers), [in a container](#in-a-container-docker-in-docker).
## GitHub Actions
```yaml
jobs:
e2e:
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v4
- uses: jairoFernandez/litek8s/.github/actions/setup-litek8s@v0.2.1
- run: |
helm install hello ./chart --wait
helm test hello
```
The action ([action.yml](../.github/actions/setup-litek8s/action.yml))
downloads the release for the runner's architecture and checks it against
the release's SHA256SUMS (with `install.sh`), installs what the rootless
cluster needs if the image lacks it (crun, uidmap, slirp4netns; GitHub's
Ubuntu images have them), runs `sudo litek8s dev setup` for Ubuntu's
AppArmor restriction on user namespaces, starts the cluster and waits for
every node to be Ready. Later steps get `KUBECONFIG` and, unless
`docker: false`, `DOCKER_HOST` (plus `DOCKER_BUILDKIT=0` and
`COMPOSE_BAKE=false`, so builds use litek8s's own builder). If a step of the
action fails it prints `litek8s dev status`, pods, events and the cluster's
logs; `litek8s-diagnose` (on the PATH,
[diagnose.sh](../.github/actions/setup-litek8s/diagnose.sh)) does the same
for your own steps:
```yaml
- if: failure()
run: litek8s-diagnose
```
| input | default | |
|---|---|---|
| `version` | `latest` | a release tag, e.g. `v0.2.1` |
| `binary` | | a litek8s binary to use instead (built earlier in the job) |
| `cluster-name` | `ci` | kube context `litek8s-` |
| `nodes` | `1` | control plane + N-1 workers, joined like real nodes |
| `docker` | `true` | export `DOCKER_HOST` for docker, compose, Testcontainers |
| `builder` | `litek8s` | `buildkit` to build through buildx's BuildKit container instead |
| `network` | `slirp4netns` | pods' egress; `pasta` is litek8s's desktop default (see below) |
| `publish` | `localhost` | `all` to publish NodePorts and ports on every address |
| `wait` | `120` | seconds to wait for the nodes |
Outputs: `version`, `context`, `kubeconfig`, `api-url`, `docker-socket`,
`docker-host`, `ready-seconds`. Linux runners only (`ubuntu-*`, x86_64 and
arm64): on macOS the dev cluster is a VM, and GitHub's macOS runners allow
no nested virtualization.
Pin the action to a release tag (`@v0.2.1` or later), as above; `@main`
follows the repository.
### Examples
[examples/github-actions/](../examples/github-actions/README.md) has three
workflows to copy, each run on every commit of this repository (the
`examples` job of [test.yml](../.github/workflows/test.yml), through
[run.sh](../examples/github-actions/run.sh), which executes an example's
`run:` steps in order):
- **[helm](../examples/github-actions/helm/workflow.yml)**: `helm lint`,
`helm install --wait`, `helm test`, upgrade and rollback of a small chart
(a Deployment behind a Service with a readiness probe, a ConfigMap, a test
pod reaching the Service by DNS). Replaces `helm/kind-action`.
- **[compose](../examples/github-actions/compose/workflow.yml)**:
`docker compose up --build --wait` of a Python app (built from its
Dockerfile) and Redis with healthchecks, tested from the runner through
the published port and from a container on the project network.
Replaces the runner's dockerd.
- **[multi-node](../examples/github-actions/multi-node/workflow.yml)**: three
nodes; a Deployment spread one replica per node by a topology spread
constraint, a DaemonSet, a Service reached from every node, a drained
worker's pod leaving it.
## Numbers
The same examples on the same kind of runner (GitHub's `ubuntu-24.04`,
4 vCPU, 16 GB), with litek8s v0.2.0 from its release, against kind
(`helm/kind-action@v1`, kind v0.33.0, its default node image) and against
the runner's own dockerd; [ci-bench.yml](../.github/workflows/ci-bench.yml),
run by hand. Medians of 5 runs ([run 39559413930](https://github.com/jairoFernandez/litek8s/actions/runs/39559413930), attempts 1 to 5); every job starts cold (no cache: the
images are pulled from Docker Hub in every job, by every engine).
| example | engine | time to ready | example steps | job total | memory idle | memory peak |
|---|---|---|---|---|---|---|
| helm | **litek8s** | **2.6 s** | **10.2 s** | **21 s** | **12 MiB** | **42 MiB** |
| helm | kind | 44.7 s | 22.7 s | 79 s | 449 MiB | 549 MiB |
| multi-node (3 nodes) | **litek8s** | **2.5 s** | **4.5 s** | **15 s** | **28 MiB** | **82 MiB** |
| multi-node (3 nodes) | kind | 49.7 s | 11.1 s | 72 s | 715 MiB | 827 MiB |
| compose | **litek8s** | 2.2 s | **21.0 s** | **31 s** | 13 MiB | 46 MiB |
| compose | runner's dockerd | **0 s** (already running) | 34.0 s | 44 s | 0 MiB | **28 MiB** |
Ranges over the 5 runs: litek8s time to ready 2.1 to 4.1 s, kind 40.8 to
53.2 s; job totals litek8s 13 to 38 s, kind 67 to 85 s, dockerd 37 to 49 s.
- A Kubernetes job is **3.7 to 4.8 times shorter** (median job totals), and
the cluster takes **1/37 (helm) to 1/26 (multi-node) of kind's memory**:
kind runs a whole node (systemd, containerd, kubelet, etcd, the API
server, kube-proxy, CoreDNS, kindnet) per container.
- The example steps themselves are faster too (helm 10.2 s against
22.7 s, multi-node 4.5 s against 11.1 s); these runs do not break down
where kind's time goes (image pulls into the nodes, pod start, Service
rules).
- Against the runner's dockerd, compose builds and starts faster on
litek8s (litek8s's builder against BuildKit's) and the whole job is 13 s
shorter, though dockerd needs no setup at all. Its containers take less
memory than litek8s's whole sandbox, which also holds a Kubernetes API
server and node; dockerd and containerd themselves are not counted.
- Not compared: the runner's preinstalled kind (`kind create cluster`
without the action) and k3d; a cluster image cached between runs. The
litek8s numbers are for v0.2.0 with the action's defaults
(slirp4netns).
### In a container (Docker in Docker)
Where a GitLab.com job runs: a privileged container on the same runner.
Docker in Docker is the `docker:27-dind` image starting its own dockerd
(plus kind v0.33.0 in it for helm); litek8s is
[litek8s-ci.sh](../examples/ci/litek8s-ci.sh) in `ubuntu:24.04`, its apt
packages and download included in the time to ready. The harness's own
tools are in place before the clock starts.
[in-container.sh](../.github/ci-bench/in-container.sh), the
`in-container` job of [ci-bench.yml](../.github/workflows/ci-bench.yml),
litek8s v0.2.1; medians of 5 runs (kind in dind: 4, one `helm test` pod
failed inside kind).
| example | engine | time to ready | example steps | job total | container memory peak |
|---|---|---|---|---|---|
| helm | **litek8s** | **9.2 s** | **12.5 s** | **39 s** | **356 MiB** |
| helm | kind in dind | 40.0 s | 18.5 s | 70 s | 2,960 MiB |
| compose | litek8s | 5.3 s | 19.3 s | 40 s | 353 MiB |
| compose | dind | **1.0 s** | 21.0 s | **38 s** | 345 MiB |
- With Kubernetes, litek8s is ready **4× sooner** and the Helm job is
**1.8× shorter**; kind in dind pulls its ~1 GB node image and runs a
whole node inside the job's container.
- Docker only, it is a draw: dind's dockerd is up in about a second, and
compose runs at the same pace on both.
- The memory peak is the job container's cgroup (`memory.peak`), page
cache included: kind's node image accounts for most of its 2.9 GiB.
What the columns measure:
- **Time to ready**: from the start of the setup step to every node
Ready: download and checks included for litek8s, kind's node image pull
and cluster creation for kind (whose action waits for the control plane;
the job then waits for the nodes, as litek8s's action does). dockerd is
already running on the runner, so it has nothing to wait for.
- **Example steps**: the example's own `run:` steps (image pulls, Helm, the
tests).
- **Job total**: the job as GitHub times it, checkout and the measuring
steps included.
- **Memory idle / peak**: cgroup `memory.current - inactive_file` (what
`docker stats` shows), sampled every 200 ms from the moment the cluster
is ready to the end of the example: for litek8s its whole sandbox (API
server, controllers, node agents, slirp4netns, pods and Docker API
containers), for kind and dockerd every Docker container (kind's node
containers with everything in them; compose's services). The runner's
dockerd and containerd run in every case and are not counted.
Caching the image store between runs (`actions/cache`) is not offered:
the unpacked layers belong to the cluster's subordinate user ids, which the
runner user cannot read back into a cache archive, and pulls from Docker
Hub took 0.7 to 6 s per image on these runners.
For this repository's own CI, see [CI of litek8s itself](#ci-of-litek8s-itself).
## Other CI systems
On any Linux machine or VM a job controls (GitLab CI with a shell or
privileged Docker executor, Jenkins agents, Buildkite, CircleCI's machine
executor), [examples/ci/litek8s-ci.sh](../examples/ci/litek8s-ci.sh) does
what the action does and prints the environment to export:
```bash
eval "$(sh examples/ci/litek8s-ci.sh)"
kubectl get nodes
docker compose up -d --wait
```
Or by hand:
```bash
sudo apt-get install -y crun uidmap slirp4netns
curl -fsSL https://raw.githubusercontent.com/jairoFernandez/litek8s/main/install.sh | sh
export PATH="$HOME/.local/bin:$PATH"
sudo "$(command -v litek8s)" dev setup # Ubuntu >= 23.10: AppArmor allows litek8s user namespaces
LITEK8S_DEV_NETWORK=slirp4netns litek8s dev up --name ci
export DOCKER_HOST="unix://$HOME/.litek8s/ci/docker.sock"
```
GitLab CI (GitLab.com's Linux runners run jobs in privileged containers,
which `docker:dind` needs too; as root the cluster runs without a user
namespace). Not run on GitLab itself: the same container setup is tested
on GitHub's runner, below.
```yaml
k8s-e2e:
image: ubuntu:24.04
script:
- apt-get update -qq && apt-get install -y -qq curl ca-certificates
- eval "$(sh examples/ci/litek8s-ci.sh)"
- curl -fsSLo /usr/local/bin/kubectl "https://dl.k8s.io/release/v1.33.0/bin/linux/amd64/kubectl" && chmod +x /usr/local/bin/kubectl
- kubectl apply -f deploy/ && kubectl rollout status deployment/myapp --timeout=2m
```
`litek8s-ci.sh` is tested in [ci-bench.yml](../.github/workflows/ci-bench.yml)
both ways on GitHub's Ubuntu runner: as the runner user with sudo, and as
root in a privileged `ubuntu:24.04` container. In 5 runs each the cluster
was up and an nginx Deployment rolled out in a median job time of 26 s
(user) and 28 s (container, apt included). In a container `$HOME` sits on
the container's overlayfs, which cannot hold litek8s's own overlay mounts
(as with Docker in Docker); the script puts litek8s's data on a tmpfs
there.
## pasta, slirp4netns and GitHub's runners
Pods (and the node agent pulling images) reach the Internet through a
userspace network stack: pasta when it is installed, else slirp4netns. On
GitHub's Ubuntu runners pasta (Ubuntu 24.04's `passt 2024-02-20`, and the
2026-10-02 static build as well) drops TCP flows now and then: the
connection looks established inside the cluster, the server never sees the
data. Of 5 runs each that started a cluster, pulled two images and started a
pod, 3 hung with Ubuntu's pasta, 1 with the latest pasta and none with
slirp4netns; with the registry client fixed below (three images per run),
pasta still failed 2 of 5, slirp4netns none, and its pulls were as fast
(0.7 to 5.9 s per image against 1.7 to 5.0 s). So:
- the action uses slirp4netns (`network: slirp4netns`, which sets
`LITEK8S_DEV_NETWORK=slirp4netns`; litek8s 0.2.0 does not read that
variable yet, so for it the action moves `/usr/bin/pasta` aside on the
runner);
- litek8s's registry client now gives each address of a registry 10 s to
connect and 15 s for the TLS handshake before it tries the next one, and
reads small responses within a minute: before, a dropped flow left the
pull, and the pod in `ContainerCreating`, hanging for good.
On desktops pasta stays the default; `LITEK8S_DEV_NETWORK=slirp4netns
litek8s dev up` picks slirp4netns there too.
## CI of litek8s itself
[test.yml](../.github/workflows/test.yml) on every push and pull request:
| job | what |
|---|---|
| `cargo test (ubuntu-latest)`, `(macos-latest)` | every unit and integration test with [cargo-nextest](https://nexte.st) |
| `build-linux` | the commit's binary for the jobs below |
| `compat` | the compatibility corpus's API tiers (L0 discovery, L1 objects and CRDs, 50 cases) against `litek8s apiserver --no-node` on the runner: no Docker, no VM |
| `example (helm\|compose\|multi-node)` | the GitHub Actions examples on the commit's binary through the local action |
What made it faster (several runs each, warm caches):
| | before (cargo test) | after (cargo nextest, no debug info) |
|---|---|---|
| `cargo test (ubuntu-latest)` job | 171 s (158 to 213) | **98 s** (95 to 156) |
| its test step (build + run) | 158 s | **85 s** |
| `cargo test (macos-latest)` job | 293 s (237 to 371) | **218 s** (105 to 250) |
| its test step (build + run) | 267 s | **187 s** |
| workflow, first job start to last job end | 298 s (240 to 378) | **224 s** (109 to 257), with the compat and example jobs added |
| test execution alone (ubuntu) | 65 s | 41 to 47 s |
Before: main's test.yml of 690a2d5 rerun on that commit with main's warm
cache, 4 times ([run 39557051657](https://github.com/jairoFernandez/litek8s/actions/runs/39557051657));
after: this test.yml, 5 attempts of [run 39559413919](https://github.com/jairoFernandez/litek8s/actions/runs/39559413919),
warm branch cache. Medians, ranges in parentheses. GitHub's macOS runners
vary the most (105 to 250 s for the same job). The new jobs run beside the
macOS one: `build-linux` 66 s, then `compat` 52 s and the examples 13 to
33 s each.
- **No debug info** (`CARGO_PROFILE_DEV_DEBUG=0`): the crate and its tests
compile and link faster, and the cache is smaller (Linux 268 MiB
instead of 346, macOS 293 instead of 314).
Backtraces keep function names, not line numbers.
- **cargo-nextest**: every test in its own process, test binaries side by
side. `tests/ha.rs` alone took 38 s of `cargo test`'s 65 s, run after the
400 unit tests; now the rest runs beside it. `.config/nextest.toml` keeps
the HA tests one at a time, as `cargo test` did, and stops a hung test
after 5 minutes. There are no doctests, so nothing is lost.
- **build.rs reran on every build in CI**: it asked cargo to watch
`.git/packed-refs`, which a fresh checkout lacks, and cargo reruns a build
script (and rebuilds the crate) for a watched file that does not exist.
Harmless with one `cargo test`, ~50 s with a build step and a test step.
- **The cache**: the last runs on main said "No cache found" because 0.2.0
changed the job matrix, so every cache key; nothing was wrong with the
keys. The cache is now saved from main only (branches and pull requests
restore main's).
Not done: merging the four `tests/*.rs` into one test binary (three fewer
links of the crate). With nextest running them side by side and no debug
info, it was not worth losing `cargo test --test ha` and the per-area files.
`cargo test` still runs every test locally; `cargo nextest run` matches CI.
---
# `litek8s ui`: the dev environment in a terminal
`litek8s ui` is litek8s's answer to OrbStack's and Docker Desktop's windows:
see, configure and operate the local dev environment without remembering
commands. It is part of the same binary, keyboard-driven, fits an 80x24
terminal, and works the same locally and over SSH:
```bash
litek8s ui # this machine's dev clusters
litek8s ui --cluster ci # starting on the local cluster `ci`
ssh -t box litek8s ui # another machine's, in your terminal
```
Nine views, `1`-`9` (or `tab`) to switch, and a tenth, `0`, on clusters
that run Argo CD or litecd; `?` for the keys of each:
| view | what it gives you |
|---|---|
| **1 Home** | the shown cluster's state (up, down, starting, stopping), mode, uptime; its CPU, memory and disk against what the VM or rootless sandbox has; in a box of its own, the Docker socket's containers, images and volumes and what they take; the API server; whether `docker` and `kubectl` point at it. `u` up, `d` down, `R` delete (asks first) |
| **2 Clusters** | the local clusters (state, API port, nodes, memory, age) and your kubeconfig's other contexts; `n` new (name, nodes), `u`/`d` up/down, `D` delete, `enter` shows it in every view (and offers to point `kubectl` and `docker` at it); see [Clusters](#clusters) |
| **3 Containers** | `docker ps -a`, grouped by compose project, with each project's memory against its budget, CPUs and OOM kills: `l` logs (follow), `s` a shell in the container, `u`/`x`/`r` start/stop/restart, `d` remove; on a project row they act on the whole project, `space` folds it |
| **4 Images** | sizes, age, which containers and pods use each; `d` remove |
| **5 Volumes** | sizes, which containers mount each, compose project; `d` remove |
| **6 Free space** | what can be reclaimed: stopped containers, dangling images, build cache, unused images, unused volumes; select with `space`, `p` frees it |
| **7 Kubernetes** | a cluster UI for any cluster or context (litek8s, k3s, EKS, GKE, …), k9s-style: `:` any resource (CRDs too), `/` filters and selectors, describe, YAML, edit with a server dry run and a diff, logs of pods and whole workloads, shells, port-forwards, xray trees; see [Kubernetes](#kubernetes-a-cluster-ui) |
| **8 Settings** | CPUs, memory, the VM's disk (grows in place), shared directories, API port, where published ports listen; switch the docker and kubectl contexts to litek8s and back; edit the settings file in `$EDITOR` |
| **9 Observe** | the cluster's alerts and its pods by CPU or memory with sparklines of the last 30 minutes, from the API server's built-in metrics; every log of a pod (gone containers included), a regex search of every pod's logs, acknowledge alerts; see [Observe](#observe) and [observability.md](observability.md) |
| **0 Apps** | only where the cluster serves `applications.argoproj.io`: Argo CD's Applications (run by Argo CD or by litecd) with sync, health, revision, last sync and what the operation says; one app's resources, sync history and last result; sync (with prune, and force on Argo CD), refresh, hard refresh, terminate a sync (Argo CD); see [Apps](#apps-argo-cd-and-litecd) |
Everything that deletes or stops asks first, and says what goes: removing a
compose project lists its containers, deleting a cluster says it deletes
every object, image and volume and where they are, freeing space adds up
what it frees.
## Look
The UI is drawn in the spirit of [btop](https://github.com/aristocratos/btop):
rounded boxes with their titles set into the border, a header with the tabs
and the shown cluster's badge (state, mode, uptime), meters that go from
green through yellow to red with eighth-of-a-cell precision, history graphs
for CPU and memory, breakdowns as inline bars, and the same status glyphs
everywhere: `●` running, `◐` pending or starting, `✕` failed or crashing,
`○` completed or exited. Lists have bold headers, dim secondary text, rows
colored by state (plain while fine, yellow on the way, red when failing,
dim when done) and, when the terminal says what its background is, a
subtle zebra; the footer's keys are chips. The accent is litek8s's violet.
Before and after, on the Mac with the same cluster (`--cpus 4 --memory
4096`, a 3-replica deployment, a CPU-burning one, a crash-looping pod, a
pending one, a finished job, a compose project), in a dark terminal:
The same in a light terminal, and the Kubernetes view (dark, with the
zebra rows the terminal's reported background allows):
As text, 80x24 before:
```
litek8s 1 Home 2 Clusters 3 Ctrs 4 Images 5 Vols 6 Space 7 Kube 8 Settings
┌ Cluster ui-design ───────────────────┐┌ Resources ───────────────────────────┐
│● up macOS VM, up 48m 12s ││CPU ██ 24% of 1 CPU = 6% of 4 │
│API http://127.0.0.1:51814 ││Memory █████ 644 MB / 4.1 GB │
│Docker …tek8s/ui-design/docker.sock ││Disk 252 MB used, 44.4 GB free │
│Engine API 27.5.1 (aarch64) ││Ctrs 3/5 running Pods 6/9 │
│VM IP 192.168.64.37 ││Images 3 (93.3 MB) │
└──────────────────────────────────────┘└──────────────────────────────────────┘
┌ Memory · 644 MB (what macOS charges the VM) b: next ────────────────────────┐
│litek8s (ui-design) 11.3 MB API server, controllers, node agent │
│pods & containers 26.5 MB │
│Linux kernel + base 331 MB fixed cost: kernel, root fs in RAM │
│cache 130 MB reclaimable │
│freed in the VM 146 MB macOS takes it back when it needs it │
└──────────────────────────────────────────────────────────────────────────────┘
┌ Your tools point at ─────────────────────────────────────────────────────────┐
│✗ docker context default │
│ DOCKER_HOST unset │
│✓ kubectl context litek8s-ui-design (~/.kube/config) │
└──────────────────────────────────────────────────────────────────────────────┘
Switch docker/kubectl to litek8s in Settings (8). Help: ?
u up d down R delete b breakdown 2 clusters ? help
```
and after: one column, the header a single line, the cluster's meters and
pods by state, this cluster's Docker API in a box of its own (its
containers, images, volumes and what they take), the breakdown `b` picks,
and where the tools point when there is room:
```
litek8s DEV 1c5f21c16f3c · … 1 Home 2 3 4 5 6 7 8 9 ● dev · macOS VM
╭─ ● dev · macOS VM · up 6m 18s ───────────────────────────────────── cluster ─╮
│ cpu ▎░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 1% of 7 CPUs · 5% of one │
│ memory ███████████▒▒▒▒▒▒▒▒▒▒░░░░░░░░░ 36% 1.5 GiB in use · VM limit 4.0 GiB │
│ disk ████████▍░░░░░░░░░░░░░░░░░░░░░ 28% 6.0/80.0 GiB · Mac 16.7 GB free │
│ pods ● 21 running │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ docker · ~/.litek8s/docker.sock ─────────── cpu 100.0% of one · mem 200 MB ─╮
│ ● 2/2 running images 25 · 4.6 GB vols 1 · 983 MB │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Memory · 3.2 GB (what macOS charges; VM ceiling 2.6/4.0 GiB) b: next ──────╮
│ litek8s (dev) ▋ 234 MB API server, controllers, node ag… │
│ pods ██▌ 990 MB │
│ Docker containers ▌ 200 MB Docker API containers, builds (n… │
│ Linux kernel + base ▉ 342 MB fixed cost: kernel, page tables │
│ cache ██▎ 880 MB reclaimable │
│ freed, still charged █▍ 555 MB macOS keeps it until the VM rest… │
╰────────────────────────────────────────────────────── bar: VM limit 4.0 GiB ─╯
Switch docker/kubectl to litek8s in Settings (8). Help: ?
u up d down R delete b breakdown c copy API 2 clusters ? help
```
A wide and tall terminal (here 120x40) gets the boxed header, CPU and
memory side by side with their history (one column per sample, 2 s apart,
for as long as Home has been on screen), disk, and the pods and the docker
box under them:
```
╭─ litek8s DEV 1c5f21c16f3c ──────────────────────────────────────────────────────────── ● dev · macOS VM · up 6m 12s ─╮
│ 1 Home 2 Clusters 3 Containers 4 Images 5 Volumes 6 Free space 7 Kubernetes 8 Settings 9 Observe │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ cpu · the VM process ────────────────────── bar: 1 CPU ─╮╭─ memory · what macOS charges… ─── bar: VM limit 4.0 GiB ─╮
│ ││ │
│ ▂ ││ ▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁▁ │
│ ▅▂▁▁▁▂▂▁▂▁▁▂▁▂▃█▂▁▁ ││ ███████████████████ │
│ ▏░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 1% of 7 CPUs · 4% of one ││ ██████▒▒▒▒▒▒░░░░░ 36% 1.5 GiB in use · VM limit 4.0 GiB │
│ litek8s (dev) ▏ 0.2% API server, contro… ││ litek8s (dev) ▍ 234 MB API server, contro… │
│ pods ▏ 1.8% ││ pods █▍ 993 MB │
│ pod openbao/open… ▏ 0.4% ││ Docker containers ▎ 200 MB Docker API contain… │
│ pod argocd/argoc… ▏ 0.4% ││ Linux kernel + base ▌ 340 MB fixed cost: kernel… │
│ pod argocd/argoc… ▏ 0.3% ││ cache █▎ 880 MB reclaimable │
│ Docker containers ██████ 100.0% Docker API contain… ││ freed, still charg… ▊ 554 MB macOS keeps it unt… │
│ container burner ██████ 100.0% ││ │
│ guest kernel & oth… ▏ 1.7% interrupts, networ… ││ │
│ VM overhead ▏ 0.5% virtual devices, v… ││ │
╰──────────────────────────────────────────────────────────╯╰──────────────────────────────────────────────────────────╯
╭─ disk · litek8s and its VM disk ─────────── bar: 7.1 GB ─╮╭─ pods ───────────────────────────────────────────────────╮
│ █████▎░░░░░░░░░░░░░ 28% 6.0/80.0 GiB · Mac 16.7 GB free ││ pods ● 21 running │
│ litek8s itself ▏ 180 MB binary, VM kernel … │╰──────────────────────────────────────────────────────────╯
│ cluster dev █████▍ 6.4 GB │╭─ docker · …cker.sock ─── cpu 100.0% of one · mem 200 MB ─╮
│ images ████ 4.7 GB 1.3 GB reclaimable… ││ containers ● 2/2 running │
│ container layers ▏ 1.4 MB what containers wr… ││ images 25 · 4.6 GB vols 1 · 983 MB │
│ volumes █▌ 1.7 GB 982.5 MB reclaimab… ││ │
│ logs ▏ 4.5 MB pods and container… ││ │
│ builds ▏ 4.1 kB running builds, ca… ││ │
│ objects ▏ 15.8 MB Kubernetes objects… ││ │
│ other ▏ 1.6 MB metadata, IP alloc… ││ │
│ rest of the VM disk ▏ 2.2 MB other clusters, fi… ││ │
│ deleted in the VM ▍ 480 MB disk.img gives it … ││ │
╰──────────────────────────────────────────────────────────╯╰──────────────────────────────────────────────────────────╯
╭─ connections · your tools point at ──────────────────────────────────────────────────────────────────────────────────╮
│ API http://127.0.0.1:58913 ✗ docker context orbstack │
│ Docker ~/.litek8s/docker.sock DOCKER_HOST unset │
│ Engine API 27.5.1 (aarch64) ✓ kubectl context litek8s-dev │
│ VM IP 192.168.64.57 kubeconfig ~/.kube/config │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Switch docker/kubectl to litek8s in Settings (8). Help: ?
u up d down R delete b breakdown c copy API 2 clusters ? help
```
**Terminals.** Nothing needs a Nerd Font: only box-drawing and block
characters and a few common symbols. The UI adapts at start:
| | how it is decided | what changes |
|---|---|---|
| colors | `NO_COLOR` (non-empty): none; `COLORTERM=truecolor`/`24bit`: 24-bit; a `TERM` with `256` (and tmux, kitty, alacritty, ghostty, wezterm): 256; else 16 | 256: each color the nearest of the xterm palette; 16: the terminal's own named colors (made for its background), the selection in reverse video; none: bold, dim, underline and reverse only |
| background | `background:` in ui.yaml; else asked of the terminal once with OSC 11 (see below); else `COLORFGBG` | dark: lighter tones; light: darker ones and pale selections; unknown: mid tones that read on both, and a strong selection with its own text color; zebra rows only when the exact color is known |
| characters | `ascii: true` in ui.yaml, `LITEK8S_UI_ASCII=1`, or `TERM=linux`/`vt100`/`vt220` | box corners `+`, lines `-` `|`, blocks `#`, `●` `*`, `◐` `~`, `✕` `x`, `○` `o`, `…` `~`: a last pass over the finished frame, so the views draw one set of symbols |
| theme | `theme:` in ui.yaml | `default`; `high-contrast`: the terminal's bright colors, bold failures, no gray text or borders; `mono`: no colors at all |
```yaml
# ~/.litek8s/ui.yaml
theme: default # high-contrast, mono
background: auto # dark, light: skips asking the terminal
ascii: false # true: ASCII for fonts without box and block characters
```
Settings (8) shows what was found (`Look: default, 24-bit color, dark
background`).
**Asking for the background** costs nothing that can fail: before the input
reader starts, the UI writes an OSC 11 query followed by a DA1 query and
reads the answers straight from the terminal. Terminals answer in order and
practically all answer DA1, so the wait ends at that answer: 0.1 ms in
tmux on the Mac, 11 ms over SSH to omarchy (one round trip); a terminal
that answers neither costs at most 150 ms once, and the UI starts with the
colors for either background. (The cursor-position query that once failed
the start in terminals that never answer is still not used.) It is skipped
when ui.yaml sets `background`, the theme is `mono`, or there are 16 colors
or fewer.
**Cost.** The history is a ring of 240 samples (two `f32` each), filled by
Home's 2 s samples only while Home is on screen, so it adds no timer and
no wake-up; hidden, nothing samples faster than before. Idle 60 s per view,
main against this look:
| | Home | Containers | Kubernetes |
|---|---|---|---|
| macOS, 120x40, CPU (`ps`) | 0.07 % → 0.08 % | 0.00 % → 0.02 % | 0.00 % → 0.00 % |
| Linux (omarchy), 80x24, CPU (`/proc`) | 0.12 % → 0.12 % | 0.02 % → 0.00 % | 0.00 % → 0.00 % |
| Linux, wake-ups (context switches) per minute | 124 → 130 | 19 → 8 | 13 → 13 |
| Linux, 120x40, Home | 0.12 % → 0.12 %, 137 → 119 wake-ups | | |
The first frame comes 1.0 ms → 1.2 ms after start (Mac, tmux). The
binary grows by 70 KiB stripped on macOS arm64 (35 460 136 → 35 532 200
bytes; 42 798 032 → 42 889 088 as built) and 12 KiB on Linux x86_64
(41 294 312 → 41 306 696), with no new dependency.
## Screens
Captured from real sessions on the macOS VM and, over SSH, the Linux
rootless sandbox; the test runs used scratch paths and a second API port,
shown here as a default install has them (`~/.litek8s`, `~/.kube/config`,
6443).
Home:
```
litek8s DEV 1c5f21c16f3c · … 1 Home 2 3 4 5 6 7 8 9 ● dev · macOS VM
╭─ ● dev · macOS VM · up 6m 18s ───────────────────────────────────── cluster ─╮
│ cpu ▎░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 1% of 7 CPUs · 5% of one │
│ memory ███████████▒▒▒▒▒▒▒▒▒▒░░░░░░░░░ 36% 1.5 GiB in use · VM limit 4.0 GiB │
│ disk ████████▍░░░░░░░░░░░░░░░░░░░░░ 28% 6.0/80.0 GiB · Mac 16.7 GB free │
│ pods ● 21 running │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ docker · ~/.litek8s/docker.sock ─────────── cpu 100.0% of one · mem 200 MB ─╮
│ ● 2/2 running images 25 · 4.6 GB vols 1 · 983 MB │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Memory · 3.2 GB (what macOS charges; VM ceiling 2.6/4.0 GiB) b: next ──────╮
│ litek8s (dev) ▋ 234 MB API server, controllers, node ag… │
│ pods ██▌ 990 MB │
│ Docker containers ▌ 200 MB Docker API containers, builds (n… │
│ Linux kernel + base ▉ 342 MB fixed cost: kernel, page tables │
│ cache ██▎ 880 MB reclaimable │
│ freed, still charged █▍ 555 MB macOS keeps it until the VM rest… │
╰────────────────────────────────────────────────────── bar: VM limit 4.0 GiB ─╯
Switch docker/kubectl to litek8s in Settings (8). Help: ?
u up d down R delete b breakdown c copy API 2 clusters ? help
```
The memory meter is what the cluster's owners hold (litek8s, pods,
other clusters, the kernel and base; not the Docker API's containers,
which the docker box shows) against the VM's `--memory`, the
sandbox's limit or, without one, the host's RAM, as stacked parts; cache
(`▒`, dim) and pages the guest freed but macOS still charges (`▒`, darker)
follow, and do not count as in use. Its percentage is colored by how short
the guest really is (its MemAvailable, plus what the VM's balloon would give
back), not by how full the meter looks: macOS's footprint of the VM also
keeps pages the guest freed long ago and can exceed the VM's size while the
guest has gigabytes available. It turns red, and says `short`, only when
less than a tenth or so is left.
Under the meters, Home explains its figures: what memory, CPU and disk are
made of, as inline bars. Every bar is its row's value against a scale the
box names (`bar: 1 CPU`, `bar: VM limit 4.0 GiB`, `bar: 305 MB`, at the top
right of the box or, when the title needs the room, the bottom right), never
against the largest row, so bar and number agree: a row at 0.1 % of a CPU
is a sliver, one at 50 % half a bar, one past a whole CPU a full bar (its
number says how far past); a value that rounds to nothing prints `<0.1%`
with at most a hairline, and zero has no bar. One box at a time on 80x24 (`b` shows the next; a
tall terminal shows all three). Memory, on the Mac (a fresh VM running the
compat corpus):
```
╭─ Memory · 589 MB (what macOS charges; VM ceiling 1.0/4.0 GiB) b: next ──────╮
│ litek8s (dev) ▏ 11.9 MB API server, controllers, node ag… │
│ pods & containers ▏ 635 kB │
│ Linux kernel + base ▌ 214 MB fixed cost: kernel, page tables │
│ cache ▊ 296 MB reclaimable │
│ freed, still charged ▏ 65.8 MB macOS keeps it until the VM rest… │
╰────────────────────────────────────────────────────── bar: VM limit 4.0 GiB ─╯
```
| line | what it is | from |
|---|---|---|
| litek8s (cluster) | the shown cluster's own processes (API server, controllers, node agent, crun shims) on all its nodes | its cells' cgroups: usage less file pages, less the pods' |
| pods | its pods | the `pod*` cgroups under each node |
| Docker containers | the cluster's Docker API's containers and image builds: they run on its node, but are not the cluster's (the cluster meter leaves them out, the docker box shows them) | the `docker/*`, `build/*` cgroups under each node |
| other clusters | the same three for every other running cluster, together | their cells |
| Linux kernel + base (macOS) | what the VM costs before anything runs: memory the kernel sets aside at boot (its image and the page structures, ~3.5 % of the VM's size), its unreclaimable slab, stacks and page tables, and tmpfs (an older guest's initramfs, the root filesystem held in RAM); also the VM's own init, cell manager and DHCP client | the VM's size less `MemTotal` and the balloon, `/proc/meminfo` |
| sandbox (Linux) | the supervisor, pasta and the cell manager | the sandbox's cgroup less the clusters' |
| cache | file pages: images, logs, the binary; the kernel drops them under pressure | `/proc/meminfo` (VM) or the sandbox's `file` pages |
| freed, still charged (macOS) | pages the guest used once and has freed (or handed to the balloon since): Virtualization.framework keeps charging them to the VM until it stops, however much macOS needs memory (it compresses or swaps them like any other); the guest reuses them before it touches more. Restarting the VM (`dev down`, `dev up`) hands them back | the footprint less what the guest uses |
The title also says how much of its memory the guest may use now when that
is less than all of it: the VM's ceiling, which grows on demand and shrinks
when the guest is idle ([dev.md](dev.md#memory-on-macos)); what is above it
was never touched, and macOS charges nothing for it.
The headline figure is what macOS charges the VM process (its physical
footprint, what Activity Monitor shows) or, on Linux, the sandbox's cgroup.
The guest's numbers come from its cgroups and `/proc/meminfo` through the
VM's control channel; a remote litek8s (`--context`) answers
`GET /litek8s/v1/memory` on its API server with its node's split (litek8s,
pods and containers, cache) and its meminfo, which Home shows instead. It
is sampled with the rest of Home: every 2 s while Home is on screen, not
otherwise.
CPU, between the last two samples (2 s), in % of one CPU, on the Mac with
three busy pods:
```
╭─ CPU · 26.0% of one CPU = 6% of 4 CPUs (the VM process) b: next ────────────╮
│ litek8s (ui-design) ▏ 0.1% API server, controllers, node… │
│ pods & containers ██▌ 25.4% │
│ pod default/burner-f5… ▉ 8.4% │
│ pod default/burner-f5… ▉ 8.3% │
│ pod default/burner-f5… ▉ 8.3% │
│ guest kernel & other ▏ 0.5% interrupts, network, the VM's… │
│ VM overhead ▏ <0.1% virtual devices, vCPU exits (… │
╰───────────────────────────────────────────────────────────────── bar: 1 CPU ─╯
```
| line | what it is | from |
|---|---|---|
| litek8s (cluster) | the shown cluster's own processes on all its nodes | its cells' `cpu.stat` less its pods' |
| pods, and the top 3 | every pod of the cluster, the busiest by name | the `pod*` cgroups (`usage_usec`) |
| Docker containers, and the top 3 | the cluster's Docker API's containers and builds | the `docker/*`, `build/*` cgroups |
| other clusters | the rest of the clusters together | their cells |
| guest kernel & other (macOS) | busy time the guest does not account to a cluster: interrupts, the network stack, the VM's init | the guest's `/proc/stat` less the clusters |
| VM overhead (macOS) | what the VM process ran beyond the guest's busy time: virtual devices, vCPU exits | the VM process's CPU time (macOS) less the guest's `/proc/stat` |
| sandbox (Linux) | the supervisor, pasta and the cell manager | the sandbox's cgroup less the clusters' |
The title's figure is the VM process's CPU time as macOS counts it, or the
sandbox's cgroup. The cluster's cpu meter is that less the Docker
containers' share, and its memory meter what is in use less theirs: the
Docker API's containers run in the same VM or sandbox, but they are not
the cluster's, so Home puts them in the docker box under the cluster
(`docker · ~/.litek8s/docker.sock ── cpu 6.0% of one · mem 1.1 GB`, then
the containers by state, images and volumes). Both meters are against
the VM's `--cpus` and `--memory` (or the sandbox's limits, else the
machine's CPUs), never against the Docker engine's `/info`: with
`--docker-host` the Docker views show another engine, and the docker box
then gives that engine's own size (`its engine: 16 CPUs · 34.4 GB`)
instead of the cluster's.
Disk, on the Mac with the workloads above (the walk is at most a minute
old):
```
╭─ Disk · 305 MB (litek8s and its VM disk) b: next ───────────── bar: 305 MB ─╮
│ litek8s itself ██████▍ 178 MB binary, VM kernel and root fs │
│ cluster ui-design ███▊ 104 MB │
│ images ███▋ 99.9 MB layers shared by pods, Docker, bu… │
│ container layers ▏ 823 kB what containers wrote │
│ volumes ▏ 340 kB Docker volumes, emptyDirs, PVs │
│ logs ▏ 69.6 kB pods and containers, 10 MiB each │
│ objects ▏ 1.9 MB Kubernetes objects, keys │
│ other ▏ 1.1 MB metadata, IP allocations │
│ rest of the VM disk ▏ 2.2 MB other clusters, filesystem │
│ deleted in the VM ▊ 20.9 MB disk.img gives it back within a m… │
╰──────────────────────────────────────────────────────────────────────────────╯
```
| line | what it is |
|---|---|
| litek8s itself | the `litek8s` binary; on macOS also the VM's kernel, root image and initrd and the signed copy of the binary the VM runs from |
| images | the unpacked layers of every image and of the build cache: one content-addressed store per node, shared by pods, Docker and builds, each layer once |
| container layers | what pod containers and Docker containers wrote (their overlay upper directories) |
| volumes | Docker volumes (named and anonymous), pods' emptyDirs, local PVs, ConfigMap and Secret files |
| logs | pod and container logs (10 MiB and one rotated file each) |
| builds | builds in progress and `RUN --mount=type=cache` directories |
| objects | Kubernetes objects (the write-ahead log, compacted) and the cluster's keys |
| other nodes | a multi-node cluster's workers (each with its own image store) |
| other | metadata, IP allocations, staging |
| rest of the VM disk (macOS) | the other clusters and the filesystem's own blocks |
| deleted in the VM (macOS) | blocks the guest freed that `disk.img` still holds: the guest trims its disk once it has been quiet for 30 s (at most once a minute), and Virtualization.framework punches the trimmed blocks out of the sparse image, so this goes back to about zero shortly after a prune |
Where images, container layers and volumes have something to reclaim, the
note says how much and points at the Free space view (6), which frees it:
those amounts come from the Docker API's `/system/df` (unused images and
build cache, stopped containers, unused volumes). The categories come from
`GET /litek8s/v1/disk` on the cluster's API server: one walk of the node's
data directory that counts allocated blocks and hard links once, classifies
each file by where it is (so the parts add up to the whole), and runs at
most once a minute, in the background (Home asks every 30 s while it is on
screen). `/system/df`'s `LayersSize` now counts each layer once and images
report their `SharedSize`, as Docker does.
Against a remote litek8s (`--context`), the three boxes show its API server's
node: `GET /litek8s/v1/memory`, `/litek8s/v1/cpu` (cumulative counters, two
samples make the percentages) and `/litek8s/v1/disk`, and its machine's
`/proc/stat` for the rest of its CPU. Memory is the whole node's, as `free`
sees it: litek8s, pods & containers, the operating system (kernel, its slab
and page tables, services such as sshd or tailscaled) and the page cache, out
of the node's RAM; the graphs fill as on a local cluster. `u`, `d` and `R`
act on this machine's dev clusters, so Home refuses them for a remote one,
and the Docker views (3 to 6), which read this machine's dev cluster's Docker
API unless `--docker-host` names another, say so instead of failing.
Against a cluster that is not litek8s (k3s, kubeadm, a managed one: its
`/version` has no `+litek8s`), there are no such endpoints, so Home reads
the public API instead. The header names what it is (`k3s v1.35.0+k3s1`,
`RKE2`, `EKS`, `GKE`, else `Kubernetes`). CPU and memory are what `kubectl top
nodes` shows: each node's use from `metrics.k8s.io` (metrics-server's
NodeMetrics, asked every 2 s while Home is on screen) against what its Node
object has allocatable (listed every 30 s); the meters add up the cluster or
show the node `n` picked, and each scope's history graph fills from then on.
The breakdowns are per node: memory, CPU and disk as shares of each node's
allocatable (its root filesystem for disk), past 80 % in yellow and past 90 %
in red, with the prefix all node names share left out (`…x7k2p`). Disk comes
from each kubelet's `/stats/summary` through the API server's node proxy
(`nodes/proxy` get, every 30 s); refused by RBAC, Home says "not available"
and stops asking. Without metrics-server the meters show the allocatable
figures and say `metrics.k8s.io not served` instead of waiting. The
connections box says where the figures come from (no Docker, engine or VM:
those are this machine's). Home only reads there; the Kubernetes view stays
read-only unless `W` or `writableContexts`.
Under the CPU and memory meters, a line says what the pods **reserve**
(their requests) and **may use** (their limits) of the nodes shown, against
those nodes' allocatable resources, the way the scheduler and `kubectl
describe node` count them (a pod's requests are the larger of its
containers' sum and its largest init container; Succeeded and Failed pods
reserve nothing), and how many pods have no limit:
```
reserved 10.6 GiB 35% · limits 28.1 GiB 93% · 14 pods without a limit
```
The meter is what is used now; this line is what the scheduler has
promised. Limits above 100% (red) mean the pods could together ask for
more than the nodes have. With several nodes, `n` steps from the whole
cluster to each node, and the line follows. In the Kubernetes view, the
nodes table (`:nodes`) gets the same per node: `CPU REQ`, `CPU LIM`, `MEM
REQ`, `MEM LIM`, each with its share of the node's allocatable, next to
`CPU` and `MEM` (use, from metrics.k8s.io).
After `u` on a stopped cluster, the output of `litek8s dev up` shows under
it (here on Linux, over SSH):
```
╭─ output ─────────────────────────────────────────────────────────────────────╮
│$ litek8s dev up │
│litek8s-dev ready in 0.10s (rootless, http://127.0.0.1:6443, kubectl context │
╰──────────────────────────────────────────────────────────────────────────────╯
```
On Linux the mode is `rootless sandbox`, and the meters are against the
machine (here omarchy's 16 CPUs and 31.3 GiB, over SSH, idle) or, with a CPU
or memory limit set in Settings, against the limit; figures too small to
round to anything read `<1%` and `<0.1%`:
```
litek8s 1 Home 2 Clust 3 Ctrs 4 Imgs 5 Vols 6 Space 7 Kube 8 Set 9 Obs ● dev
╭─ ● dev · rootless sandbox · up 16h 18m ────────────────────────── resources ─╮
│ cpu ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ <1% of 16 CPUs · <1% of one │
│ memory ▎░░░░░░░░░░░░░░░░░░░░░░░░░░░░ <1% 44.6 MB in use · host RAM 31.3 GiB │
│ disk ░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ <1% 191 MB used · 1.5 TB free │
│ pods ● 0 running │
│ ctrs ● 0/1 running ✕ 1 failed images 3 · 122 MB │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ CPU · 0.1% of one CPU = <1% of 16 CPUs (the sandbox) b: next ──────────────╮
│ litek8s (dev) ▏ <0.1% API server, controllers, node agent │
│ pods & containers 0.0% │
│ sandbox ▏ <0.1% supervisor, pasta, cell manager │
╰───────────────────────────────────────────────────────────────── bar: 1 CPU ─╯
```
Containers, with a compose project (a worker that exited with 3) and a
confirmation:
```
litek8s 1 Home 2 Clust 3 Ctrs 4 Imgs 5 Vols 6 Space 7 Kube 8 Set 9 Obs
╭─ Containers ─────────────────────────────────────────────────── 3/5 running ─╮
│ NAME STATE IMAGE PORTS │
│ api ● running nginx:1.27-alpine 18080->80/tcp │
│ scratch ○ exited busybox:1.36 │
│▌▾ shop ◐ 2/3 up compose project │
│ cache ● running redis:7-alpine 6379/tcp │
│ web ● running nginx:1.27-alpine 80/tcp │
│ worker ╭─ ⚠ Stop project shop? ────────────────────╮ │
│ │ shop-cache-1 (redis:7-alpine, running) │ │
│ │ shop-web-1 (nginx:1.27-alpine, running) │ │
│ │ shop-worker-1 (busybox:1.36, exited) │ │
│ │ │ │
│ │ y yes n/esc no │ │
│ ╰───────────────────────────────────────────╯ │
╰──────────────────────────────────────────────────────────────────────────────╯
l logs s shell u start x stop r restart d remove / filter ? help
```
### Containers and budgets
On litek8s a compose project can have a resource budget
([docker.md](docker.md#resource-budgets-for-compose-projects)). Its row
then shows the project's memory against the budget (the meter Home uses),
the CPUs it uses out of its budget, and how many processes the OOM killer
took; a container the OOM killer stopped says `OOMKilled` instead of
`exited`, and the kill shows in the status line as it happens
(`shop-worker-1 was OOM-killed: compose project shop hit its 256Mi budget`):
```
╭─ Containers ─────────────────────────────────────────────────── 2/4 running ─╮
│ NAME STATE IMAGE PORTS │
│▌scratch ○ exited nginx:1.27 │
│ ▾ shop ◐ 2/3 up ████▌░ 75% 192Mi/256Mi cpu 0.5/2 ✗ 1 OOM│
│ web ● running nginx:1.27 8080->80/tcp │
│ db ● running nginx:1.27 8080->80/tcp │
│ worker ✕ OOMKilled nginx:1.27 │
╰──────────────────────────────────────────────────────────────────────────────╯
```
A project without a budget shows its memory (`mem 115Mi · no budget`).
Use is sampled every 2 s while the view is on screen (one request to
`/litek8s/v1/docker/projects`) and not at all otherwise; other Docker
engines have no budgets and are not asked again.
`l` follows a container's (or pod's) log; `↑`/`PgUp` scroll back and stop
following, `G` or `f` follow again, `w` wraps long lines. Escape sequences
are stripped, so colored logs cannot paint over the UI:
```
╭─ logs: api ─────────────────────────────────────────────────────── ● follow ─╮
│ 2026/10/20 05:02:04 [notice] 1#1: OS: Linux 6.12.113-0-virt │
│ 2026/10/20 05:02:04 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 4096:4096 │
│ 2026/10/20 05:02:04 [notice] 1#1: start worker processes │
│ 2026/10/20 05:02:04 [notice] 1#1: start worker process 25 │
│ 2026/10/20 05:02:04 [notice] 1#1: start worker process 26 │
│ 2026/10/20 05:02:04 [notice] 1#1: start worker process 27 │
│ 2026/10/20 05:02:04 [notice] 1#1: start worker process 28 │
╰──────────────────────────────────────────────────────────────────────────────╯
esc back f follow w wrap ↑↓ scroll g/G top/end ? help
```
`s` suspends the UI and opens a shell in the container (bash if it has one,
else sh) with your terminal's size, following resizes; exit the shell to
come back. It is `docker exec -it` over the Docker API, so it needs no
`docker` CLI.
Images:
```
litek8s 1 Home 2 Clust 3 Ctrs 4 Imgs 5 Vols 6 Space 7 Kube 8 Set 9 Obs
╭─ Images ─────────────────────────────────────────────────────── 3 · 93.3 MB ─╮
│ IMAGE SIZE AGE USED BY │
│▌busybox:1.36 4.1 MB 1250d scratch +7 │
│ nginx:1.27-alpine 49.7 MB 551d api +4 │
│ redis:7-alpine 39.5 MB 32d shop-cache-1 │
╰──────────────────────────────────────────────────────────────────────────────╯
d remove / filter 6 free space ? help
```
Free space, and what `p` asks before it frees anything:
```
litek8s 1 Home 2 Clust 3 Ctrs 4 Imgs 5 Vols 6 Space 7 Kube 8 Set 9 Obs
╭─ Reclaimable space ───────────────────────────────────────────── 0 B in all ─╮
│ WHAT ITEMS SIZE │
│▌[✓] Stopped containers 2 0 B │
│ [✓] Dangling images (untagged, unused) 0 0 B │
│ [✓] Build cache 0 0 B │
│ [ ] Unused images (no container or pod) 0 0 B │
│ [ ] Unused volumes (data is lost) 1 0 B │
╰────────────────╭─ ⚠ Free space? ───────────────────────────╮─────────────────╯
Selected 0 B │ Stopped containers: 2 (0 B) │
│ About 0 B freed. This cannot be undone. │
litek8s data 31│ │s.litek8s/dev
disk free 47.│ y yes n/esc no │
╰───────────────────────────────────────────╯
Docker and Kubernetes share one image store: images no container or pod
uses are unused (pods pull them again when needed). Volumes are never
selected by default: they hold data.
space select p free selected ? help
```
Kubernetes, all namespaces (`0`); `ctrl-z` would keep the crashing and
pending ones:
```
litek8s 1 Home 2 Clust 3 Ctrs 4 Imgs 5 Vols 6 Space 7 Kube 8 Set 9 Obs
⎈ litek8s-ui-design ns all │ pods
╭─ pods(all) [9] ──────────────────────────────────────────────────────────────╮
│ NAMESPACE NAME READY STATUS RESTARTS AGE │
│ ● default burner-f57f75b8d-2gpz6 1/1 Running 0 8m40s │
│ ● default burner-f57f75b8d-qvcq2 1/1 Running 0 8m40s │
│ ● default burner-f57f75b8d-vfp2z 1/1 Running 0 11m │
│ ✕ default crashy 0/1 CrashLoopBackOff 14 49m │
│ ○ default migrate-m72pw 0/1 Completed 0 49m │
│ ◐ default waiting 0/1 Pending 0 49m │
│ ● default web-574b8887b7-8xkrz 1/1 Running 0 49m │
│ ● default web-574b8887b7-f84qb 1/1 Running 0 49m │
│ ● default web-574b8887b7-fjmtf 1/1 Running 0 49m │
╰──────────────────────────────────────────────────────────────────────────────╯
: cmd / filter l logs s shell d describe F fwd m menu ? help
```
More of it in [Kubernetes](#kubernetes-a-cluster-ui).
Settings of a cluster started with `--cpus 4` while the file names none (so
its default, 7, would need a restart), and what the UI found about the
terminal:
```
╭─ Settings ───────────────────────────────────────────────────────────────────╮
│ SETTING VALUE │
│▌VM CPUs 7 (default) │
│ VM memory (MiB) 4096 (default) │
│ Shared directories $HOME (default) │
│ API port (dev) 6443 (default) │
│ Published ports on localhost only │
│ docker context default (enter: use litek8s) │
│ kubectl context litek8s-ui-design ✓ (enter: switch back) │
│ Config file ~/.litek8s/config.yaml (enter: $EDITOR) │
╰──────────────────────────────────────────────────────────────────────────────╯
⟳ Restart needed for CPUs: Home (1), d then u.
CPUs, memory and shares size the macOS VM.
Settings apply on the next `litek8s dev up` (its flags win).
Look: default, 24-bit color, background unknown (ui.yaml)
enter edit del default space toggle ? help
```
## Clusters
`litek8s dev` runs as many clusters as you like in one VM or sandbox
([dev.md](dev.md#several-clusters)); view 2 lists them, with the VM's or
sandbox's size and what it uses, and the other contexts of your kubeconfig
(a home cluster, a cloud one): `enter` on one shows it in the Kubernetes
view, [read-only](#read-only-clusters) unless you allow it, and
`showOtherContexts: false` in [`ui.yaml`](#uiyaml) leaves them out of the
list. On the Mac, with a 3-node cluster and a stopped one:
```
litek8s 1 Home 2 Clust 3 Ctrs 4 Imgs 5 Vols 6 Space 7 Kube 8 Set 9 Obs ● dev
╭─ Local clusters ──────────────── shared VM: 4 CPUs, 4.3 GB max, 412 MB used ─╮
│ NAME STATE API NODES MEMORY AGE CONTEXT │
│▌› dev ● up 127.0.0.1:6443 1 27.2 MB 8m litek8s-dev │
│ ci ● up 127.0.0.1:61325 3 5.6 MB 2m litek8s-ci │
│ old ○ down - 1 - 3d litek8s-old │
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Other kubeconfig contexts ── read-only by default · enter: Kubernetes view ─╮
│ home https://192.168.1.10:6443 │
╰──────────────────────────────────────────────────────────────────────────────╯
All clusters share one VM (its CPUs and memory: Settings).
n new u up d down D delete t tree enter use ? help
```
`›` marks the cluster the other views show; `●` up, `◐` starting or
stopping, `○` down. MEMORY is the cluster's own:
its nodes' processes and pods, counted the way `docker stats` counts a
container (the binary and the VM's kernel are shared and not in it); the
title's figure is the whole VM (or sandbox). NODES shows `2/3` while a node
is not Ready.
`n` asks for a name and a node count, and says what the cluster shares:
```
╭─ New cluster ────────────────────────────────────────────────────╮
│ › Name ci2█ │
│ Nodes 1 │
│ │
│ Runs in the shared VM: 4 CPUs, 4096 MiB for all clusters. │
│ Context litek8s-ci2, socket ~/.litek8s/ci2/docker.sock │
│ Nodes after the first are workers joined like real nodes. │
│ │
│ tab next field enter create esc cancel │
╰──────────────────────────────────────────────────────────────────╯
```
and runs `litek8s dev up --name ci2 --nodes 3 --no-switch` (kubectl's
current context stays as it is), its output under the list. `D` says what
goes before it goes (here on Linux):
```
╭─ Delete cluster t1? ───────────────────────────────────────────────────────╮
│Stops it and DELETES all of its data: │
│Kubernetes objects, pods, Docker containers, images, volumes │
│and the build cache, in ~/.local/share/litek8s/dev/clusters/t1/data. │
│The kubectl context litek8s-t1 is removed. This cannot be undone. │
│ │
│ y yes n/esc no │
╰────────────────────────────────────────────────────────────────────────────╯
```
`enter` on a local cluster shows it everywhere (Home, Containers, Images,
Volumes, Free space and Kubernetes follow its Docker socket and API) and
then asks whether `kubectl` and `docker` should point at it too
(`current-context` and the docker context `litek8s-`); the UI
remembers it for next time. `enter` on another context shows that context
in the Kubernetes view only (nothing to start or stop there); `enter` on a
local cluster comes back.
While a cluster or its engine is down, Home says so instead of counting
nothing: `pods –`, `ctrs – (engine not running)`, not `0/0` and `0 (0 B)`.
## Observe
View 9, on the cluster's built-in metrics, logs and alerts
([observability.md](observability.md)):
```
litek8s 1 Home 2 Clust 3 Ctrs 4 Imgs 5 Vols 6 Space 7 Kube 8 Set 9 Obs ● dev
╭─ Alerts ────────────────────────────────────────────── 3 firing · 0 pending ─╮
│ ✕ firing NodeFilesystemAlmostFull 7m Node obs-smoke has 4.853% disk lef…│
│ ◐ firing KubePodCrashLooping 6m default/crasher-778d767f48-4zrkx (…│
│ ◐ firing KubePodRestartLoop 5m default/crasher-778d767f48-4zrkx (…│
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ Pods by CPU ─────────────────────────── % of limit, r: of request · 30 min ─╮
│ NAMESPACE POD CPU/LIMIT % MEM/LIMIT % CPU │
│▌shop api-0 475m/500m 95% 243Mi/256Mi 95% ███████████│
│ shop worker-1 300m/100m r300% 100Mi/200Mi r 50% ▅▅▅▅▅▅▅▅▅▅▅│
│ shop cache-2 20m/— — 50Mi/— — ▁▁▁▁▁▁▁▁▁▁▁│
│ │
╰──────────────────────────────────────────────────────────────────────────────╯
l logs L search logs s sort a/A ack n namespace ? help
```
- **Alerts** first: `✕` firing and critical, `◐` firing, `○` pending or
acknowledged (dim); `a` acknowledges the selected pod's alerts, `A` all of them (each
silenced until it resolves; asks first).
- **Pods** by CPU (cores, `m` = millicores) or memory (working set; `s`
switches), top consumers first. Each figure is shown against the pod's
limit (`used/limit` and the share, colored green to red), summed over its
containers and sidecars; a pod with a container without a limit has none.
Without a limit the request stands in (`r` before the share; above 100 %
is fine there, the pod is just using more than it reserved); with
neither, `/—`. Then a sparkline over the last 30 minutes (one cell per
~2 minutes; a gap where there was no sample): the sorted resource's on
80 columns, both on wide terminals.
`n` cycles namespaces, `/` filters.
- **Logs**: `l` opens every log of the selected pod over the last hour,
following, including containers that restarted or are gone; `L` asks
for a regex and searches every pod's logs (of the namespace shown) over
the last hour, then follows.
It reads `/prometheus/api/v1/query_range` (two queries) and
`/litek8s/v1/alerts` every 10 s while the view is on screen, and nothing
otherwise; against an API server with `--no-observability` it says so.
## Settings: `~/.litek8s/config.yaml`
The Settings view edits a small file that `litek8s dev up` reads (its flags
win over it); `$LITEK8S_CONFIG` moves it. Every change is validated before it
is written: CPUs between 1 and the machine's, memory between 1024 MiB and
its RAM, a disk of 16 to 4096 GiB, a port of 1024 or above, shared directories that exist, are
absolute, do not overlap and are not already shared.
```yaml
cpus: 4 # macOS: the VM's CPUs (default: half the Mac's, at most 8)
# Linux: a CPU limit on everything the dev cluster runs (default: none)
memory: 4096 # MiB; macOS: the VM's ceiling (default 4096); Linux: a limit (default: none)
disk: 64 # GiB, macOS: the VM's data disk; bigger grows it at the VM's next start, never shrinks
shares: # macOS: directories shared with the VM at the same path (default: $HOME;
- /Users/me # /tmp and /var/folders are always shared)
- /Volumes/src
port: 6443 # the API server's port on localhost
publish: localhost # where published container ports and NodePorts listen:
# localhost (default) or all (other machines on the network too)
docker: # Docker API: compose project budgets, container defaults
projects: {shop: {memory: 2Gi, cpus: 2}}
default_project: {memory: 4Gi}
```
The file is read when the cluster starts. The cluster records what it
started with, so the Settings view (and nothing else) tells you which
changes still need `d` and `u` on Home. "Config file" opens it in
`$VISUAL`/`$EDITOR` (`vi` otherwise) and checks it when the editor exits;
a changed `docker:` section goes to the running cluster's Docker API at
once (no restart, see [docker.md](docker.md#resource-budgets-for-compose-projects)).
## Contexts: pointing your tools at litek8s
Home shows, and Settings switches:
- **docker context.** "use litek8s" creates (or updates) a `litek8s`
context for `unix://~/.litek8s/docker.sock` (for another cluster,
`litek8s-` for `~/.litek8s//docker.sock`) and makes it current, with
`docker context` when the CLI is installed and by writing the CLI's
context store (`$DOCKER_CONFIG/contexts`, `config.json`) when it is not.
"switch back" returns to the context that was current before
(`default` if none was recorded). `DOCKER_HOST`, when set in your shell,
wins over any context; Home says so.
- **kubectl context.** "use litek8s-dev" makes the shown cluster's context
(`litek8s-`) current (adding it if the cluster runs); "switch back" restores the
previous current context. Nothing else in the kubeconfig changes.
What was current before is kept in `~/.litek8s/ui-state.json`.
## Kubernetes: a cluster UI
View 7 is a cluster UI in the spirit of [k9s](https://k9scli.io), for any
cluster: the local dev clusters, and every context of your kubeconfig
(k3s, kind, EKS, GKE, AKS, …). Its keys are k9s's where k9s has one, so
k9s habits work; it is lighter (see [the numbers](#kubernetes-performance))
and comes with litek8s.
```bash
litek8s ui --context prod -n payments # start on a context and namespace
```
or `:ctx` inside to pick one. Credentials are what kubectl reads:
`$KUBECONFIG` (all its files, merged as kubectl merges them), tokens, token
files, client certificates, basic auth, and **exec plugins**
(`gke-gcloud-auth-plugin`, `aws eks get-token`, `kubelogin`, …), run when a
token is needed and again when it expires or the server refuses it. Every
request goes to the context the UI shows; `:kubectl` passes it as
`--context`, so the kubeconfig's current context is never used by accident.
### Read-only clusters
Only litek8s's own clusters can be changed from the UI by default. Any
other context (a cloud cluster, a colleague's) is **read-only**: you can
list, describe, read YAML and logs, follow the tree and port-forward, but
deletes, edits, applies, scaling, rollout restarts and undos, cordon and
drain, running a CronJob, shells, `:kubectl` commands that change things
(anything but `get`, `describe`, `logs`, `top`, `rollout status`, …) and
plugins not marked `mutates: false` are refused, with a message saying how
to allow it. The header shows ` read-only ` for such a context.
`W` turns writing on for the context shown: it asks you to type the
context's name (as GitHub does for dangerous actions), and from then on the
header shows a red ` WRITE · `. `tab` in that dialog also saves
it to `writableContexts` in `ui.yaml`; without it, write mode lasts until
the UI exits. `W` again goes back to read-only at once (and takes the
context out of `writableContexts` if it was there). `W` is reserved: your
hotkeys cannot take it.
A context counts as litek8s when it is one of `litek8s dev`'s clusters
(`litek8s-`, `lkdev-` with `$LITEK8S_HOME`, checked against the
clusters on disk) or its API server says so (`/version` reports
`v1.33.0+litek8s`; asked once per context). To change another cluster from
here, list it in `ui.yaml`:
```yaml
writableContexts: [staging, home]
```
`readOnly: true` makes every cluster read-only, litek8s's included; `W`
then says so.
### On other Kubernetes clusters
The Kubernetes view speaks only the standard API, so it works the same on
EKS, GKE, AKS, Civo, k3s, kind or any conformant cluster: discovery
(aggregated, else per group), server-side Tables, list+watch, describe,
YAML, edit with a server dry run, logs, exec (WebSocket `v5`/`v4`),
port-forward (SPDY), the tree and xray, `:kubectl`. Credentials come from
the kubeconfig as kubectl's do (exec plugins, tokens, client certificates,
`tls-server-name`); nothing impersonates anyone. What is litek8s's own, and
what you see elsewhere:
| | on litek8s | on another cluster |
|---|---|---|
| changes (delete, edit, scale, shells, …) | allowed | read-only until `W` or `writableContexts` |
| CPU and MEM columns | from `metrics.k8s.io` when served | the same: shown when the cluster has metrics-server, absent otherwise |
| Observe (9): pod CPU/memory history, alerts, log search | the API server's built-in metrics, alerts and log store | not there: the view says the cluster is not litek8s (logs: `l` on a pod or workload) |
| Home (1) with `--context`: node memory, CPU, disk | litek8s's node endpoints | Nodes' allocatable, `metrics.k8s.io` and the kubelets' stats when allowed ([Screens](#screens)) |
| Docker views (3-6) | the cluster's Docker socket | whatever `--docker-host` points at, independent of the context |
| the tree's Services → pods | from the Service Table's Selector column | the same (kubectl's wide column; older servers: read as objects) |
### Command mode (`:`)
| command | what it shows or does |
|---|---|
| `:pods`, `:po`, `:deploy`, `:dp`, `:svc`, `:ing`, `:no`, `:ns`, `:ev`, `:crd`, `:cm`, `:sec`, `:pvc`, … | that resource type: any type the cluster serves, custom resources and aggregated APIs included, by plural, singular, short name, kind or `plural.group` (`:certificates.cert-manager.io`, `:cert`) |
| `:pods kube-system`, `:pods -n kube-system`, `:deploy -A` | in a namespace, or all of them |
| `:pods -l app=web,tier!=db`, `:pods -f status.phase=Running`, `:pods /nginx` | with a label selector, a field selector (sent to the server), a filter |
| `:ctx` / `:ctx prod` | pick a context / switch to one (the views reconnect) |
| `:ns` / `:ns web` | the namespaces / switch the namespace |
| `:tree`, `:tree shop`, `:tree -A --all` | the cluster as [`litek8s tree`](#tree) draws it, live (below); `--all` adds old ReplicaSets |
| `:xray deploy [ns]` | owner trees: deploy → ReplicaSets → pods → containers (also `sts`, `ds`, `rs`, `job`, `cronjob` → jobs → pods, `svc` → its pods, `node` → its pods) |
| `:kubectl get events -A`, `:!top pods`, `!` | kubectl on the UI's context and namespace; its output in a pager (`exec -it`, `edit`, `attach` get the terminal) |
| `:pf` | the running port-forwards |
| `:aliases` | every resource type, to pick one |
| `:help`, `:q` | the keys; quit |
`tab` (or `→`) completes resource names, commands, namespaces after `-n` or
`:ns`, contexts after `:ctx`; the first candidate shows greyed after the
cursor. `↑`/`↓` walk the history (kept in `~/.litek8s/ui-history`).
Aliases of your own come from [`ui.yaml`](#uiyaml).
Columns are the server's: the same Table kubectl prints (custom resources'
printer columns included), `ctrl-w` adds the wide ones (`-o wide`). Ages are
computed from the creation time, so they move on. Rows are colored by
health: pods by their status and readiness, workloads by ready counts,
nodes by conditions, anything with a Status column by its words.
### Filters (`/`)
The list narrows as you type; `tab` switches the mode:
| filter | keeps |
|---|---|
| `web` | rows whose name or any cell contains it (substring, any case) |
| `^web-\d+` | the same as a regex (mode `regex`) |
| `wbf` | fuzzy: the letters in order, tighter matches first (mode `fuzzy`) |
| `!running` | the rows that do *not* match |
| `-l app=web,tier in (a,b),!canary` | rows whose labels match the selector (matched here, instantly) |
| `-f status.phase=Running` | a field selector, sent to the server (a list of its own) |
They combine (`!running -l app=web`). Each resource type keeps its last
filter while the UI runs, and the table's title shows it. `ctrl-z` keeps
only the rows that are not fine (crashing, pending, not ready); `esc`
clears marks, then the filter, then goes back.
Sorting: `<` and `>` pick the column (an arrow marks it), `O` reverses it,
`N` `A` `S` sort by name, age (newest first), status as in k9s. Numbers sort
as numbers, `2/3` by its counts, `512Mi` and `250m` by size.
### Acting on objects
`m` opens the selected row's actions as a menu; each has a key:
| key | on | does |
|---|---|---|
| `enter` | deploy, sts, ds, rs, job, svc, node | its pods (by its selector, or on the node) |
| `enter` | pod / ns / crd / cronjob | its containers / switch to it / its objects / its jobs |
| `d` / `y` | anything | describe (kubectl's, events at the end) / YAML (`m` there shows managedFields) |
| `e` | anything | edit in `$EDITOR` (below) |
| `ctrl-d`, `del` | anything | delete, asking first; `p` there picks the cascade (Background, Foreground, Orphan), `f` no grace period |
| `ctrl-k` | pod | kill: delete with no grace period (asks) |
| `space` | anything | mark; actions apply to every marked row (`ctrl-\` unmarks) |
| `l` / `p` | pod, container, workload, svc | logs / the previous containers' logs |
| `s` | pod, container | a shell (pick the container when there are several) |
| `s` | deploy, sts, rs | scale |
| `r` / `u` / `H` | deploy (`r`: sts, ds too) | rollout restart / undo (to the previous revision) / history and status |
| `F` | pod, svc, deploy, sts, rs | port-forward (a service's or workload's goes to one of its ready pods) |
| `f` | | the port-forwards (`d` stops one) |
| `t` | anything | the [tree](#tree) of the namespace, the cursor on this object (`t` again: back) |
| `c` / `u` / `r` | node | cordon / uncordon / drain (evictions, honoring PodDisruptionBudgets; DaemonSet pods stay) |
| `T` | cronjob | run it now (a Job from its template) |
| `x` | secret / workloads | decode its data / its xray tree |
| `enter`, `x` / `s` / `p` / `r` / `R` / `T` | Argo CD Application | its resources / sync (asks) / sync and prune (asks) / refresh / hard refresh / terminate the running sync (asks); see [Applications](#argo-cd-applications) |
| `o` / `J` / `E` | pod / anything owned / anything | its node / its owner / its events |
| `c` / `C` | anything | copy the name / the YAML (OSC 52, so it works over SSH) |
| `n` / `0` | | pick a namespace / all namespaces and back |
**Edit** works like `kubectl edit`, with the server checking it first: the
object goes to `$EDITOR` as YAML; what comes back is sent as a dry run
(`PUT ?dryRun=All`), so the API server validates it (schemas, admission
webhooks) without changing anything. A mistake comes back into the editor
as comment lines (`# Error: ...`) over your edit; a valid change shows as a
diff, which `y` applies (`e` edits again, `esc` drops it):
```
╭─ edit deploy/team-0/web (validated by the server; not applied yet) y apply · e edit · esc drop ─╮
│@@ -13,7 +13,7 @@ │
│ uid: 256ba4d2-d410-42be-99f6-f48045d36606 │
│ spec: │
│ progressDeadlineSeconds: 600 │
│- replicas: 3 │
│+ replicas: 4 │
│ revisionHistoryLimit: 10 │
```
### Logs
`l` on a pod with several containers asks which (or all of them); on a
deployment, statefulset, daemonset, job or service it follows every
container of every pod it selects (up to `logs.maxStreams`, 30), each with
a colored prefix, interleaved by time (here nginx's start-up lines):
```
╭─ logs deploy/default/web (all pods) ────────────────── ● follow · 3 streams ─╮
│ fjmtf/nginx 2026/10/20 05:02:04 [notice] 1#1: nginx/1.27.5 │
│ fjmtf/nginx 2026/10/20 05:02:04 [notice] 1#1: built by gcc 14.2.0 (Alpine 14.│
│ fjmtf/nginx 2026/10/20 05:02:04 [notice] 1#1: OS: Linux 6.12.113-0-virt │
│ fjmtf/nginx 2026/10/20 05:02:04 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 4096:│
│ fjmtf/nginx 2026/10/20 05:02:04 [notice] 1#1: start worker processes │
╰──────────────────────────────────────────────────────────────────────────────╯
esc back f follow / filter t time w wrap p previous ^s save ? help
```
`f` pauses and follows again, `/` keeps the matching lines (highlighted;
`tab` for regex or fuzzy), `t` timestamps, `w` wrap, `p` the previous
containers' logs, `c` copies, `ctrl-s` saves to a file (in
`$TMPDIR/litek8s/`), `x` clears. The buffer keeps the last `logs.buffer`
lines (10 000); each stream starts with its last `logs.tail` lines (100).
### Shells and port-forwards
`s` suspends the UI and opens a shell (bash if the image has it, else ash
or sh) in the container, over the API server's exec WebSocket, resized
with the terminal; exiting it comes back to the same view with its exit
code. `F` asks for `local:remote` with the object's ports listed (names
work: `8080:http`); forwards listen on 127.0.0.1 until stopped or the UI
exits, show as `⇄n` in the header, and `f` lists them with their
connection counts.
### xray
```
╭─ xray deployments(all) ──────────────────────────────────────────────────────╮
│ ▾ ● deploy default/burner 3/3 ready │
│ ▾ ● rs burner-f57f75b8d 3/3 ready, revision 1 │
│ ▾ ● pod burner-f57f75b8d-2gpz6 Running 1/1 │
│ ● container burn running, ready, busybox:1.36 │
│ ▾ ● pod burner-f57f75b8d-qvcq2 Running 1/1 │
│ ● container burn running, ready, busybox:1.36 │
│ ▾ ● pod burner-f57f75b8d-vfp2z Running 1/1 │
│ ● container burn running, ready, busybox:1.36 │
│ ▾ ● deploy default/web 3/3 ready │
│ ▾ ● rs web-574b8887b7 3/3 ready, revision 1 │
│ ▾ ● pod web-574b8887b7-8xkrz Running 1/1 │
│ ● container nginx running, ready, nginx:1.27-alpine │
╰──────────────────────────────────────────────────────────────────────────────╯
```
`enter` folds a branch; `d`, `y`, `e`, `l`, `s`, `ctrl-d` act on the row's
object (on a container row: its logs and shell); `ctrl-r` reloads it.
### Argo CD Applications
`:apps` (or `:applications`) lists Argo CD's Applications with their Sync
Status and Health Status columns, colored by both (Degraded red; out of
sync, Progressing or Missing yellow). They are read and driven through the
Kubernetes API alone, so this works with Argo CD or any controller that
reconciles its `argoproj.io` manifests, on any cluster:
| key | does |
|---|---|
| `enter`, `x` | the apps of the namespace down their resources, from each app's status: sync and health of every resource, the health message (`exceeded its progress deadline`), `to prune`, hooks; an app another app manages (app of apps) is drawn under its parent; it follows the apps live (loaded again when one changes, cursor and folds kept) |
| `D` | the diff with git: per resource out of sync, missing or to prune, `-` live and `+` git, only the fields git sets, Secret values masked. Read from the Secret `.diff` next to the app (key `diff.json`, label `gitops.litek8s.dev/diff`), which the controller writes after each comparison: `{revisions, resources: [{group, version, kind, namespace, name, status, desired, live, omitted}]}`, the live object cut to the fields of `desired` (plus those the last apply set), `null` for the side that does not exist. Argo CD keeps its diffs in argocd-server instead (`argocd app diff`) |
| `s` | sync: writes the `operation` the argocd CLI writes (the app's sync options, the hook strategy), after asking; refused while another operation runs |
| `p` | the same with prune: what git no longer has is deleted |
| `r` / `R` | refresh / hard refresh (the `argocd.argoproj.io/refresh` annotation) |
| `T` | terminate the running sync (as `argocd app terminate-op`), after asking |
```
╭─ xray applications(argocd) ──────────────────────────────────────────────────╮
│ ▾ ✕ app web OutOfSync · Degraded @ 0123456 │
│ ✕ deployment web/web OutOfSync · Degraded: exceeded its progress deadli…│
│ ● service web/web Synced · Healthy │
│ ◐ configmap web/old OutOfSync, to prune │
╰──────────────────────────────────────────────────────────────────────────────╯
esc back enter fold s sync r refresh d describe ^r reload ? help
```
Argo CD v3 keeps each resource's health out of the Application's status
(`resourceHealthSource: appTree`: it lives in Argo CD's own cache); where
the status has none, the tree reads the live object and judges it the way
Argo CD does for the kinds that say why an app is not Healthy
(Deployments, StatefulSets, DaemonSets, ReplicaSets, Jobs, Pods, PVCs: a
failed Job's message, a rollout past its progress deadline, a crash loop).
On an app row of that tree `s`, `p`, `r`, `R`, `T` act on it, on a
resource row `d`, `y`, `e`, `l` act on the object; `ctrl-r` reloads it.
Syncs, refreshes and terminations are changes: read-only clusters refuse
them like any other. The same reads and actions are tools of
[`litek8s mcp`](agents.md) (`applications_list`, `application_status`,
`application_sync`, `application_refresh`).
### tree
`t` on any list (or `:tree`, or `t` on a cluster in the Clusters view)
shows what `litek8s tree --context `
prints, and keeps it current: nodes, then each namespace's workloads down
their controllers (CronJob → Job → Pod, Deployment → ReplicaSet → Pod,
StatefulSet, DaemonSet and Job → Pod, pods with no owner) and its services
with how many of the pods they select are ready. Both are drawn by one
builder (`src/tree.rs`); the UI feeds it the rows its informers already
keep for the nine kinds, so the tree costs no requests of its own and
changes as the watches do (a 5 000-pod tree is rebuilt in about 8 ms, at
most once a second while pods keep changing).
```
╭─ tree all namespaces ─────────────────────────────────────────── 11 objects ─╮
│ ├─ ◐ namespace default │
│ │ ├─ ◐ Pod solo Pending │
│ │ └─ ● Service kubernetes ClusterIP 10.43.0.1 443 │
│ ├─ ● namespace kube-system │
│ │ └─ ● Service kube-dns ClusterIP 10.43.0.10 53,53 │
│ └─ ◐ namespace shop │
│ ├─ ● CronJob tick schedule */5 * * * * │
│ ├─ ◐ Deployment api 0/3 ready │
│ │ └─ ◐ ReplicaSet api-6bcdd669f6 0/3 ready │
│ │ ├─ ◐ Pod api-6bcdd669f6-mm7gl Pending │
│ │ ├─ ◐ Pod api-6bcdd669f6-mvswq Pending │
│ │ └─ ◐ Pod api-6bcdd669f6-wn8fn Pending │
│ ├─ ◐ Job once 0/1 done │
│ │ └─ ◐ Pod once-927zc Pending │
│ └─ ◐ Service api ClusterIP 10.43.0.11 80 → 0/3 pods │
╰──────────────────────────────────────────────────────────────────────────────╯
t/esc back space fold enter describe / filter l logs 0 all ns ? help
```
Each line has its state's glyph and color (running and ready green,
pending or partly ready yellow, crashing or failed red); a namespace takes
the worst of what is in it. `space` folds and unfolds a branch (`▸ 3 more`
says what it hides), `→` unfolds, `←` folds or goes up to the parent;
`enter` describes the object (on a namespace line it folds), `d` `y` `e`
`l` `s` `F` `c` `ctrl-d` act on it as in the tables; `/` keeps the
matching lines with what they are under; `0` switches between the
namespace and all of them, `H` shows the old ReplicaSets, `ctrl-r` lists
everything again, `t` (or `esc`) goes back to the list. Opened with `t`
on a deployment, pod, job…, the cursor starts on that object. A cluster in
tree mode stays in it when you switch context. A list the cluster refuses
(`nodes is forbidden`) says so at the top and the rest still shows. On an
older litek8s whose Service Table has no Selector column, the tree reads
the services as objects, so they still show `→ ready/total pods` as
`litek8s tree` does. Your `ui.yaml` hotkeys and plugins come before the
built-in keys: one bound to `t` takes it.
### `ui.yaml`
`~/.litek8s/ui.yaml` (next to `config.yaml`; `$LITEK8S_UI_CONFIG` to put it
elsewhere), every part optional:
```yaml
aliases: # :cp → pods -n kube-system -l tier=control-plane
cp: pods -n kube-system -l tier=control-plane
hotkeys: # a key runs a command line
shift-1: ":pods -A"
f2: "!get events -A --sort-by=.lastTimestamp"
plugins: # your commands, on the selected row
- name: dive
key: ctrl-i
scopes: [pods] # resource names, or all
command: dive
args: [$COL-IMAGE] # $NAME $NAMESPACE $CONTEXT $RESOURCE $CONTAINER $COL-
- name: stern
key: shift-l
scopes: [deploy]
command: stern
args: [-n, $NAMESPACE, $NAME]
background: true # output in a pager instead of the terminal
confirm: false # true: ask first
mutates: false # it only reads: also runs on read-only clusters
theme: default # high-contrast, mono; see Look
background: auto # dark, light
ascii: false # true (or LITEK8S_UI_ASCII=1): ASCII only
skin: # names, 0-255 or #rrggbb, over the theme's colors
accent: cyan
selected: darkgray
warn: yellow
bad: red
logs: {tail: 100, buffer: 10000, timestamps: false, wrap: false, maxStreams: 30}
readOnly: false # true: no deletes, edits, scaling, shells, anywhere
writableContexts: [] # other clusters that may be changed (W with "remember" adds here)
showOtherContexts: true # false: Clusters lists litek8s's clusters only
kubectl: kubectl # which kubectl `:kubectl` runs
```
Hotkeys and plugins are checked before the built-in keys, so they can take
any of them. Variables are also passed to plugins as environment variables.
`skin` colors go over the theme's (see [Look](#look)); `NO_COLOR` and
`theme: mono` ignore them. `--mouse`
selects rows with a click and scrolls with the wheel.
### Kubernetes keys
| where | keys |
|---|---|
| tables | `:` command, `/` filter, `t` tree, `W` write mode, `!` kubectl, `?` help, `m` actions, `enter` open, `esc` back, `n` namespace, `0` all namespaces, `space` mark, `<` `>` `O` `N` `A` `S` sort, `ctrl-w` wide, `ctrl-z` faults, `ctrl-r` reload, `f` port-forwards, and [the actions](#acting-on-objects) |
| pagers (describe, YAML, kubectl, help) | `/` search, `n`/`N` next/previous match, `w` wrap, `←→` scroll sideways, `e` edit, `y`/`d` YAML/describe, `m` managedFields, `c` copy, `ctrl-s` save, `esc` back |
| logs | `f` follow/pause, `/` filter, `t` timestamps, `w` wrap, `p` previous, `c` copy, `x` clear, `ctrl-s` save, `esc` back |
| xray | `enter` fold, `d` `y` `e` `l` `s` `ctrl-d` on the row, `ctrl-r` reload; on an Application row `s` `p` `r` `R` `T` |
| tree | `t`/`esc` back, `space` fold, `←` `→` fold/up and unfold, `enter` describe, `d` `y` `e` `l` `s` `F` `c` `ctrl-d` on the row, `/` filter, `0` all namespaces, `H` old ReplicaSets, `ctrl-r` reload |
`1`-`8` and `tab` still switch views, `q` quits.
### For other views and tools
The Kubernetes view's pieces are separate so other parts of litek8s can use
them: `kube::ops` (one async function per API action), `kube::informer`
(compact, watched lists), `kube::describe`, and extra table columns:
a `KMsg::Ext { kind, column, values }` message adds a column (values by
`namespace/name`) to every table of that kind. The CPU and MEM columns
from `metrics.k8s.io` are one; an observability view can add its own
(request rates, error counts) the same way, without touching the table
code.
### Kubernetes performance
Lists are kept as compact rows (the server's cells, name, labels, owners;
no spec, status or annotations), read straight from the API's pages into
them; a table builds only the rows on screen; filtering and sorting run
over the rows in memory on each keystroke. Measured on an M-series MacBook
against a litek8s API server with no nodes (`litek8s apiserver --no-node`)
holding 50, 1 000 and 5 000 pod objects (two containers each, 50
namespaces), both UIs on the pods of all namespaces in a 120x40 `tmux`
window, release build, k9s v0.50.16 (Homebrew), 60 s idle, RSS and CPU
time from `ps` (perf script: create the pods with the API, start the UI,
wait for the list, sample, filter, quit):
| pods | | startup to the list | RSS | CPU idle | filter keystroke to screen |
|---|---|---|---|---|---|
| 50 | **litek8s ui** | 0.09 s | 19.7 MiB | 0.00 % | 0.5 ms (25 ms on screen) |
| | k9s | 1.87 s | 101.7 MiB | 0.40 % | (24 ms on screen) |
| 1 000 | **litek8s ui** | 0.16 s | 29.4 MiB | 0.02 % | 0.6 ms (22 ms on screen) |
| | k9s | 1.64 s | 153.9 MiB | 3.17 % | (23 ms on screen) |
| 5 000 | **litek8s ui** | 0.83 s | 61.0 MiB | 0.02 % | 1.0 ms (30 ms on screen) |
| | k9s | 3.05 s | 359.7 MiB | 15.4 % | (26 ms on screen) |
| 5 000, 10 updates/s | **litek8s ui** | 1.23 s | 43.6 MiB | 1.28 % | 0.9 ms (26 ms on screen) |
| | k9s | – | 361.3 MiB | 15.2 % | (38 ms on screen) |
- *Startup* is from starting the process to the full list on screen,
watched from outside every 10 ms (it includes `tmux` starting it);
`LITEK8S_UI_STATS=1` puts litek8s's own figure at 45 ms, 112 ms and
788 ms (most of it the API server paging 5 000 pods).
- *Filter keystroke*: the first figure is litek8s's own, from the key to
the frame written (`LITEK8S_UI_STATS`); in brackets, from `tmux
send-keys` to a changed `tmux capture-pane`, which cannot go below about
22 ms: both UIs are at that floor.
- *Idle*: pods bound to a node that does not exist, so nothing changes.
The last two rows are pods the server's scheduler keeps retrying (about
10 pod updates a second at 5 000 pods): litek8s sends a table at most
once a second while changes keep coming and redraws only then; k9s
re-renders its tables every 2 s whether or not anything changed.
- Over SSH on omarchy (Linux x86_64, rootless dev cluster, pods table of a
namespace with two running pods, 100x30): 0 CPU ticks in 60 s, 24 MiB
RSS, 2 threads.
## How it talks to litek8s
Only through what litek8s already serves, so a remote litek8s looks the
same:
| what | from |
|---|---|
| containers, images, volumes, sizes, logs, exec, prune | the Docker Engine API on the socket (`GET /containers/json`, `/system/df`, `/events`, `POST /exec/{id}/start` upgraded to a raw stream, ...) |
| resource types | discovery: aggregated (`/api`, `/apis` as `APIGroupDiscoveryList`, two requests), else the legacy per-group documents |
| every list | `?limit=2000&includeObject=Metadata` as a server-side Table (`as=Table;g=meta.k8s.io`), then a watch from its resourceVersion (Tables too) |
| describe, YAML, edit, delete, scale, rollouts, cordon, drain | `GET`, `PUT` (with `?dryRun=All` first for edits), `DELETE` with `propagationPolicy`, `PATCH .../scale`, merge and JSON patches, `POST .../eviction` |
| logs | `.../log?follow=true×tamps=true&tailLines=N` per pod and container |
| shell | `GET .../exec` upgraded to a WebSocket (`v5.channel.k8s.io`, else `v4.channel.k8s.io`) |
| port-forward | `POST .../pods/{name}/portforward` upgraded to SPDY/3.1 (`portforward.k8s.io`), one session per local connection, listening on 127.0.0.1 |
| CPU and memory columns | `metrics.k8s.io` (when the cluster serves it), every 15 s while pods or nodes are shown |
| `:kubectl`, plugins | `kubectl --context `, the plugin's command, as child processes |
| state, CPU, memory, disk of the dev cluster | the dev supervisor's state file (`dev.json`: mode, port, pid, cgroup or VM process, what it started with), its cgroup or the VM process |
| up, down, delete, new cluster | `litek8s dev up/down/delete --name `, run as a child process (its output shows on Home and Clusters) |
| the local clusters, their memory | the clusters' files in the dev data directory (`cluster.json`, `dev.json`), one request to the VM or sandbox (`env.sock`) for every cluster's memory, `GET /api/v1/nodes` of each running one for its Ready nodes |
```bash
litek8s ui --docker-host tcp://10.0.0.5:2375 --context remote # another litek8s
litek8s ui --docker-host unix:///run/user/1000/litek8s.sock # e.g. a socket forwarded with ssh -L
litek8s ui -n demo # start in a namespace
```
Without flags the UI follows this machine's dev cluster: the Kubernetes side
uses the API port the running cluster recorded (and reconnects when it
moves), and says "not running" rather than falling back to whatever listens
on 6443.
## Apps: Argo CD and litecd
View 0 shows up on clusters that serve Argo CD's Application CRD
(`applications.argoproj.io`), whoever reconciles it: Argo CD, or
litecd, which reads and writes
the same objects. It replaces the basics of Argo CD's web UI with nothing
but the Kubernetes API: the list, an app's resources and history, and the
actions, which patch the Application the way the argocd CLI does, so they
work for either controller and need neither's API server nor a login.
```
litek8s 1 Home 2 Clust 7 Kube 8 Set 9 Obs 0 Apps ● lkmgmt · remote
╭─ Applications · litecd latest · argocd ────────────── 40 apps · 2 OutOfSync ─╮
│ NAME SYNC HEALTH REV SYNCED MESSAGE │
│▌● argo Synced Healthy 0.20.1 5h │
│ ● cert-manager Synced Healthy v1.14.4 23h │
│ ◐ development Synced Progressing f69cca8 23h │
│ ● developme…nt-groveback Synced Healthy f69cca8 5h │
│ ◐ production-env…ifybot OutOfSync Healthy f69cca8 3m │
│ ◐ registry OutOfSync Progressing f69cca8 23h sync running…│
│ ✕ web OutOfSync Degraded 0123456 9m sync failed:…│
╰──────────────────────────────────────────────────────────────────────────────╯
╭─ argocd/argo ─────────────────────────────────── enter: resources & history ─╮
│ source https://argoproj.github.io/argo-helm argo-workflows@0.20.1 │
│ dest in-cluster/argo · auto (prune, self-heal) │
│ status Synced · Healthy compared 11s ago │
╰──────────────────────────────────────────────────────────────────────────────╯
enter open s sync r/R refresh T terminate D diff A appsets ? help
```
- **The list**: every Application of every namespace, by name. The glyph
is health and sync together (`✕` Degraded, `◐` Progressing, Missing, out
of sync or syncing, `●` Synced and Healthy); `REV` the revision it is
synced to (a commit's first 7 characters, a chart's version), `SYNCED`
when its last operation ended (else the last deploy in its history), and
the message of a running or failed operation, else its first condition,
else its health message. The title says which controller runs them (the
Deployment or StatefulSet in the apps' controller namespace: litecd, or
Argo CD's application controller, with its image's tag); the right side
counts the apps, degraded, out of sync, progressing and syncing. Under
the list, the selected app's source(s), destination and sync policy.
Wide terminals (140 columns) add the project and a short source. `/`
filters by name, namespace, project, sync or health (`/OutOfSync`).
- **`A`** switches to the ApplicationSets (when there are any): their
generators, how many apps each owns, and an error condition if one is
set.
- **`enter`** opens the app: its source(s), destination, project and
policy, the revision, when it synced and was last compared, the
operation (phase, who started it, its message), the health message and
conditions; under it the resources of its status with their sync and
health (`requires pruning`, hooks). Argo CD v3 leaves the resources'
health out of the status (`resourceHealthSource: appTree`); there it is
read from the live objects (as the Kubernetes view's tree does) while the
app is open, and child Applications (app of apps) get their own status's
health. `h` cycles the list between resources, the **sync history**
(newest first: id, revision, when, by whom or `auto`, source) and the
**last sync result** (each resource's result and message). `enter` on a
resource describes it in the Kubernetes view; `esc` there comes back.
- **`D`** the diff with git (the [published diff](#argo-cd-applications)
Secret litecd writes; Argo CD keeps its own in argocd-server), `d` the
Application described, both in the Kubernetes view; `esc` comes back.
Actions, on the selected app (list) or the open one:
| key | does |
|---|---|
| `s` | sync: a dialog says what it does; `p` turns on **prune** (deletes what git no longer has), `f` **force** (deletes and recreates what cannot be patched, as `argocd app sync --force`); `y` writes the `operation` the argocd CLI writes (the app's sync options, the hook strategy, `prune`, `hook.force`), `n`/`esc` cancels. Refused while another operation runs. **force is Argo CD only**: litecd has no force sync, so the dialog leaves it out there |
| `r` / `R` | refresh / hard refresh: the `argocd.argoproj.io/refresh` annotation (`normal`, `hard`: render the manifests again) |
| `T` | terminate the running sync (as `argocd app terminate-op`: its phase to `Terminating`), after asking; only while one runs. **Argo CD only**: litecd cannot terminate a sync, and the view says so |
They are changes, so they follow the Kubernetes view's rule: allowed on
litek8s clusters, refused (and the footer says why) on other contexts
unless `W` or `writableContexts` allows them, and everywhere with
`readOnly: true`. The view lists `/apis/argoproj.io/v1alpha1/applications`
and `.../applicationsets` every 5 s while it is on screen and not at all
otherwise (once more right after an action); the controller is looked up
once per cluster (`list` on Deployments and StatefulSets of its namespace;
without that permission it says `controller unknown`). On a cluster
without the CRD the tab is not there and `0` says why.
## Light on the machine
The UI draws only when a key, a window resize or new data changes what is
on screen, and waits on sockets otherwise:
- **No polling for lists.** Containers, images and volumes are re-listed
only after Docker events (`GET /events`), coalesced over 150 ms; sizes
(`/system/df`, which walks directories) only after events that change
them and at most every 2 s. Compose projects' use against their budgets
is the exception: every 2 s while Containers is on screen, never
otherwise. Pods, deployments and namespaces come from
Kubernetes watches, coalesced over 100 ms.
- **Clusters** lists the clusters every 2 s while the view is on screen
(their files, one memory request to the VM or sandbox, a node list from
each running cluster) and not at all otherwise: a cluster nobody looks
at is not polled.
- **Home's breakdowns** come with its samples: memory and CPU counters
every 2 s while Home is on screen (on the Mac one request over the VM's
control channel), the disk categories every 30 s from a walk the API
server repeats at most once a minute; nothing while Home is hidden.
Measured on omarchy, Home with the CPU box over a busy pod: 0.08 % CPU,
1.8 wake-ups/s, 18 MiB RSS.
- **Home** samples the shown cluster's state, CPU and memory every 2 s while Home
or Free space is on screen, every 10 s on Settings (for its restart
note), every 60 s otherwise (to notice the API port moving); the data
directory's size (a directory walk) at most every 30 s and only while it
is shown. When the engine is down it is retried with backoff up to 10 s.
- **The look** adds no timer: Home's history graphs are filled by its own
2 s samples while it is on screen, and the terminal's background is
asked once, before the first frame (see [Look](#look)).
- **Input** comes from crossterm's event stream, which waits in the
terminal reader (no timer); while a shell or an editor has the terminal
the stream is dropped, so nothing else reads keystrokes. The only timers
are a status message fading after 8 s and, on the views that show ages,
a redraw every 30 s.
- **Two threads**: `litek8s ui` runs on a single-threaded async runtime
(the servers in the same binary use one thread per core) plus
crossterm's terminal reader; a blocking-pool thread comes and goes for
the data directory walk. The log view keeps the last 5000 lines, each
cut at 4096 characters.
Measured on Linux x86_64 (omarchy, release build, rootless dev cluster with
a 3-container compose project and a 3-replica deployment, 80x24 in `tmux`,
after two window resizes), idle 60 s in each view; CPU from
`/proc//stat`, wake-ups are voluntary context switches of all its
threads:
| view | CPU | wake-ups | RSS | threads |
|---|---|---|---|---|
| Home (samples every 2 s) | 0.07 % | 3.6/s | 16.5 MiB | 3 (the size walk's blocking thread) |
| Clusters (2 clusters, one with 3 nodes; every 2 s) | 0.08 % | 2.3/s | 17.3 MiB | 2 |
| Containers | 0.00 % | 0.3/s | 16.5 MiB | 2 |
| Kubernetes | 0.02 % | 0.4/s | 16.5 MiB | 2 |
| Settings | 0.02 % | 1.0/s | 16.6 MiB | 2 |
| a log following 1 line/s | 0.03 % | 2.4/s | 19.1 MiB (peak) | 2 |
(The Clusters row is from a later round with named clusters, same machine and
method; in that round Home measured 0.03 % and 1.8/s, Containers 0.00 % and
0.1/s: with Clusters off screen, nothing samples the clusters.)
On macOS (M-series MacBook, `litek8s dev up --cpus 4 --memory 4096`, the
same workload, 80x24 in `tmux`), idle 60 s in each view; CPU from `ps`
CPU time, idle wake-ups and footprint (what Activity Monitor calls Memory)
from `top`:
| view | CPU | idle wake-ups | footprint | RSS | threads |
|---|---|---|---|---|---|
| Home (samples every 2 s) | 0.12 % | 0.02/s | 6.5 MiB | 20.2 MiB | 3 |
| Containers | 0.00 % | 0.02/s | 6.5 MiB | 20.2 MiB | 2 |
| Kubernetes | 0.02 % | 0.02/s | 6.5 MiB | 20.2 MiB | 2 |
| Settings | 0.02 % | 0.02/s | 6.5 MiB | 20.2 MiB | 2 |
| a log following 1 line/s | 0.07 % | 0.03/s | 6.6 MiB | 20.2 MiB | 2 |
(RSS counts the shared pages of the 39 MB binary; the footprint is the UI's
own memory.)
Startup: on Linux the first frame is drawn 0.7 ms after the UI starts and
the first one with the engine's containers 2.7 ms after; on the Mac 1.6-2.8
ms and 7-13 ms (the Docker socket is relayed into the VM over vsock).
`LITEK8S_UI_STATS=1 litek8s ui` prints these, and the number of frames
drawn, when it exits: 156 frames in 315 s for the Linux run above, nearly
all of them Home samples and log lines.
Size: the UI (ratatui and crossterm, no other new dependency) adds to the
release binary:
| | main (c47079e) | with the UI | delta |
|---|---|---|---|
| macOS arm64, stripped | 30.86 MB | 32.43 MB | +1.49 MiB |
| macOS arm64, as built | 37.39 MB | 39.28 MB | +1.80 MiB |
| Linux x86_64, stripped | 35.95 MB | 37.63 MB | +1.61 MiB |
| Linux x86_64, as built (with symbols) | 45.49 MB | 47.65 MB | +2.06 MiB |
Most of it is the UI's own code (about 650 KiB of symbols); ratatui and
crossterm are about 200 KiB.
The Kubernetes view's cluster UI (src/ui/kube) adds 1.92 MiB more to the
stripped macOS arm64 binary (33.41 MB on main c620693, 35.43 MB with it;
40.37 → 42.76 MB as built), with no new dependency. The [look](#look)
(src/ui/theme.rs) adds 70 KiB on macOS and 12 KiB on Linux.
## Keys
| where | keys |
|---|---|
| everywhere | `1`-`9`, `0`/`tab`/`shift-tab` views, `↑↓`/`jk`/`PgUp`/`PgDn`/`g`/`G` move, `/` filter the list (`esc` clears), `ctrl-r` refresh, `?` help, `q` or `ctrl-c` quit |
| Home | `u` up, `d` down (asks), `R` delete the cluster (asks), `b` the next breakdown (memory, CPU, disk), `c` copy the API address |
| Clusters | `n` new (`tab` between name and nodes), `u` up, `d` down (asks), `D`/`del` delete (asks), `enter` show it (a local cluster: every view, then asks about `kubectl`/`docker`; another context: the Kubernetes view), `t` straight into its tree, `c` copy its context name |
| Containers | `l`/`enter` logs, `s` shell, `u` start, `x` stop (asks), `r` restart (asks), `d`/`del` remove (asks), `space` fold a project |
| Images, Volumes | `d`/`del` remove (asks) |
| Free space | `space` select a category, `p`/`enter` free the selected (asks) |
| Kubernetes | see [its keys](#kubernetes-keys); `?` there lists the ones of what is shown |
| Settings | `enter` edit, `del` back to the default, `space` toggle / switch a context |
| Observe | `l`/`enter` the pod's logs, `L` search every pod's logs, `s` sort by CPU/memory, `n`/`N` namespace, `a` acknowledge the pod's alerts, `A` all (asks) |
| Apps | `enter` open the app (`esc` back), `h` resources / sync history / last result, `s` sync (`p` prune, `f` force; asks), `r`/`R` refresh / hard refresh, `T` terminate the sync (asks), `D` diff, `d` describe, `A` ApplicationSets; on a resource `enter` describes it |
| logs | `f` follow, `w` wrap, `c` clear, `esc` back |
`--mouse` turns on clicks (tabs, rows) and the wheel; it is off by default
because it takes over the terminal's own text selection.
Copying: `c` puts the shown text on the clipboard (OSC 52, which reaches
terminals over SSH and tmux, and `pbcopy` on a Mac); in the Kubernetes view
`c` copies a name and `C` a YAML. The terminal's own selection works too, but
the screen redraws every 2 s, which can clear it in some terminals; on macOS,
option-drag selects even with `--mouse`.
## Limits
- Describe is kubectl's for pods, deployments, nodes and services, and its
generic form (fields as "Title Case" keys) for every other type.
- Rollout undo and history are for deployments (StatefulSets' and
DaemonSets' ControllerRevisions are not read).
- With `publish: all` the ports listen on every interface, but `docker ps`
still shows them as `127.0.0.1:`: the engine in the VM or sandbox
does not know where the host side listens.
- Exec credential plugins that return client certificates (rather than a
token) are not supported; `auth-provider` only with a token in the
kubeconfig (OIDC's `id-token`).
---
# Observability: metrics, logs and alerts, built in
A home or edge cluster wants to know what its pods cost, find the error in
last night's logs and get a message when a disk fills or a pod crash-loops.
The usual answer, kube-prometheus-stack plus Loki, costs more than litek8s
itself (see [the measurements](#measured)). litek8s has it built in:
- **Metrics**: every node agent samples its machine and every container
(cgroup v2: CPU, memory split into anonymous and page cache, I/O, OOM
kills, throttling; the pod's network on its veth; persistent volume use)
every 10 s; the API server adds object state (kube-state-metrics' names)
and its own request metrics. All of it goes into an embedded time-series
store: Gorilla-compressed chunks, a write-ahead log, three resolution
tiers (10 s for 6 h, 1 min for 7 d, 10 min for 90 d by default), a hard
memory budget.
- **A Prometheus-compatible API** with a practical PromQL subset, so
Grafana uses litek8s directly as a Prometheus data source, and dashboards
written for cAdvisor, node-exporter and kube-state-metrics partly work
unchanged. `/metrics` stays as it was for a real Prometheus, and
`/prometheus/federate` exports everything.
- **Logs**: search and follow across every pod and node (also of
containers that are gone) from the CRI log files the nodes already write;
no log store, no shipper.
- **Alerts**: built-in rules (crash loops, OOM kills, restart loops, nodes
not ready, node pressure, full disks and volumes, low memory), your own as
`AlertRule` objects in Prometheus rule syntax, notifications to a webhook
(Alertmanager's payload), ntfy or Telegram, silences and acks.
- **`kubectl top` and `litek8s top`** read the resource metrics API
(`metrics.k8s.io/v1beta1`) the API server serves from the same store: no
metrics-server ([below](#resource-metrics-api-kubectl-top)).
- **`litek8s ui`** shows it all in its Observe view (9).
Everything is on by default in `litek8s apiserver` and `litek8s dev`;
`--no-observability` turns it off.
## Quick start
```bash
litek8s logs -l app=web --since 1h --grep 'ERROR|panic' # every pod of app=web, all nodes
litek8s logs -n shop -f # follow a namespace
litek8s query 'topk(5, sum by (namespace, pod) (rate(container_cpu_usage_seconds_total[5m])))'
litek8s alerts # what is firing
litek8s alerts ack KubePodCrashLooping # mute until it resolves
litek8s top # nodes and pods, live
kubectl top pods -A --containers # no metrics-server needed
```
Grafana: add a **Prometheus** data source with URL
`http://:6443/prometheus` (or start the API server with
`--prometheus-listen 0.0.0.0:9090` and use `http://:9090`).
Checked with Grafana 12.2: the data source's health check ("Successfully
queried the Prometheus API"), range queries (GET or POST), the metric
browser and label values. Dashboards built on the raw cAdvisor,
kube-state-metrics and node-exporter series (the `job`, `image!=""`,
`container!=""` selectors they use match) work; the kubernetes-mixin
dashboards kube-prometheus-stack ships read its recording rules
(`node_namespace_pod_container:…`), which litek8s does not have.
The CLI finds the API server like `litek8s ui`: `--context`, else
`$KUBECONFIG`'s current context, else the local dev cluster.
## Metrics
### What is collected
Every series carries `job` and `instance` labels as Prometheus' usual
scrape jobs would, so dashboards' selectors match.
| job | where | series (labels) |
|---|---|---|
| `kubelet`, `metrics_path="/metrics/cadvisor"` | each node agent, every container (crun: its cgroup; Docker runtime: CPU and working set only) | `container_cpu_usage_seconds_total`, `container_cpu_user_seconds_total`, `container_cpu_system_seconds_total`, `container_cpu_cfs_periods_total`, `container_cpu_cfs_throttled_periods_total`, `container_cpu_cfs_throttled_seconds_total`, `container_memory_working_set_bytes`, `container_memory_usage_bytes`, `container_memory_rss` (anonymous), `container_memory_cache` (page cache), `container_spec_memory_limit_bytes`, `container_oom_events_total`, `container_fs_reads_bytes_total`, `container_fs_writes_bytes_total` (`namespace`, `pod`, `container`, `image`, `node`); per pod: `container_network_{receive,transmit}_{bytes,packets}_total`, `..._packets_dropped_total` (`interface="eth0"`) |
| `kubelet`, `metrics_path="/metrics"` | each node agent, every mounted persistent volume claim | `kubelet_volume_stats_used_bytes`, `_capacity_bytes`, `_available_bytes` (`namespace`, `persistentvolumeclaim`) |
| `node-exporter` | each node agent | `node_cpu_seconds_total{mode}`, `node_memory_{MemTotal,MemAvailable,MemFree,Cached,Buffers,SwapTotal,SwapFree}_bytes`, `node_load{1,5,15}`, `node_filesystem_{size,free,avail}_bytes` (the node's data directory), `node_network_{receive,transmit}_{bytes,packets,errs}_total{device}`, `node_disk_{read,written}_bytes_total`, `node_disk_io_time_seconds_total`, `machine_cpu_cores`, `up` |
| `litek8s` | each node agent | where the node's resources go, as `litek8s ui`'s Home explains them: `litek8s_node_memory_bytes{owner="litek8s|pods|cache"}`, `litek8s_node_cpu_seconds_total{owner="litek8s|pods"}`, `litek8s_node_disk_bytes{category}` (every 15 min) |
| `kube-state-metrics` | the API server, from its object store | `kube_pod_info`, `kube_pod_status_phase`, `kube_pod_status_ready`, `kube_pod_status_scheduled`, `kube_pod_container_info`, `kube_pod_container_status_{restarts_total,ready,running,waiting,terminated}`, `..._waiting_reason`, `..._terminated_reason`, `..._last_terminated_reason`, `..._last_terminated_exitcode`, `kube_pod_container_resource_{requests,limits}`, `kube_node_info`, `kube_node_status_condition`, `kube_node_status_{capacity,allocatable}`, `kube_node_spec_unschedulable`, `kube_deployment_*`, `kube_statefulset_*`, `kube_daemonset_*`, `kube_job_status_*`, `kube_job_failed`, `kube_cronjob_*`, `kube_persistentvolumeclaim_*`, `kube_persistentvolume_*`, `kube_namespace_status_phase` |
| `apiserver` | the API server | its `/metrics`: `apiserver_request_total`, `apiserver_longrunning_requests` (kube-apiserver's names and labels), `litek8s_apiserver_request_duration_seconds` (histogram), `litek8s_store_*`, …; `process_resident_memory_bytes`, `process_cpu_seconds_total`, `up` |
| `litek8s` | the API server | the store's own health: `litek8s_tsdb_{series,head_bytes,index_bytes,disk_bytes,wal_bytes,blocks}`, `..._series_{created,refused,removed}_total`, `..._samples_{appended,out_of_order}_total` |
| – | the alert evaluator | `ALERTS{alertname, alertstate="pending|firing", …}` as Prometheus writes it |
Sampling is aligned to the interval (10 s by default: the first tier's
resolution), so timestamps are regular. A series that stops (a deleted
pod's, a node that disconnects) gets a staleness marker, as in Prometheus:
it leaves instant queries at once instead of lingering for 5 minutes.
### The store
`src/obs/tsdb/`, data under `/obs/`:
- **Series** are interned label sets with an inverted index (label pair →
series) in memory; their definitions are appended to `series.log`.
- **Samples** go to `wal.log` first (fsynced every interval), then to the
series' in-memory **head chunk**: Gorilla compression, delta-of-delta
timestamps (a regular sample costs one bit) and XOR-ed values (an
unchanged value one bit). Every 20 minutes (or at 120 samples) heads are
cut into the current window's **block** file and the WAL is truncated.
- **Blocks** are one file per tier and time window (2 h for the raw tier).
An open block is append-only with a CRC per chunk; a torn write at its end
(a crash mid-write) is cut off when it is opened again. When the window
is over it is **sealed**: rewritten sorted by series with an index at the
end, then renamed into place (atomically).
- **Tiers and downsampling.** Sealing a window also downsamples it into the
next tier: counters (`*_total`, `*_count`, `*_sum`, `*_bucket`, or typed
counter) keep the last value of each bucket, gauges the average; buckets
are stamped at their start. Each tier deletes blocks older than its
retention once they have been downsampled. A query reads the coarsest tier
whose resolution is at most its step (and at most a third of its shortest
`rate` window) among those whose retention covers its start, and fills the
part the coarser tier has not received yet from the finer one, downsampled
on the fly. Instant queries read the finest tier that covers them.
- **Crash safety.** Anything acknowledged is in the WAL; replay skips what
already reached a block; a seal or a downsample interrupted by a crash
leaves either the old or the new file, and duplicate samples are dropped
when read. Tests: `survives_restart_through_wal_and_blocks`,
`torn_wal_tail_is_ignored`, `open_append_reopen_seal_query`.
- **Memory budget.** Only the label index and one head chunk per series are
in memory; blocks are read per query. `--obs-memory` (24 MiB; 8 MiB in a
`litek8s dev` cluster) sets the series limit (two thirds of the budget at
~900 bytes a series: 18 000 series by default); new series beyond it are
refused and counted (`litek8s_tsdb_series_refused_total`, and the
built-in alert `Litek8sMetricsSeriesRefused`). A query loads at most 1
million samples (16 MB while it runs). Series whose data has left every tier are dropped from
the index (and `series.log` is compacted).
Flags of `litek8s apiserver`:
| flag | default | |
|---|---|---|
| `--no-observability` | | no metrics, log search or alerts |
| `--obs-retention` | `10s:6h,1m:7d,10m:90d` | tiers, `:`; the first resolution is the sampling interval; each resolution a multiple of the previous one |
| `--obs-memory` | `24Mi` | memory for series and heads |
| `--obs-log-retention` | `7d` | how long each node keeps logs of removed containers |
| `--obs-log-max-size` | `512Mi` | the most each node keeps of them |
| `--no-builtin-alerts` | | only your own rules |
| `--prometheus-listen` | | also serve the Prometheus API at the root of this address |
### Querying: the PromQL subset
Supported:
- selectors with `=`, `!=`, `=~`, `!~` (regexes anchored, as in
Prometheus), `{__name__=~"..."}`, range selectors `[5m]`, `offset`;
- `rate`, `irate`, `increase` (extrapolated like Prometheus, counter resets
handled), `delta`, `idelta`, `changes`, `resets`, `deriv`,
`predict_linear`, `{avg,sum,min,max,count,last,stddev,stdvar,quantile,present,absent}_over_time`;
- `sum`, `avg`, `min`, `max`, `count`, `group`, `stddev`, `stdvar`, `topk`,
`bottomk`, `quantile`, `count_values`, with `by (…)` / `without (…)`
before or after;
- `+ - * / % ^ atan2`, comparisons with and without `bool`, `and`, `or`,
`unless`, vector matching with `on`/`ignoring` and
`group_left`/`group_right` (with extra labels), unary minus;
- `histogram_quantile`, `label_replace`, `label_join`, `abs`, `ceil`,
`floor`, `round`, `exp`, `ln`, `log2`, `log10`, `sqrt`, `sgn`, `clamp`,
`clamp_min`, `clamp_max`, `time`, `timestamp` (the evaluation time),
`vector`, `scalar`, `absent`, `sort`, `sort_desc`, `day_of_week`,
`day_of_month`, `days_in_month`, `hour`, `minute`, `month`, `year`.
Not supported: subqueries (`x[1h:5m]`), the `@` modifier, native
histograms, recording rules, exemplars (always empty).
### HTTP API
Under `/prometheus` on the API port (and at the root of
`--prometheus-listen`), GET or POST (form-encoded), Prometheus' response
format:
| path | |
|---|---|
| `/api/v1/query` | `query`, `time` |
| `/api/v1/query_range` | `query`, `start`, `end`, `step` (11 000 points at most) |
| `/api/v1/series` | `match[]`, `start`, `end` |
| `/api/v1/labels`, `/api/v1/label//values` | `match[]`, `start`, `end` |
| `/api/v1/metadata` | types and help of the API server's own metrics |
| `/api/v1/rules`, `/api/v1/alerts` | the alert rules and active alerts |
| `/api/v1/status/buildinfo` | reports Prometheus 2.53 (Grafana picks its features by it) |
| `/federate` | `match[]`: the latest sample of every matching series, text format |
### Resource metrics API (`kubectl top`)
The API server serves `metrics.k8s.io/v1beta1`, what metrics-server
serves, from the store: `NodeMetrics` and `PodMetrics`, get and list,
label and field selectors, per namespace or across all, and a table for
`kubectl get pods.metrics.k8s.io`. So `kubectl top nodes`, `kubectl top
pods -A [--containers]` and other metrics-server clients work with
nothing installed (litek8s has no HorizontalPodAutoscaler controller yet). The numbers are metrics-server's:
| | CPU | memory |
|---|---|---|
| container | rate of `container_cpu_usage_seconds_total` between its last two samples (the `window`, 10 s by default) | last `container_memory_working_set_bytes` |
| node | rate of `node_cpu_usage_seconds_total` | last `node_memory_working_set_bytes` |
The two node series carry the kubelet's `/metrics/resource` names
(`job="kubelet"`, `metrics_path="/metrics/resource"`). An agent samples
them from the node's own cgroup when it has one (each node of a `litek8s
dev` cluster is a cell with its own cgroup; an agent in a container), else
from the machine: busy modes of `/proc/stat` and `MemTotal - MemAvailable`.
Reading `/proc` in the macOS dev VM would count the memory balloon as used
and give every node the whole VM. Nodes of agents from before these series
fall back to node-exporter's (`node_cpu_seconds_total` without `idle`,
`iowait`, `steal`; `MemTotal - MemAvailable`), and a node with no node
series at all (the Docker runtime on macOS has no `/proc`) reports the sum
of its containers.
A pod shows up one sample interval after it started (two CPU samples are
needed), and a series older than three intervals is not current (a node
that stopped reporting drops out, `kubectl top nodes` shows ``).
The group is built in (litek8s registers no APIService objects for its
built-in groups), in discovery (also aggregated v2) while observability
runs. To run metrics-server instead, install it as usual (its Helm chart
or `components.yaml`): its APIService `v1beta1.metrics.k8s.io` points at
its Service and takes the group over, every request under
`/apis/metrics.k8s.io/v1beta1` goes to metrics-server from then on, and
uninstalling it gives the group back to the store. (A Local APIService
object, as kube-apiserver keeps for its own groups, would make `helm
install metrics-server` fail: Helm does not adopt objects it did not
create.)
`litek8s top` shows the current kube context's (or `--context`'s) nodes (usage against allocatable) and pods,
busiest first, refreshed every `--interval-secs` on a terminal and printed
once when piped or with `--once`; `-n`, `-l`, `--containers` and
`--sort-by cpu|memory|name` narrow and order it. It reads only the
metrics API, so it works against any cluster with metrics-server too.
## Logs
There is no log store: a search goes to every node, and each node agent
reads the CRI log files it already writes (`