Skip to content
KIKAPU/ guide
Open console

Developer guide

The Kikapu Developer Guide

For contributors

A task-oriented guide to building and contributing to Kikapu. For the conceptual tour of how the system works, read the Kitabu.

Start building →Read the source

The Kikapu Developer Guide

A task-oriented guide for contributing to Kikapu. It answers "how do I…" — set up, build, test, add a change, and ship it — and points at the invariants you must not break. For the conceptual tour of why the system is shaped the way it is, read the Kitabu; this guide is the how.

Kikapu is a Go control plane that schedules tenant workloads onto opted-in Linux nodes, runs them with rootless Podman under per-tenant Linux users, and reaches them through an agent-initiated tunnel. Everything below assumes the public source tree at github.com/isnakolah/kikapu.

Terms before tools

  • Repository (repo): checked-out source directory containing this guide, go.mod, Makefile, cmd/, internal/, site/, and docs/. During delivery migration it also contains immutable legacy plan/ sources; those are not current work state.
  • Package: one Go source directory. go test ./internal/server/... tests that package tree.
  • Control plane: API, scheduler, PostgreSQL state, tunnel terminus, and routing integration.
  • Node runtime: unprivileged agent, fixed-verb supervisor boundary, tenant users, rootless Podman, and outbound tunnel on opted-in host.
  • Gate: repeatable command checking a behavior set. Green local gate proves only environment where it ran; it does not prove provider custody, production, or another host OS.

1. Getting started

You need Go (the pinned toolchain in go.mod), and — for the runtime and end-to-end suites — a Linux host with rootless Podman, subordinate ID ranges, systemd, and loginctl. Unit and PostgreSQL tests run fine in the devcontainer.

  • make tools — check the devcontainer tools (sqlc, migrate, golangci-lint, caddy) are present.
  • make deps — download Go module dependencies.

First checkout loop:

git status --short --branch
go version
make tools
make deps
make build
make test

Read each failure before continuing. make tools identifies missing repository-pinned tools. make build proves binaries compile on current host. make test proves portable unit behavior; neither command proves Linux host integration or production readiness.

The devcontainer under .devcontainer/ provisions the supported toolchain; the dev/ directory holds VM provisioners (Vagrant, Lima, Multipass, cloud-init) for the Linux-host suites.

2. Build and run locally

  • make build — build kikapu, kikapu-agent, kikapu-server, kikapu-supervisor, and kikapu-observer into bin/.
  • make control-init / make control-start / make control-stop — verify a TUF-selected immutable image and operate the local rootless-Podman control-only container. Set KIKAPU_ROOT_DOMAIN; control-init is explicitly development-only and resolves the image from signed metadata. The container publishes only host loopback.
  • make agent — run kikapu-agent locally against deploy/local-agent.yaml.

The CLI is manifest-first: you describe a deployment in a kikapu.yaml and run kikapu deploy --file kikapu.yaml. The Kitabu walks a first controlled deployment end to end.

3. Repository map and the layering discipline

Command binaries live under cmd/ (kikapu, kikapu-agent, kikapu-server, kikapu-supervisor, kikapu-observer, plus the release helpers kikapu-tuf, kikapu-update-manifest, and the kikapu-slo gate). Everything importable is under internal/:

  • internal/server — HTTP API, layered handlers → services → repositories. Handlers parse and render; services hold business rules; repositories own SQL.
  • internal/db — migrations, hand-written queries, sqlc-generated sqlc, and the repo/pg PostgreSQL implementations (the scheduler lives here).
  • internal/agent — the node agent: reconciler, tenant manager, supervisor split, signed desired-state cache, quarantine, heartbeat, update checker.
  • internal/runtime — the rootless-Podman runtime and the security flag set.
  • internal/manifest — shared manifest parsing used by both the CLI and server.
  • internal/domain — plain domain types and the errors.go sentinel vocabulary.
  • internal/trust, internal/economy, internal/coordinator, internal/mesh — trust, CU economy, federation, and mesh surfaces.

The rule that keeps this clean: business logic flows handlers → services → repos, domain types stay plain, and errors are wrapped sentinels rendered through one RFC 7807 problem writer (internal/server/handlers/error.go). When in doubt, match the surrounding code rather than adding an abstraction.

4. Common tasks

Add or change a database migration. Add paired NNNNNN_name.up.sql / .down.sql files under internal/db/migrations, write the query in internal/db/queries, then regenerate: make sqlc. make sqlc-check fails the build if the committed generated code drifts from the queries. Apply and roll back locally with make migrate-up / make migrate-down. Update the affected repo/pg repository and its tests in the same change.

Add an API endpoint. Add the route and a handler in internal/server, put the rules in a service, and reach data only through a repository interface. Return wrapped domain sentinels; let the problem writer map them to status codes. Never read tenant, node, or auth scope from a request body — those come from the authenticated API key.

Add a CLI command. Extend internal/cli; keep it manifest-first and scriptable, and route all server calls through internal/client.

Add a configuration value. Parse and validate it in internal/platform/config so a bad value fails startup, not runtime. Dangerous features default off and are enabled by a single environment variable with a documented, lossless rollback.

5. Testing

Kikapu prefers behavioral, property-named tests over coverage numbers — the test list is the security-property inventory. The gate ladder, by make target:

  • make test — unit tests, race detector, ./internal/....
  • make test-integration — repositories and server against a real PostgreSQL.
  • make test-mesh — the federation/mesh harness.
  • make test-reset-drills — the trust-plane reset levels.
  • make test-host — Linux + root: real rootless-Podman tenant lifecycle and isolation (build tag hostintegration).
  • make e2e — Linux VM: the full acceptance journey (build tag e2e).
  • make test-chaos / make test-load — the deterministic economy chaos simulation and the load/receipt-replay harness.
  • make verify — docs, lint, unit, and PostgreSQL integration gate.
  • make verify-fabric — broad portable fabric readiness gate.
  • make verify-fabric-linux — fabric gate plus supported Linux host and E2E.

Run a single package with go test ./internal/<pkg>/.... When you touch a security boundary, add a test that asserts the boundary and fails without your fix. Do not use t.Skip to hide unfinished work, and do not retry a flaky test to green — fix the flake.

6. The maintenance loop

Every tracked change follows same loop:

  1. Run make project-sync, then make project-context ISSUE=NNN. GitHub Issues, native relationships, and Kikapu Delivery are current delivery authority.
  2. Claim one Ready, unblocked leaf Issue with make project-claim ISSUE=NNN BRANCH=agent/NNN-short-slug; wait for Project Writer receipt before creating worktree or branch.
  3. Make scoped change behind narrow interfaces. Do not edit Project fields, relationships, evidence, implementation logs, or generated Wiki pages directly.
  4. Update reader-facing docs when behavior changes. Record exact acceptance and proof through make project-complete; Project Writer alone records canonical evidence and completion state.
  5. Verify correct make gate, resync before commit, then use #NNN type(scope): summary with exact Refs: #N footer.

Copy-paste loop for a small Go change:

git status --short --branch
rg "symbol or behavior" internal
# Edit narrow owning package and its tests.
go test ./internal/package-path/...
make lint
make test
git diff --check
git diff --stat

Replace package-path with actual directory. Add site, docs, integration, host, or aggregate gates when touched behavior needs them. Small package test gives early feedback, not final evidence.

Agent routing lives in AGENTS.md; concise human workflow lives in CONTRIBUTING-AI.md.

7. How the documentation keeps itself current

Two mechanisms carry the parts of the maintenance loop that are easy to forget.

The CLI reference is generated, not written. tools/gen-cli-docs walks the cobra command tree and renders the full command and flag inventory. Regenerate with make docs-cli after adding or changing a command; make docs-cli-check runs in CI and fails the build if the committed file has drifted. Treat it like the sqlc gate — the generated file is an output, never an input.

Ownership is machine-readable. docs/ownership.json maps source paths to Book, operations, Developer guide, generated CLI reference, or AUP. make docs-check validates canonical set, Markdown links/anchors, terminology, ownership impact declarations, generated CLI freshness, and instruction budgets. docs-sync workflow reads same ownership map and checks documentation ownership on the PR. It never edits or pushes a contributor branch; those changes belong to the leased Issue and normal PR review. It does not run on pull requests from forks.

make docs-check treats legacy plan/ files as archived migration input, not current documentation or a documentation-impact bypass. It checks current source ownership and current reader-facing documents; Project Writer records the Issue contract's documentation decision.

Neither mechanism replaces delivery governance. During GitHub-first migration, plan/ is immutable source material only. GitHub Issues, native relationships, Project fields, and writer evidence hold delivery state; documentation explains behavior and links those records.

8. GitHub Project Writer activation

Kikapu Project Writer GitHub App has only repository-scoped Issue write plus read-only Checks, Contents, and Pull requests permissions. Its installation token can create trusted evidence comments, but GitHub does not expose private user-owned ProjectV2 records to that token. Keep KIKAPU_PROJECT_WRITER_ENABLED unset until secret KIKAPU_PROJECT_OWNER_PROJECT_TOKEN holds an owner credential that can read and mutate Kikapu Delivery. Writer workflow uses it only for ProjectV2 reads and fields, while its short-lived KIKAPU_PROJECT_WRITER_TOKEN App token owns Issue comments and native relationships. Enable only after split-path integration test passes. Do not replace this gate with a broadly installed App or direct agent Project mutations.

Project cache sync reads Project fields, native graph, and ordinary evidence in bounded GraphQL pages. Only Issues with more than one GraphQL comment page need REST comment pagination; this avoids turning normal agent startup into a repository-wide request burst.

9. Invariants you must never break

These are non-negotiable. A change that relaxes any of them is wrong even if it makes a test pass:

  • Workload hardening is fixed. Every container runs with --cap-drop ALL, no-new-privileges, a read-only root, a size-capped nosuid,nodev,noexec tmpfs, the fixed uid/gid maps, and loopback-only ports. The forbidden-flag list (--privileged, host namespaces, device mounts, runtime sockets, extra capabilities, mounts outside the tenant root) is enforced as a test.
  • Placement is the scheduler's. A deployment request can never name a node, a node public address, or a host port; nodes are reached only over the authenticated tunnel.
  • Secrets never get logged. Raw API keys and environment values are redacted before any diagnostic; environment values are encrypted at rest when enabled.
  • No shell command strings. Podman and mount are invoked through structured process arguments, never a composed shell line.
  • Identity comes from the key. Tenant, node, and auth scope are derived from the API key, never from a request body.

One honest limitation, stated plainly: Kikapu isolates tenants from each other and the host from tenants — it does not protect a tenant from a malicious node owner with host root, who can read everything their node runs.

9. Navigating the code, and shipping a release

Community site deployment

Changes below site/ on main automatically run the Cloudflare Pages workflow. The workflow builds and checks only the public and console bundles before publishing site/dist/ to https://kikapu.nomonlab.com/. Changes outside site/ do not publish the community site.

For an intentional manual republish, run make site-deploy CONFIRM_PRODUCTION_DEPLOY=yes after make site-check. Treat a successful local build, remote workflow, and external production smoke as separate evidence.

For questions about architecture or file relationships, the repository ships a knowledge graph: graphify query "<question>" returns a scoped subgraph, graphify path "<A>" "<B>" traces relationships, and graphify explain "<concept>" focuses one concept. After changing code, run graphify update . to keep it current.

When changing .github/workflows/, run make actions-check; CI uses the same pinned actionlint version. Host acceptance has a twelve-minute Go deadline inside a fifteen-minute Actions envelope so cleanup and diagnostics retain a bounded window.

Releases are cut from a tag: scripts/build-release.sh cross-compiles the archives and generates the signed update manifest, kikapu-tuf signs the TUF metadata, and the release workflow publishes assets, the container image, and the distribution adapters. The operational detail lives in the operations guide; the update-awareness design is in the Kitabu.

Questions, patches, and design discussion are welcome on GitHub and the community Discord.