Skip to content

0021. Observability, database and Kubernetes operator tools

Status

Accepted

Context

Projects that use this image operate Kubernetes clusters with Flux (ADR 0020), and they also need to validate Prometheus rules and Alertmanager config, work with CloudNativePG databases, lint GitHub Actions workflows and prose, and back up data. The image lacked these CLIs: promtool, amtool, helm, kubectl-cnpg, cmctl, actionlint, restic, k9s, stern, kubeconform, vale, and the PostgreSQL client (psql, pg_restore). yq was also on the request, but mikefarah's yq is already installed (see below). Installing them at runtime does not work, because apk add needs root (sudo is scoped to four maintenance scripts) and the egress allowlist ships locked down.

Constraints specific to this image are the same as in ADR 0020: Alpine/musl (ADR 0001) on amd64 and arm64 (ADR 0012), and Renovate-pinned, checksum-verified binaries in the builder stage (ADR 0008).

Decision

Pinned upstream binaries (builder stage, one ARG + # renovate: comment each, sha256sum -c before install, both arches, all in the cli-tools Renovate group):

Tool Source Checksum source
promtool prometheus/prometheus tarball. Only promtool is extracted, not the prometheus server. sha256sums.txt
amtool prometheus/alertmanager tarball. Only amtool is extracted, not the alertmanager server. sha256sums.txt
helm https://get.helm.sh/helm-v<ver>-linux-<arch>.tar.gz; Renovate tracks the helm/helm release tag .sha256sum sidecar
kubectl-cnpg cloudnative-pg/cloudnative-pg tarball (Intel is x86_64 in the name) cnpg-<ver>-checksums.txt
cmctl cert-manager/cmctl bare binary checksums.txt
actionlint rhysd/actionlint tarball actionlint_<ver>_checksums.txt
restic restic/restic .bz2 single binary, unpacked with bunzip2 SHA256SUMS
k9s derailed/k9s tarball (k9s_Linux_<arch>, no version in the name) checksums.sha256
stern stern/stern tarball checksums.txt
kubeconform yannh/kubeconform tarball (no version in the name) CHECKSUMS
vale errata-ai/vale tarball (Intel is Linux_64-bit in the name) vale_<ver>_checksums.txt

kubectl-cnpg is installed as /usr/local/bin/kubectl-cnpg, so kubectl's plugin discovery makes kubectl cnpg ... work.

Alpine package (Layer A, pinned to the major in the package name):

  • postgresql18-client provides psql, pg_restore, pg_dump and the other client tools. They link dynamically against libpq, so apk is the natural route. The package name carries the major on purpose: a future Alpine default cannot swap the client under users, and moving to 19 is a deliberate edit.

Already present, no change: yq is in Layer A and Alpine's yq resolves to the yq-go package, which is mikefarah's implementation (v4.53.3 at the time of writing; the Python yq is a different package). A BATS test now asserts that, so a change in the provider would fail the build.

Rejected:

  • Everything from apk where Alpine has it. Alpine 3.24 carries actionlint, alertmanager, cmctl, helm, k9s, prometheus, restic, vale and yq-go, but several lag upstream by a lot (read from the aarch64 index, against upstream at the time of writing): helm 3.19.0 against 4.3.1, prometheus 3.5.3 against 3.15.1, cmctl 2.4.1 against 2.6.2, restic 0.18.1 against 0.19.1, vale 3.13.0 against 3.26.0. Alpine has no stern, kubeconform or kubectl-cnpg at all. The apk prometheus and alertmanager packages also bundle the servers. Mixing routes per tool would make the version story harder to explain than using the ADR 0020 route for every single-binary CLI.
  • Bumping yq to upstream (4.54.1). It is already installed and one patch behind; moving it to the builder stage would add a pin for no functional gain.
  • The prometheus and alertmanager servers. Only the CLIs were requested.

Consequences

Accepted:

  • The image gets larger. Measured on arm64, uncompressed: promtool 112 MB, cmctl 65 MB, amtool 28 MB, restic 28 MB, about 233 MB for those four alone. The other seven were not measured. The Layer A and Layer B changes invalidate the layers below them for one build.
  • Helm 4 is a major release. Alpine would have given Helm 3. Projects with Helm 3 flags or plugins should read the upstream release notes.
  • postgresql18-client updates with the Alpine base (18.x patch releases), not through a Renovate PR. The Alpine release for a new PostgreSQL major is a manual change.
  • 11 new cli-tools entries mean the group's weekly Renovate PR will touch more tools; each bump is a one-line ARG change.
  • No egress allowlist entries were added. The tools talk only to the clusters, registries, databases and backup targets a project configures.

Not verified when this was written:

  • get.helm.sh could not be reached from the authoring container (HTTP 502 through the proxy), so the helm checksum sidecar format (<digest> <file>) follows upstream's documented layout and is confirmed only by the CI build.
  • The other ten downloads were fetched for arm64 and each checksum was verified with sha256sum -c --ignore-missing, and the archive layout was listed. The amd64 asset names were checked only by their presence in the checksum files. None of the binaries were executed locally (the sandbox denied it), so the BATS version commands are the first run. kubectl-cnpg version and cmctl version --client are assumed to work without a cluster.
  • The apk versions above were read from the aarch64 index; the x86_64 index was not inspected.