Kubernetes service that can reap namespaces that have not run pods recently.
This service is intended for namespaces such as user namespaces that might run job like pods but where the namespace doesn't necessarily need to always exist like if the user account was disabled.
Currently the namespaces to reap can be based on namespace regular expression and/or namespace labels . A namespace is reaped if the age of the namespace is past a certain threshold and no recent pods have run in that namespace. A Prometheus instance running kube-state-metrics is required to check for recently run pods. See Changing what is reaped for details on how to configure reaping behavior.
Metrics about the count of reaped namespaces, duration of last reaping, and error counts can be queried using Prometheus /metrics endpoint exposed as a Service on port 8080.
Currently this code is built and tested against Kubernetes 1.29.x.
The Kubernetes APIs used by this project do not tend to change between Kubernetes releases so it is likely this code will work on all 1.x releases of Kubernetes.
Only Helm 3 is supported.
helm repo add k8-namespace-reaper https://osc.github.io/k8-namespace-reaper
helm install k8-namespace-reaper k8-namespace-reaper/k8-namespace-reaper \
-n k8-namespace-reaper --create-namespace \
--prometheus-address=http://prometheus:9090
For Open OnDemand the following adjustments can be made to get a working install using Helm:
helm install k8-namespace-reaper k8-namespace-reaper/k8-namespace-reaper \
-n k8-namespace-reaper --create-namespace \
--prometheus-address=http://prometheus:9090 \
--set config.namespaceLabels='app.kubernetes.io/name=open-ondemand' \
--set config.namespaceLastUsedAnnotation='openondemand.org/last-hook-execution'
First install the necessary Namespace and RBAC resources:
kubectl apply -f https://github.com/OSC/k8-namespace-reaper/releases/latest/download/namespace-rbac.yaml
For Open OnDemand a deployment can be installed using Open OnDemand specific deployment:
kubectl apply -f https://github.com/OSC/k8-namespace-reaper/releases/latest/download/ondemand-deployment.yaml
A more generic deployment:
kubectl apply -f https://github.com/OSC/k8-namespace-reaper/releases/latest/download/deployment.yaml
NOTE Both the OnDemand and generic deployments require modifications to set Prometheus address. The generic deployment also needs to be told which namespaces to reap.
If you wish to scope the namespaces searched for reaping change either --namespace-labels flag (comma separated) to limit namespaces searched by label, or a namespace regular expression with --namespace-regexp. The namespace regular expression is also used to limit the scope of the Prometheus query, so that regular expression must also be valid for PromQL.
The minimum age of a namespace to reap is set with --reap-after. This flag also sets how far back to look for active namespaces by looking at pod metrics. If --reap-after is default of 168h then a namespace older than 7 days with no pods active in last 7 days will be deleted.
Use --namespace-last-used-annotation to define a namespace annotation that marks when the namespace was last used.
A namespace will not be reaped if that last usage is more recent than the duration defined with --last-used-threshold.
The k8-namespace-reaper is intended to be deployed inside a Kubernetes cluster. It can also be run outside the cluster via cron.
The following flags and environment variables can modify the behavior of the k8-namespace-reaper:
| Flag | Environment Variable | Description |
|---|---|---|
| --namespace-labels | NAMESPACE_LABELS | Sets namespaces labels for which namespaces to consider for reaping, required if --namespace-regexp is not set. |
| --namespace-regexp | NAMESPACE_REGEXP | Sets namespace regular expression for which namespaces to consider for reaping, required if --namespace-labels is not set. |
| --namespace-last-used-annotation | NAMESPACE_LAST_USED_ANNOTATION | Annotation of when namespace was last used, must be Unix timestamp |
| --prometheus-address | PROMETHEUS_ADDRESS | Prometheus address, eg: http://prometheus:9090, this is required |
| --prometheus-timeout=30s | PROMETHEUS_TIMEOUT=30s | Prometheus query timeout Duration |
| --reap-after=168h | REAP_AFTER=168h | Duration minimum age of namespaces to reap as well as how far back to look for active pods |
| --last-used-threshold=4h | LAST_USED_THRESHOLD=4h | How long after last used can a namespace be reaped (must be a Duration) |
| --interval=6h | INTERVAL=6h | Duration between each reaping execution when run in loop |
| --listen-address=:8080 | LISTEN_ADDRESS=:8080 | Address to listen for HTTP requests |
| --no-process-metrics | PROCESS_METRICS=false | Disable metrics about the running processes such as CPU, memory and Go stats |
| --run-once | RUN_ONCE=true | Set to only execute reap code once and exit, ie used when run via cron |
| --kubeconfig | KUBECONFIG | The path to Kubernetes config, required when run outside Kubernetes |
| --log-level=info | LOG_LEVEL=info | The logging level One of: [debug, info, warn, error] |
| --log-format=logfmt | LOG_FORMAT=logfmt | The logging format, either logfmt or json |