Talos cluster credentials expired, can't access cluster

[bcspragu]

Hi there,

I can no longer connect to my Talos cluster:

> kubectl get nodes
E1006 19:24:27.025594 1695958 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
E1006 19:24:27.030988 1695958 memcache.go:265] "Unhandled Error" err="couldn't get current server API group list: the server has asked for the client to provide credentials"
[ ... repeated a few more times ... ]
error: You must be logged in to the server (the server has asked for the client to provide credentials)

Checking my ~/.kube/config, my users.user.client-certificate-data has expired. E.g. echo '<client certificate data>' | base64 -d | openssl x509 -text -noout contains:

        Validity
            Not Before: Oct  6 01:23:22 2023 GMT
            Not After : Oct  5 01:23:32 2024 GMT

So that explains that problem, I just don't know how to fix it. I tried to generate a new config with:

> talosctl kubeconfig -n ... -e ... --talosconfig ./talosconfig
error copying: rpc error: code = Unavailable desc = connection error: desc = "error reading server preface: remote error: tls: bad certificate"

Looking in my talosconfig, the server certificates look good until 2033, so I'm not sure what's actually going on here. Any ideas?

This is just a single-node cluster in my home, so I have physical access to the machine if need be.

[smira]

Kubernetes config can always be refreshed with talosctl kubeconfig, but you need Talos API access for that.

I guess both expired for you, you can check your talosconfig with talosctl --talosconfig=./talosconfig config info.

If your client-side talosconfig expired, the only option left is to recover it via secrets.yaml or machine configuration: https://www.talos.dev/v1.8/talos-guides/howto/cert-management/#generating-new-client-configuration

[bcspragu]

Thanks, that worked perfectly! For anyone else stumbling across this thread, here's what I did:

# Extract the CA cert + key from the control plane config
yq -r .machine.ca.crt controlplane.yaml | base64 -d > ca.crt
yq -r .machine.ca.key controlplane.yaml | base64 -d > ca.key

# Generate fresh credentials
talosctl gen key --name admin
talosctl gen csr --key admin.key --ip 127.0.0.1
talosctl gen crt --ca ca --csr admin.csr --name admin

# Update the `talosconfig` with the new values
# I generated them with:
# For ca: base64 -w0 ca.crt
# For crt: base64 -w0 admin.crt
# For key: base64 -w0 admin.key
vim talosconfig

# Finally, update your `~/.kube/config` with the new creds
talosctl kubeconfig -n ... -e ... --talosconfig ./talosconfig

Not sure if the docs should be updated or my version of yq is just weird, but the yq eval ... invocations didn't work for me:

> yq eval '.machine.ca.crt' controlplane.yaml | base64 -d > ca.crt
usage: yq [-h] [--yaml-output] [--yaml-roundtrip]
          [--yaml-output-grammar-version {1.1,1.2}] [--width WIDTH]
          [--indentless-lists] [--explicit-start] [--explicit-end]
          [--in-place] [--version]
          [jq_filter] [files ...]
yq: error: argument files: can't open '.machine.ca.crt': [Errno 2] No such file or directory: '.machine.ca.crt'

[daemonp]

Thank you for posting this @bcspragu and @smira for the answer

It's also worth mentioning that talosctl gen csr --key admin.key --ip 127.0.0.1 defaults to 24 hours and the --hours flag can be used to set for longer period.

This is the oneliner I'm using for updates:

yq -r .machine.ca.crt controlplane.yaml | base64 -d > ca.crt && \
yq -r .machine.ca.key controlplane.yaml | base64 -d > ca.key && \
talosctl gen key --name admin && \
talosctl gen csr --key admin.key --ip 127.0.0.1 && \
talosctl gen crt --ca ca.crt --csr admin.csr --name admin --days 8760 && \
yq eval '.contexts.home.ca = "'"$(base64 -w0 ca.crt)"'" | .contexts.home.crt = "'"$(base64 -w0 admin.crt)"'" | .contexts.home.key = "'"$(base64 -w0 admin.key)"'"' -i talosconfig

[bcspragu]

It's also worth mentioning that talosctl gen csr --key admin.key --ip 127.0.0.1 defaults to 24 hours and the --hours flag can be used to set for longer period.

I should really know more about how the PKI works in Talos/K8s, but it looks like even though the admin.crt is only valid for a day, the generated talosctl kubeconfig is valid for one year.

[dpoon]

The solution from @daemonp works, with a tweak. The command

talosctl gen crt --ca ca.crt --csr admin.csr --name admin --days 8760

… should be:

talosctl gen crt --ca ca --csr admin.csr --name admin --hours 8760

Also note that on a macOS client, base64 -w0 should instead be base64 -b0.