Changing node's role from worker to etcd,controller doesn't remove nginx-proxy

[matthewmelvin]

If I change the roles of a worker node to be a controller node, everything says it has run fine, but nginx-proxy container is not removed from the node, so the kube-apiserver container never starts properly, erroring with...

Error: failed to create listener: failed to listen on 0.0.0.0:6443: listen tcp 0.0.0.0:6443: bind: address already in use

As a worker:

root@testvm36:~# docker ps | grep -v k8s_
CONTAINER ID        IMAGE                                                                                                  COMMAND                  CREATED             STATUS              PORTS               NAMES
31d5b94b9a64        rancher/hyperkube:v1.21.7-rancher1                                                                     "/opt/rke-tools/en..."   8 minutes ago       Up 8 minutes                            kube-proxy
0fedf6de0347        rancher/hyperkube:v1.21.7-rancher1                                                                     "/opt/rke-tools/en..."   9 minutes ago       Up 9 minutes                            kubelet
68c065f75167        rancher/rke-tools:v0.1.78                                                                              "nginx-proxy CP_HO..."   9 minutes ago       Up 9 minutes                            nginx-proxy
root@testvm36:~#

As a controller:

root@testvm36:~# docker ps | grep -v k8s_
CONTAINER ID        IMAGE                                                                                                  COMMAND                  CREATED             STATUS                          PORTS               NAMES
bb93b1f41de8        rancher/hyperkube:v1.21.7-rancher1                                                                     "/opt/rke-tools/en..."   3 minutes ago       Up 3 minutes                                        kubelet
00e748bdbf52        rancher/hyperkube:v1.21.7-rancher1                                                                     "/opt/rke-tools/en..."   3 minutes ago       Up 3 minutes                                        kube-scheduler
d1cabfed7d1b        rancher/hyperkube:v1.21.7-rancher1                                                                     "/opt/rke-tools/en..."   4 minutes ago       Up 3 minutes                                        kube-controller-manager
3f32348549fa        rancher/hyperkube:v1.21.7-rancher1                                                                     "/opt/rke-tools/en..."   4 minutes ago       Restarting (1) 35 seconds ago                       kube-apiserver
ab85ff6a2fc0        rancher/rke-tools:v0.1.87                                                                              "/docker-entrypoin..."   5 minutes ago       Up 5 minutes                                        etcd-rolling-snapshots
cf1022be859f        rancher/mirrored-coreos-etcd:v3.4.16-rancher1                                                          "/usr/local/bin/et..."   6 minutes ago       Up 6 minutes                                        etcd
31d5b94b9a64        rancher/hyperkube:v1.21.7-rancher1                                                                     "/opt/rke-tools/en..."   30 minutes ago      Up 30 minutes                                       kube-proxy
68c065f75167        rancher/rke-tools:v0.1.78                                                                              "nginx-proxy CP_HO..."   31 minutes ago      Up 31 minutes                                       nginx-proxy
root@testvm36:~# 

And kube-apiserver can't start listening because nginx-proxy still is.

root@testvm36:~# docker logs kube-apiserver --tail=3
I0927 06:40:47.839062       1 server.go:629] external host was not specified, using 172.16.37.36
I0927 06:40:47.842250       1 server.go:181] Version: v1.21.7
Error: failed to create listener: failed to listen on 0.0.0.0:6443: listen tcp 0.0.0.0:6443: bind: address already in use
root@testvm36:~# lsof -i -n -P | grep 6443.*LISTEN
nginx      4339            root    4u  IPv4 23636971      0t0  TCP *:6443 (LISTEN)
nginx      4344 systemd-network    4u  IPv4 23636971      0t0  TCP *:6443 (LISTEN)
nginx      4345 systemd-network    4u  IPv4 23636971      0t0  TCP *:6443 (LISTEN)
root@testvm36:~# ps auxwwwf | grep -B 1 ngin[x]
root      4307  0.0  0.0 216196  5036 ?        Sl   23:10   0:00      \_ docker-containerd-shim 68c065f75167182834d5ac71000b4a5a4b4f71c51428d73836ca3ab7b081502f /var/run/docker/libcontainerd/68c065f75167182834d5ac71000b4a5a4b4f71c51428d73836ca3ab7b081502f docker-runc
root      4324  0.0  0.0   2312  1736 ?        Ss   23:10   0:00      |   \_ /bin/sh /usr/bin/nginx-proxy CP_HOSTS=172.16.37.35,172.16.37.37
root      4339  0.0  0.0   5856  3852 ?        S    23:10   0:00      |       \_ nginx: master process nginx -g daemon off;
systemd+  4344  0.0  0.0   6780  2084 ?        S    23:10   0:00      |           \_ nginx: worker process
systemd+  4345  0.0  0.0   6668  2168 ?        S    23:10   0:00      |           \_ nginx: worker process
root@testvm36:~# 

Using provider as...

terraform {
  required_providers {
[...]
    rke = {
      source = "rancher/rke"
      version = "= 1.3.3"
    }
[...]
}

provider "rke" {
  debug = true
  log_file = "rke.txt"
}

Resulting log is attached. Am a bit confused because it says it waited for kube-apiserver to be healthy, but it isn't?

time="2022-09-27T06:37:18Z" level=info msg="[controlplane] Successfully started [kube-apiserver] container on host [172.16.37.36]"
time="2022-09-27T06:37:18Z" level=info msg="[healthcheck] Start Healthcheck on service [kube-apiserver] on host [172.16.37.36]"
time="2022-09-27T06:37:20Z" level=info msg="[healthcheck] service [kube-apiserver] on host [172.16.37.36] is healthy"

rke.txt