ETCD members not getting removed after cluster downscaling

[Archisman-Mridha]

/kind bug

1. What kops version are you running? The command kops version, will display
this information.

v1.28.5

2. What Kubernetes version are you running? kubectl version will print the
version if a cluster is running or provide the Kubernetes version specified as
a kops flag.

v1.28.9

3. What cloud provider are you using?

AWS

4. What commands did you run? What is the simplest way to reproduce this issue?

I wanted to downscale the Control Plane from 5 to 3 master nodes. I first removed the 2 extra master nodes using these commands :

kops delete instancegroup master-3 --yes
kops delete instancegroup master-4 --yes

Then I removed the corresponding ETCD members from the cluster spec using kops edit cluster.

5. What happened after the commands executed?

master-3 and master-4 got deleted. But ETCD pods in the remaining 3 master nodes, were throwing error logs constantly : that ETCD members etcd-3 and etcd-4 aren't reachable. The 2 ETCD members belonging to the deleted master nodes, were not removed from the ETCD peers list.

When I made some changes to the master-0 node and rolled it out, it was refusing to join the cluster : since the ETCD pods in it was crashing with some error similar to this : Raft leader election failed : members etcd-3 and etcd-4 are unreachable.

6. What did you expect to happen?

ETCD members of the deleted master nodes, must have been cleaned up from the peers list of the etcd-manager-main and etcd-manager-events ETCD clusters.

7. Please provide your cluster manifest. Execute
kops get --name my.example.com -o yaml to display your cluster manifest.
You may want to remove your cluster name and other sensitive information.

8. Please run the commands with most verbose logging by adding the -v 10 flag.
Paste the logs into this report, or in a gist and provide the gist link here.

9. Anything else do we need to know?

If I first remove the corresponding ETCD members from the cluster spec by doing kops edit cluster and then delete the master nodes, everything works smoothly. The ETCD members corresponding to the deleted master nodes get cleaned up from the peers list. These are the commands I used :

kops replace -f ./kops/cluster.yaml -f ./kops/master-ig.yaml -f ./kops/nodes-ig.yaml --force
kops delete instancegroup master-3
kops delete instancegroup master-4
kops update cluster --yes
kops rolling-update cluster --instance-group=master-0 --yes