Fix race condition (also a regression of the PR 19139)

[ahrtr]

Fix #19172

Please review this PR commit by commit.

Two high level thoughts,

• There are multiple levels of goroutines. The grandparent ( StartEtcd ) creates multiple child goroutines ( client listeners, peer listeners and metrics listeners). The client listeners creates some grandson goroutines (see the first commit). Each one should only manage their immediate children.
• For sync.WaitGroup, we should always call wg.Add and wg.Wait in the same goroutine.

cc @serathius @fuweid @ivanvc @jmhbnz @joshuazh-x

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahrtr

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [ahrtr]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[codecov]

Codecov Report

Attention: Patch coverage is 77.27273% with 5 lines in your changes missing coverage. Please review.

Project coverage is 68.80%. Comparing base (ecd7cfc) to head (1dcdfc1).

Files with missing lines	Patch %	Lines
server/embed/serve.go	81.25%	1 Missing and 2 partials ⚠️
server/embed/etcd.go	66.66%	2 Missing ⚠️

Additional details and impacted files

Files with missing lines	Coverage Δ
server/embed/etcd.go	76.19% <66.66%> (+0.33%)	⬆️
server/embed/serve.go	59.38% <81.25%> (+1.58%)	⬆️

... and 21 files with indirect coverage changes

@@            Coverage Diff             @@
##             main   #19221      +/-   ##
==========================================
+ Coverage   68.76%   68.80%   +0.04%     
==========================================
  Files         420      420              
  Lines       35650    35665      +15     
==========================================
+ Hits        24513    24539      +26     
+ Misses       9714     9703      -11     
  Partials     1423     1423              

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ecd7cfc...1dcdfc1. Read the comment docs.

[ahrtr]

@fuweid @ivanvc @jmhbnz @serathius

This PR fixed a regression caused by #19139. So let's get this merged and backport to 3.5 and probably 3.4. We need to get it included in 3.5.18

[ahrtr]

/retest

[ahrtr]

/test pull-etcd-integration-1-cpu-arm64