ci: [Service Tags] add public ips with service tags for LBs during cluster creation

[k-routhu]

Reason for Change:

Adding service tags to all public ips. This change adds public ips and attaches these ips as load balancer outbound ips during cluster creation. Branch spathak/add-service-tags is where all the changes were made and tested -- created this branch based off of that one.

Testing:

• All testing was done on branch spathak/add-service-tags
• We had a couple of PRs out by Esteban and Shubham to update service tags. However, all of the PRs targeted hack/aks/Makefile, so I went ahead and made all of the updates in this branch.
• Also added ipv6 public ips to load balancer outbound ips for dualstack clusters to resolve errors

Tested changes in a couple of Cilium pipelines:

CNI Nightly Pipeline - Task
The public ips created have service tags attached to them, and the ips fall within the allowed ranges for the locations they were created

CNI Release Test - Task
All of the public ips created have service tags attached to them, and the ips fall within the allowed ranges for the locations they were created

CNI - LSG Integration Test - Task
The clusters get created here successfully with the correct public ips. The integration tests do fail, but I checked with John Payne -- he mentioned that the integration tests fail frequently because they're heavily dependent on a partner team (Canonical). He suggested running Cilium Private Test pipelines on v1.14 and v1.16 (the next two on the list) and the remaining pipelines with the service tags. The remaining pipelines look good and create public ips as expected, so I believe we should be good to go on that front.

ACN PR Pipeline - Task
All of the public ips created have service tags attached to them, and the ips fall within the allowed ranges for the locations they were created

[jpayne3506]

Formatting

[jpayne3506]

All ?= should be lined up.

[jpayne3506]

Duplicate rg-up calls when creating cluster. I would remove the rg-up from the make calls public-ipv4 and public-ipv6

[k-routhu]

updated this

[jpayne3506]

Inconsistent removal of --load-balancer-sku. Does it need to be standard or can we keep it as basic?

[k-routhu]

can be basic, I meant to remove --load-balancer-sku standard for nodesubnet-byocni-nokubeproxy-up since I'm adding --load-balancer-outbound-ips <publicip v4>,<publicip v6>

[jpayne3506]

Does this implicitly set every cluster to standard for the load balancer sku?

[rohitams]

will this rerun the public ip creation? if so, should we have the ip creation as a global step so it provisions just once?

[k-routhu]

yes, this reruns public ip creation, but since the Cilium pipelines do cluster creation in parallel, we need to provision different public ips

[rbtr]

why are you reinvoking Make for this? unless you are overriding or changing args and options versus the Make invocation you are already in, you could make the IP target a dependency here, like rg-up etc.

[thomasricci]

Are these steps run in parallel or in a sequential order?
Public IPs from ST are limited so we should try to either reuse or not create too many public ips at the same time

[k-routhu]

these steps are run in parallel, but I do believe we have the capacity to create the public ips. The majority of the clusters are created in westus and westus2, and they have 254 and 512 available vips respectively, and we create about 10-12 clusters (and public ips) in each location, and delete them all at the end

[thomasricci]

Did we request IPV6 ips for our test ST ?

[k-routhu]

discussed offline, should be good to go on ipv6

[jpayne3506]

nit: Lets keep this line in.

[jpayne3506]

All ?= should be lined up.

[jpayne3506]

Does this implicitly set every cluster to standard for the load balancer sku?

[rbtr]

are these identical except for "v4" and "v6"? consider how you could simplify and parameterize throughout, like this:

Suggested change

	public-ipv4:
	$(AZCLI) network public-ip create --name $(PUBLIC_IPv4) \
	--resource-group $(GROUP) \
	--allocation-method Static \
	--ip-tags $(IP_TAG) \
	--location $(REGION) \
	--sku Standard \
	--tier Regional \
	--version IPv4
	public-ipv6:
	$(AZCLI) network public-ip create --name $(PUBLIC_IPv6) \
	--resource-group $(GROUP) \
	--allocation-method Static \
	--ip-tags $(IP_TAG) \
	--location $(REGION) \
	--sku Standard \
	--tier Regional \
	--version IPv6
	ip:
	$(AZCLI) network public-ip create --name $(IP_PREFIX)-$(CLUSTER)-$(IPVERSION) \
	--resource-group $(GROUP) \
	--allocation-method Static \
	--ip-tags $(IP_TAG) \
	--location $(REGION) \
	--sku Standard \
	--tier Regional \
	--version IP$(IPVERSION)
	ipv4:
	@$(MAKE) ip IPVERSION=v4
	ipv6:
	@$(MAKE) ip IPVERSION=v6

[rbtr]

are these strings pre-determined?

[rbtr]

why are you reinvoking Make for this? unless you are overriding or changing args and options versus the Make invocation you are already in, you could make the IP target a dependency here, like rg-up etc.

[timraymond]

Why are we using make for this? Can't most of this be done more predictably by Bicep/ARM?

[rbtr]

Why are we using make for this? Can't most of this be done more predictably by Bicep/ARM?

@timraymond yes and no.
AzCLI is basically a wrapper to ARM when used like this and that's what it actually sends, so in a literal sense, yes. Bicep would be even better for declarative infrastructure.

I originally wrote this Makefile because I needed to build infra that required certain subscription flags enabled (which ARM may be able to do, but was documented as AzCLI commands), and a certain preview CLI installed locally and/or a preview AZ AKS plugin installed locally. This made it easy for everyone to deploy infra identically without copy/pasting out of some install steps OneNote every time.
And, more importantly to me at the time, AzCLI (including ARM, Bicep, and anything that depends on it) was and is still completely busted on Fedora. Platforming everything in the azure-cli container makes this totally portable.

I also find it nice to have a central list of capabilities. We could write Bicep for everything here, but I think I would still want the Makefile to exist as the single entrypoint to all of the bicep apply <scenario> so that it's all easy to find (and to wrap in the azcli image so that IT WORKS AT ALL 😤)