Add Event Grid System Topic for AKS (#212)

* Add Event Grid System Topic for AKS * Fix missing Azure RBAC step for workflow deployment * Fix deployment failure when admin and read-only group are the same * resource property ordering * Disable Local Accounts is GA
mspnp · Jul 19, 2021 · 83087e9 · 83087e9
1 parent 40854c3
commit 83087e9
Show file tree

Hide file tree

Showing 6 changed files with 68 additions and 29 deletions.
diff --git a/01-prerequisites.md b/01-prerequisites.md
@@ -27,7 +27,7 @@ This is the starting point for the instructions on deploying the [AKS Secure Bas
 1. While the following feature(s) are still in _preview_, please enable them in your subscription.
    1. [Register the Azure Key Vault Secrets Provider for AKS preview feature - `AKS-AzureKeyVaultSecretsProvider`](https://docs.microsoft.com/azure/aks/csi-secrets-store-driver#register-the-aks-azurekeyvaultsecretsprovider-preview-feature).
 
-   1. [Register the Disable Local Accounts preview feature - 'DisableLocalAccountsPreview'](https://docs.microsoft.com/azure/aks/managed-aad#register-the-disablelocalaccountspreview-preview-feature)
+   1. [Register the Azure Event Grid preview feature - 'EventgridPreview'](https://docs.microsoft.com/azure/aks/quickstart-event-grid#register-the-eventgridpreview-preview-feature)
 
 1. Clone/download this repo locally, or even better fork this repository.
 

diff --git a/04-networking.md b/04-networking.md
@@ -98,7 +98,7 @@ The following two resource groups will be created and populated with networking
    az deployment group create -g rg-enterprise-networking-hubs -f networking/hub-regionA.json -p location=eastus2 nodepoolSubnetResourceIds="['${RESOURCEID_SUBNET_NODEPOOLS}']"
    ```
 
-   > :book: At this point the networking team has delivered a spoke in which BU 0001's app team can lay down their AKS cluster (ID: A0008). The networking team provides the necessary information to the app team for them to reference in their Infrastructure-as-Code artifacts.
+   > :book: At this point the networking team has delivered a spoke in which BU 0001's app team can lay down their AKS cluster (ID: A0008). The networking team provides the necessary information to the app team for them to reference in their infrastructure-as-code artifacts.
    >
    > Hubs and spokes are controlled by the networking team's GitHub Actions workflows. This automation is not included in this reference implementation as this body of work is focused on the AKS baseline and not the networking team's CI/CD practices.
 

diff --git a/05-aks-cluster.md b/05-aks-cluster.md
@@ -97,6 +97,7 @@ Now that the [hub-spoke network is provisioned](./04-networking.md), the next st
            sed "s#<cluster-spoke-vnet-resource-id>#${RESOURCEID_VNET_CLUSTERSPOKE}#g" | \
            sed "s#<tenant-id-with-user-admin-permissions>#${TENANTID_K8SRBAC}#g" | \
            sed "s#<azure-ad-aks-admin-group-object-id>#${AADOBJECTID_GROUP_CLUSTERADMIN}#g" \
+           sed "s#<azure-ad-aks-a0008-group-object-id>#${AADOBJECTID_GROUP_A0008_READER}#g" \
            > .github/workflows/aks-deploy.yaml
        ```
 

diff --git a/cluster-stamp.json b/cluster-stamp.json
@@ -254,9 +254,9 @@
                     }
                 },
                 {
-                    "name": "Microsoft.Insights/default",
                     "type": "providers/diagnosticSettings",
                     "apiVersion": "2017-05-01-preview",
+                    "name": "Microsoft.Insights/default",
                     "dependsOn": [
                         "[resourceId('Microsoft.KeyVault/vaults', variables('keyVaultName'))]",
                         "[resourceId('Microsoft.OperationalInsights/workspaces', variables('logAnalyticsWorkspaceName'))]"
@@ -1168,9 +1168,9 @@
             },
             "resources": [
                 {
-                    "name": "[concat('Microsoft.Authorization/', guid(resourceId('Microsoft.ContainerService/managedClusters', variables('clusterName')), 'omsagent', variables('monitoringMetricsPublisherRole')))]",
                     "type": "providers/roleAssignments",
                     "apiVersion": "2020-04-01-preview",
+                    "name": "[concat('Microsoft.Authorization/', guid(resourceId('Microsoft.ContainerService/managedClusters', variables('clusterName')), 'omsagent', variables('monitoringMetricsPublisherRole')))]",
                     "comments": "Grant the OMS Agent's Managed Identity the metrics publisher role to push alerts",
                     "dependsOn": [
                         "[resourceId('Microsoft.ContainerService/managedClusters', variables('clusterName'))]"
@@ -1182,7 +1182,7 @@
                     }
                 },
                 {
-                    "type": "/providers/diagnosticSettings",
+                    "type": "providers/diagnosticSettings",
                     "apiVersion": "2017-05-01-preview",
                     "name": "Microsoft.Insights/default",
                     "dependsOn": [
@@ -1213,11 +1213,51 @@
                 }
             ]
         },
+        {
+            "type": "Microsoft.EventGrid/systemTopics",
+            "apiVersion": "2020-10-15-preview",
+            "name": "[variables('clusterName')]",
+            "location": "[parameters('location')]",
+            "dependsOn": [
+                "[resourceId('Microsoft.ContainerService/managedClusters', variables('clusterName'))]"
+            ],
+            "properties": {
+
+                "source": "[resourceId('Microsoft.ContainerService/managedClusters', variables('clusterName'))]",
+                "topicType": "Microsoft.ContainerService.ManagedClusters"
+            },
+            "resources": [
+                {
+                    "type": "providers/diagnosticSettings",
+                    "apiVersion": "2017-05-01-preview",
+                    "name": "Microsoft.Insights/default",
+                    "dependsOn": [
+                        "[resourceId('Microsoft.EventGrid/systemTopics', variables('clusterName'))]",
+                        "[resourceId('Microsoft.OperationalInsights/workspaces', variables('logAnalyticsWorkspaceName'))]"
+                    ],
+                    "properties": {
+                        "workspaceId": "[resourceId('Microsoft.OperationalInsights/workspaces', variables('logAnalyticsWorkspaceName'))]",
+                        "logs": [
+                            {
+                                "category": "DeliveryFailures",
+                                "enabled": true
+                            }
+                        ],
+                        "metrics": [
+                            {
+                                "category": "AllMetrics",
+                                "enabled": true
+                            }
+                        ]
+                    }
+                }
+            ]
+        },
         {
             "type": "Microsoft.Insights/metricAlerts",
-            "location": "global",
             "apiVersion": "2018-03-01",
             "name": "[concat('Node CPU utilization high for ', variables('clusterName'), ' CI-1')]",
+            "location": "global",
             "dependsOn": [
                 "[resourceId('Microsoft.ContainerService/managedClusters', variables('clusterName'))]",
                 "[resourceId('Microsoft.OperationsManagement/solutions',variables('containerInsightsSolutionName'))]"
@@ -1261,9 +1301,9 @@
         },
         {
             "type": "Microsoft.Insights/metricAlerts",
-            "location": "global",
             "apiVersion": "2018-03-01",
             "name": "[concat('Node working set memory utilization high for ', variables('clusterName'), ' CI-2')]",
+            "location": "global",
             "dependsOn": [
                 "[resourceId('Microsoft.ContainerService/managedClusters', variables('clusterName'))]",
                 "[resourceId('Microsoft.OperationsManagement/solutions',variables('containerInsightsSolutionName'))]"
@@ -1307,9 +1347,9 @@
         },
         {
             "type": "Microsoft.Insights/metricAlerts",
-            "location": "global",
             "apiVersion": "2018-03-01",
             "name": "[concat('Jobs completed more than 6 hours ago for ', variables('clusterName'), ' CI-11')]",
+            "location": "global",
             "dependsOn": [
                 "[resourceId('Microsoft.ContainerService/managedClusters', variables('clusterName'))]",
                 "[resourceId('Microsoft.OperationsManagement/solutions',variables('containerInsightsSolutionName'))]"
@@ -1360,9 +1400,9 @@
         },
         {
             "type": "Microsoft.Insights/metricAlerts",
-            "location": "global",
             "apiVersion": "2018-03-01",
             "name": "[concat('Container CPU usage high for ', variables('clusterName'), ' CI-9')]",
+            "location": "global",
             "dependsOn": [
                 "[resourceId('Microsoft.ContainerService/managedClusters', variables('clusterName'))]",
                 "[resourceId('Microsoft.OperationsManagement/solutions',variables('containerInsightsSolutionName'))]"
@@ -1413,9 +1453,9 @@
         },
         {
             "type": "Microsoft.Insights/metricAlerts",
-            "location": "global",
             "apiVersion": "2018-03-01",
             "name": "[concat('Container working set memory usage high for ', variables('clusterName'), ' CI-10')]",
+            "location": "global",
             "dependsOn": [
                 "[resourceId('Microsoft.ContainerService/managedClusters', variables('clusterName'))]",
                 "[resourceId('Microsoft.OperationsManagement/solutions',variables('containerInsightsSolutionName'))]"
@@ -1466,9 +1506,9 @@
         },
         {
             "type": "Microsoft.Insights/metricAlerts",
-            "location": "global",
             "apiVersion": "2018-03-01",
             "name": "[concat('Pods in failed state for ', variables('clusterName'), ' CI-4')]",
+            "location": "global",
             "dependsOn": [
                 "[resourceId('Microsoft.ContainerService/managedClusters', variables('clusterName'))]",
                 "[resourceId('Microsoft.OperationsManagement/solutions',variables('containerInsightsSolutionName'))]"
@@ -1512,9 +1552,9 @@
         },
         {
             "type": "Microsoft.Insights/metricAlerts",
-            "location": "global",
             "apiVersion": "2018-03-01",
             "name": "[concat('Disk usage high for ', variables('clusterName'), ' CI-5')]",
+            "location": "global",
             "dependsOn": [
                 "[resourceId('Microsoft.ContainerService/managedClusters', variables('clusterName'))]",
                 "[resourceId('Microsoft.OperationsManagement/solutions',variables('containerInsightsSolutionName'))]"
@@ -1565,9 +1605,9 @@
         },
         {
             "type": "Microsoft.Insights/metricAlerts",
-            "location": "global",
             "apiVersion": "2018-03-01",
             "name": "[concat('Nodes in not ready status for ', variables('clusterName'), ' CI-3')]",
+            "location": "global",
             "dependsOn": [
                 "[resourceId('Microsoft.ContainerService/managedClusters', variables('clusterName'))]",
                 "[resourceId('Microsoft.OperationsManagement/solutions',variables('containerInsightsSolutionName'))]"
@@ -1611,9 +1651,9 @@
         },
         {
             "type": "Microsoft.Insights/metricAlerts",
-            "location": "global",
             "apiVersion": "2018-03-01",
             "name": "[concat('Containers getting OOM killed for ', variables('clusterName'), ' CI-6')]",
+            "location": "global",
             "dependsOn": [
                 "[resourceId('Microsoft.ContainerService/managedClusters', variables('clusterName'))]",
                 "[resourceId('Microsoft.OperationsManagement/solutions',variables('containerInsightsSolutionName'))]"
@@ -1664,9 +1704,9 @@
         },
         {
             "type": "Microsoft.Insights/metricAlerts",
-            "location": "global",
             "apiVersion": "2018-03-01",
             "name": "[concat('Persistent volume usage high for ', variables('clusterName'), ' CI-18')]",
+            "location": "global",
             "dependsOn": [
                 "[resourceId('Microsoft.ContainerService/managedClusters', variables('clusterName'))]",
                 "[resourceId('Microsoft.OperationsManagement/solutions',variables('containerInsightsSolutionName'))]"
@@ -1717,9 +1757,9 @@
         },
         {
             "type": "Microsoft.Insights/metricAlerts",
-            "location": "global",
             "apiVersion": "2018-03-01",
             "name": "[concat('Pods not in ready state for ', variables('clusterName'), ' CI-8')]",
+            "location": "global",
             "dependsOn": [
                 "[resourceId('Microsoft.ContainerService/managedClusters', variables('clusterName'))]",
                 "[resourceId('Microsoft.OperationsManagement/solutions',variables('containerInsightsSolutionName'))]"
@@ -1770,9 +1810,9 @@
         },
         {
             "type": "Microsoft.Insights/metricAlerts",
-            "location": "global",
             "apiVersion": "2018-03-01",
             "name": "[concat('Restarting container count for ', variables('clusterName'), ' CI-7')]",
+            "location": "global",
             "dependsOn": [
                 "[resourceId('Microsoft.ContainerService/managedClusters', variables('clusterName'))]",
                 "[resourceId('Microsoft.OperationsManagement/solutions',variables('containerInsightsSolutionName'))]"
@@ -1825,7 +1865,7 @@
             "condition": "[variables('isUsingAzureRBACasKubernetesRBAC')]",
             "type": "Microsoft.Authorization/roleAssignments",
             "apiVersion": "2018-09-01-preview",
-            "name": "[guid(concat('aad-admin-group', resourceId('Microsoft.ContainerService/managedClusters', variables('clusterName'))))]",
+            "name": "[guid('aad-admin-group', resourceId('Microsoft.ContainerService/managedClusters', variables('clusterName')), parameters('clusterAdminAadGroupObjectId'))]",
             "dependsOn": [
                 "[resourceId('Microsoft.ContainerService/managedClusters', variables('clusterName'))]"
             ],
@@ -1840,7 +1880,7 @@
             "condition": "[variables('isUsingAzureRBACasKubernetesRBAC')]",
             "type": "Microsoft.Authorization/roleAssignments",
             "apiVersion": "2018-09-01-preview",
-            "name": "[guid(concat('aad-admin-group-sc', resourceId('Microsoft.ContainerService/managedClusters', variables('clusterName'))))]",
+            "name": "[guid('aad-admin-group-sc', resourceId('Microsoft.ContainerService/managedClusters', variables('clusterName')), parameters('clusterAdminAadGroupObjectId'))]",
             "dependsOn": [
                 "[resourceId('Microsoft.ContainerService/managedClusters', variables('clusterName'))]"
             ],
@@ -1852,10 +1892,10 @@
             }
         },
         {
-            "condition": "[variables('isUsingAzureRBACasKubernetesRBAC')]",
+            "condition": "[and(variables('isUsingAzureRBACasKubernetesRBAC'), not(equals(parameters('a0008NamespaceReaderAadGroupObjectId'), parameters('clusterAdminAadGroupObjectId'))))]",
             "type": "Microsoft.Authorization/roleAssignments",
             "apiVersion": "2018-09-01-preview",
-            "name": "[guid(concat('aad-a0008-reader-group', resourceId('Microsoft.ContainerService/managedClusters', variables('clusterName'))))]",
+            "name": "[guid('aad-a0008-reader-group', resourceId('Microsoft.ContainerService/managedClusters', variables('clusterName')), parameters('a0008NamespaceReaderAadGroupObjectId'))]",
             "dependsOn": [
                 "[resourceId('Microsoft.ContainerService/managedClusters', variables('clusterName'))]"
             ],
@@ -1867,10 +1907,10 @@
             }
         },
         {
-            "condition": "[variables('isUsingAzureRBACasKubernetesRBAC')]",
+            "condition": "[and(variables('isUsingAzureRBACasKubernetesRBAC'), not(equals(parameters('a0008NamespaceReaderAadGroupObjectId'), parameters('clusterAdminAadGroupObjectId'))))]",
             "type": "Microsoft.Authorization/roleAssignments",
             "apiVersion": "2018-09-01-preview",
-            "name": "[guid(concat('aad-a0008-reader-group-sc', resourceId('Microsoft.ContainerService/managedClusters', variables('clusterName'))))]",
+            "name": "[guid('aad-a0008-reader-group-sc', resourceId('Microsoft.ContainerService/managedClusters', variables('clusterName')), parameters('a0008NamespaceReaderAadGroupObjectId'))]",
             "dependsOn": [
                 "[resourceId('Microsoft.ContainerService/managedClusters', variables('clusterName'))]"
             ],
@@ -2042,10 +2082,6 @@
             "type": "string",
             "value": "[variables('clusterName')]"
         },
-        "agwName": {
-            "type": "string",
-            "value": "[variables('agwName')]"
-        },
         "aksIngressControllerPodManagedIdentityResourceId": {
             "type": "string",
             "value": "[resourceId('Microsoft.ManagedIdentity/userAssignedIdentities', 'podmi-ingress-controller')]"

diff --git a/github-workflow/aks-deploy.yaml b/github-workflow/aks-deploy.yaml
@@ -40,11 +40,12 @@ on:
 env:
   RESOURCE_GROUP_LOCATION: '<resource-group-location>'                              # The location where the resource group is going to be created
   RESOURCE_GROUP: '<resource-group-name>'                                           # The name for the AKS cluster resource group
-  AKS_LOCATION: '<resource-group-location>'                                        # The location where the AKS cluster is going to be deployed
+  AKS_LOCATION: '<resource-group-location>'                                         # The location where the AKS cluster is going to be deployed
   GEO_REDUNDANCY_LOCATION: '<geo-redundancy-location>'                              # The location for Azure resources that support native geo-redunancy. Should be different than the location parameter and ideally should be a paired region - https://docs.microsoft.com/en-us/azure/best-practices-availability-paired-regions. This region does not need to support availability zones.
   TARGET_VNET_RESOURCE_ID: '<cluster-spoke-vnet-resource-id>'                       # The regional network spoke VNet Resource ID that the cluster will be joined to
   K8S_RBAC_AAD_PROFILE_TENANTID: '<tenant-id-with-user-admin-permissions>'          # The tenant to integrate AKS-managed Azure AD
   K8S_RBAC_AAD_PROFILE_ADMIN_GROUP_OBJECTID: '<azure-ad-aks-admin-group-object-id>' # The Azure AD group object ID that has admin access to the AKS cluster
+  K8S_RBAC_AAD_A0008_READER_GROUP_OBJECTID: '<azure-ad-aks-a0008-group-object-id>'  # The Azure AD group object ID that has readonly access to the a0008 namespace in the AKS cluster
   CLUSTER_AUTHORIZED_IP_RANGES: '[]'                                                # By default, this deployment will allow unrestricted access to your cluster's API Server. You should limit access to the API Server to a set of well-known IP addresses (i.,e. your hub firewall IP, bastion subnet, build agents, or any other networks you'll administer the cluster from), and can do so by adding a CLUSTER_AUTHORIZED_IP_RANGES="['managementRange1', 'managementRange2', 'AzureFirewallIP/32']"" parameter.
 jobs:
   deploy:
@@ -77,6 +78,7 @@ jobs:
               targetVnetResourceId=${{ env.TARGET_VNET_RESOURCE_ID }} \
               k8sControlPlaneAuthorizationTenantId=${{ env.K8S_RBAC_AAD_PROFILE_TENANTID }} \
               clusterAdminAadGroupObjectId=${{ env.K8S_RBAC_AAD_PROFILE_ADMIN_GROUP_OBJECTID }} \
+              a0008NamespaceReaderAadGroupObjectId=${{ env.K8S_RBAC_AAD_A0008_READER_GROUP_OBJECTID }} \
               clusterAuthorizedIPRanges=${{ env.CLUSTER_AUTHORIZED_IP_RANGES}} \
               appGatewayListenerCertificate=${{ secrets.APP_GATEWAY_LISTENER_CERTIFICATE_BASE64 }} \
               aksIngressControllerCertificate=${{ secrets.AKS_INGRESS_CONTROLLER_CERTIFICATE_BASE64 }}

diff --git a/inner-loop-scripts/azcli/network-deploy.azcli b/inner-loop-scripts/azcli/network-deploy.azcli
@@ -50,7 +50,7 @@ az deployment group create --resource-group rg-enterprise-networking-hubs --temp
 
 # At this point the networking team has delivered a spoke in which BU 0001's app team can lay down
 # their AKS cluster (ID: A0008). The networking team provides the necessary information to the app team
-# for them to reference in their Infrastructure-as-Code artifacts.
+# for them to reference in their infrastructure-as-code artifacts.
 
 # Note: Hubs and Spokes are controlled by the Networking Team's GitHub actions pipelines.  This automation
 # is not included in this reference implementation as this is focused on the AKS baseline and not the networking