If you are working with Azure DevTest Labs you already know that all the nice things are built around to provide a test environment for Azure IaaS VMs. So far so good, but what can you do to provide your DevTest Labs users with access to PaaS services? Azure DevTest Labs got you covered with a feature called environments.
Environments are based on Azure Resource Manager templates and with the ARM templates you can provide every Azure service to your DevTest Labs users that supports ARM templates. Either it is a complex Azure IaaS VM environment or an Azure PaaS service.
In this blog post I will provide you with the guidance on how to enable your DevTest Labs users to deploy an Azure Container Service – Kubernetes cluster. We will use the following Azure services to achieve this:
- Azure DevTest Labs
- Azure Automation
- Azure Logic App
- Azure Event Grid
You may now think, why I need Azure Automation, Logic App and Event Grid for that and I will tell you it throughout the blog article. Ideally you already have deployed an Azure DevTest Lab. If not, go ahead and create one.
-> https://docs.microsoft.com/en-us/azure/devtest-lab/devtest-lab-create-lab
Make sure that the lab settings controlling the access rights are set to Reader.
Otherwise your DevTest Labs user would be able to deploy other Azure services you do not want into their resource group.
Moving on in our journey to provide our DevTest Labs users with a Kubernetes cluster, we must create an Azure Automation account next.
-> https://docs.microsoft.com/en-us/azure/automation/automation-create-standalone-account
Afterwards we need to know the application id of the Azure Automation Run As account. For this go into your subscription object and click on Access control (IAM).
Copy the name of the Azure Automation Run As account and run the following PowerShell command to get the application id.
DanielsTechBlog.info > Get-AzureRmADServicePrincipal -SearchString "AutomationDTL_UqiJk3zrLr9mS14HgjFlgaOWfSpULKC/PRBkBD7MJGM="
ServicePrincipalNames : {https://spn/UqiJk3zrLr9mS14HgjFlgaOWfSpULKC%2fPRBkBD7MJGM%3d, 55207697-e806-44ad-88c8-2091ed9aade9}
ApplicationId : 55207697-e806-44ad-88c8-2091ed9aade9
DisplayName : AutomationDTL_UqiJk3zrLr9mS14HgjFlgaOWfSpULKC/PRBkBD7MJGM=
Id : aef64e45-4cda-4192-bf60-ede97c9d8cb1
Type : ServicePrincipal
DanielsTechBlog.info >
Now, we select the Key Vault that got created during the DevTest Lab creation and going onto Access policies. There we will add ourselves and the Azure Automation Run As account with the permission template “Secret Management”. We will need the access to the Key Vault secrets later to provision the values of the Kubernetes Azure AD Service Principal.
Before we move on to create the necessary PowerShell runbook in our Azure Automation account, we must import and update some PowerShell modules first. In addition to the standard set of PowerShell modules we will need the AzureRM.Profile and the AzureRM.KeyVault modules.
-> https://docs.microsoft.com/en-us/azure/automation/automation-update-azure-modules
Next, create a PowerShell runbook and copy the following code into it.
Param
(
[Parameter(Mandatory = $true, HelpMessage = 'Azure Key Vault name')]
[String]
$keyVaultName
)
$aadServicePrincipalIdName = "kubernetesId"
$aadServicePrincipalSecretName = "kubernetesSecret"
$sourceKeyVaultName = "DevTestLab5996"
$targetKeyVaultName = $keyVaultName.Split("/")
$targetKeyVaultName = $targetKeyVaultName[$targetKeyVaultName.length - 1]
$servicePrincipalName = "55207697-e806-44ad-88c8-2091ed9aade9"
$connectionName = "AzureRunAsConnection"
try {
# Get the connection "AzureRunAsConnection "
$servicePrincipalConnection = Get-AutomationConnection -Name $connectionName
"Logging in to Azure..."
Add-AzureRmAccount `
-ServicePrincipal `
-TenantId $servicePrincipalConnection.TenantId `
-ApplicationId $servicePrincipalConnection.ApplicationId `
-CertificateThumbprint $servicePrincipalConnection.CertificateThumbprint
}
catch {
if (!$servicePrincipalConnection) {
$ErrorMessage = "Connection $connectionName not found."
throw $ErrorMessage
}
else {
Write-Error -Message $_.Exception
throw $_.Exception
}
}
try {
Set-AzureRmKeyVaultAccessPolicy -VaultName $targetKeyVaultName -ServicePrincipalName $servicePrincipalName -PermissionsToSecrets all
}
catch {
Write-Output 'ERROR:'
Write-Output $_
}
try {
$aadServicePrincipalId = Get-AzureKeyVaultSecret -VaultName $sourceKeyVaultName -Name $aadServicePrincipalIdName
$aadServicePrincipalSecret = Get-AzureKeyVaultSecret -VaultName $sourceKeyVaultName -Name $aadServicePrincipalSecretName
Set-AzureKeyVaultSecret -VaultName $targetKeyVaultName -Name $aadServicePrincipalIdName -SecretValue $aadServicePrincipalId.SecretValue
Set-AzureKeyVaultSecret -VaultName $targetKeyVaultName -Name $aadServicePrincipalSecretName -SecretValue $aadServicePrincipalSecret.SecretValue
}
catch {
Write-Output 'ERROR:'
Write-Output $_
}
You need to modify the following variables $sourceKeyVaultName and $servicePrincipalName. $sourceKeyVaultName is the DevTest Labs Key Vault name and $servicePrincripalName is the application id of the Azure Automation Run As account. The input parameter will come later through the Azure Event Grid and Logic App workflow and is the resource id of the target Key Vault that is exclusively created for the specific DevTest Labs user.
The PowerShell runbook does the following steps:
- Login to Azure
- Sets required access policy permissions on target Key Vault
- Reads the Kubernetes Azure AD Service Principal Id and Secret from the source Key Vault.
- Saves the Kubernetes Azure AD Service Principal Id and Secret as new secrets in the target Key Vault
As you can see the PowerShell runbook is used to provision the Kubernetes Azure AD Service Principal Id and Secret, that is needed by the Kubernetes cluster during the deployment and later for operations, to the target Key Vault of the DevTest Labs user.
Now, we can start to create our Logic App with the workflow. We will start with a blank Logic App and adding Azure Event Grid as a connector. We will get only one trigger named “Azure Event Grid – When a resource event occurs”.
This trigger kicks off the Logic App every time an event occurs in the specified resource group, so we need two conditions to filter on Key Vault creation and who initiated the creation to tightly control the kickoff of the Azure Automation runbook.
The first condition we add in the edit in advanced mode is to filter on the Key Vault creation event.
@equals(triggerBody()?['data']['operationName'], 'Microsoft.KeyVault/vaults/write')
If this is true, then the second condition comes into play to filter on the Azure AD Service Principal VSDevTestLab, that creates the specific Key Vault for the DevTest Labs user. But first we need the application id of the AAD Service Principal VSDevTestLab. Run the following PowerShell cmdlet to get it.
DanielsTechBlog.info > Get-AzureRmADServicePrincipal -SearchString "VSDevTestLab"
ServicePrincipalNames : {VSDevTestLab, 1a14be2a-e903-4cec-99cf-b2e209259a0f}
ApplicationId : 1a14be2a-e903-4cec-99cf-b2e209259a0f
DisplayName : VSDevTestLab
Id : 9d706dfc-a44a-4341-9cac-e995cc8a1530
Type : ServicePrincipal
DanielsTechBlog.info >
Then we add in the edit in advanced mode the following condition.
@equals(triggerBody()?['data']['claims']['appid'], '1a14be2a-e903-4cec-99cf-b2e209259a0f')
If this is true, we finally create the job to kick off the Azure Automation runbook.
With these two conditions, we can ensure that only at the creation stage of the specific Key Vault for the DevTest Labs user, the runbook will be kicked off.
Before the DevTest Labs users can use the Azure Container Service – Kubernetes cluster deployment, we must prepare the DevTest Lab environment. The first step is to create an Azure AD Service Principal for the Kubernetes cluster deployment & operational tasks and assign the Service Principal Contributor rights on the subscription level.
The Contributor right must be set on the subscription level due to the DevTest Labs resource deployment procedure. When a DevTest Labs user provisions a new VM or environment, the resource deployment creates a new resource group for each VM or environment deployment. With the Contributor right on the subscription level we have the advantage of the inheritance of access control permissions for the Azure AD Service Principal to all underlying resource groups.
Second step that must be done, is to save the Service Principal Id and Secret into the Key Vault that got created during the DevTest Lab creation.
Last step during the DevTest Lab environment preparation, is to connect the Azure Resource Manager template repository. Have a look at the Azure documentation on how to do that.
Now, it is time for DevTest Labs user to save his public SSH key as a secret in the DevTest Lab for accessing the deployed Kubernetes cluster later.
This action triggers the Key Vault deployment for the DevTest Labs user run by the VSDevTestLab Service Principal. The creation is a resource event that gets sent through the Azure Event Grid to the Logic App. In the Logic App this kicks off the workflow and finally ends in the Azure Automation runbook run.
The Azure AD Service Principal Id and Secret deployment will take a few minutes. Afterwards the DevTest Labs user is ready to deploy an Azure Container Service – Kubernetes cluster or even better an Azure Container Service – Managed Kubernetes cluster (AKS).
The DevTest Labs user provides only some basic inputs because we as a DevTest Labs administrator are using a nested ARM template for the deployment to provide fixed values for several parameters that the DevTest Labs user cannot change.
You can find the used ARM templates in my GitHub repository.
-> https://github.com/neumanndaniel/devtestlab
After the successful deployment the DevTest Labs user sees the Kubernetes cluster environment in the DevTest Lab with the master node and the three agent nodes. Connecting to the Kubernetes cluster requires to know the FQDN or public IP address of the master node. Just a click on the master node and the DevTest Labs user jumps into the details.
Accessing an AKS cluster is a bit different and the DevTest Labs user needs the resource group name and the cluster name for it.
-> https://docs.microsoft.com/en-us/azure/aks/kubernetes-walkthrough#connect-to-the-cluster
Hopefully you did not get tired by reading the long article, but finally got an idea on how you can provide more value to your DevTest Labs users using ARM templates for environment deployments. In addition to that you got additional input on leveraging different Azure services like Event Grid, Logic App and Azure Automation to play together.