Danny McDermott 5/20/20 Danny McDermott 5/20/20

Generate Azure Stack Hub Certificates using an Enterprise CA - the automated way

Automate your Azure Stack Hub Certificates signed by a Windows Enterprise CA with these scripts.

Anyone who has had to deploy or operate Azure Stack Hub will tell you that one of the most laborious and tricky tasks is the generation of PKI certificates for the external endpoints. There are quite stringent requirements that can be found here.

I have already written about and created a script that can be used to generate Lets Encrypt signed certificates, but that doesn’t meet everyone’s requirements.

A common scenario is to use a Certificate Authority hosted on Windows Server. Here are the high level tasks that need to be carried out:

Generate the requests
Submit the request to the CA
Approve the request
Retrieve the signed cert
Import the signed cert
Export the certificate as a Pfx file with password
Place the exported Pfx files into specific folders for use by Azure Stack Hub
Test the certs for validity

Microsoft have helped by creating the Azure Stack Readiness Checker PowerShell module, which really is a great tool for generating the certificate requests and checking the validity of the Pfx files, but we need a process to automate all of the steps above.

To help with that, I’ve created a script that will take some inputs and do the rest for you. You can find it here.

The main script is New-AzsHubCertificates.ps1. It takes the following parameters:

  
    Parameter
    Type
    Description
  
    azsregion
    String
    Azure Stack Hub Region Name
  
    azsCertDir
    String
    Directory to store certificates
  
    CaServer
    String
    IP address fo the Certificate Authority Server
  
    IdentitySystem
    String
    Either AAD or ADFS
  
    CaCredential
    Credential
    Credentials to connect to the Certificate Authority Server
  
    AppServer
    Boolean
    Choose if App Service Certs should be generated
  
    DBAdapter
    Boolean
    Choose if DB Adapter Cert should be generated
  
    EventHubs
    Boolean
    Choose if Event Hubs Cert should be generated
  
    IoTHubs
    Boolean
    Choose if IOT Hub Cert should be generated
  
    SkipDeployment
    Boolean
    Choose if you do not require the deployment certificates generating
  
    pfxPassword
    SecureString
    Password for the Pfx Files

It utilizes the Azure Stack Readiness Checker module to generate the cert requests and to also validate that the certificates are fit for purpose, so ensure that you have the module installed. The script also needs to be run from an elevated session as it needs to import and export certificates from the Computer store that you run it from (needless to say you need to run it from a Windows client :) ).

The first thing the script does is create the folder for the certificates. It takes the input you specify and will create a sub-folder from there for the Azure Stack Hub Region, so you could run this script on the same system for multiple stamps.
Next, it uses the New-AzsCertificateSigningRequest function from the Readiness checker module to create the requests. it places them in the \requests directory.
After this, WinRM config is checked to see if the IP address of the CA server exists in Trusted Hosts. If not, it is added. All other entries are maintained.
A Session is established to the CA server, and the request files are copied to it.
On the CA server, Certutil.exe is used to retrieve the Public CA certificate and stored in a file.
On the CA server, CertReq.exe is used to submit each of the requests to the CA.
On the CA server, each request is approved using CertUtil -resubmit .
On the CA server, each Signed cert is retrieved and stored as a p7b and crt file.
Each signed cert is copied from the CA server to the local system where you’re running the script.
The Public CA certificate is copied from the CA server to the local system.
On the CA server, the working folder is deleted.
The remote PS Session is removed.
The Public CA certificate is imported in to the computers root store.
The directory structure required for validating certificates is created.
The signed certificates are imported.
The private certificates are exported and saved as a pfx file with password to the corresponding directory.
The certificates are validated using the Invoke-AzsCertificateValidation function from the Readiness checker module.

The folder structure for the certificates is as follows:

  
    C:\AZSCERTS\AZS1
+---AAD
|   +---ACSBlob
|   |       blob.pfx
|   |
|   +---ACSQueue
|   |       queue.pfx
|   |
|   +---ACSTable
|   |       table.pfx
|   |
|   +---Admin Extension Host
|   |       adminhosting.pfx
|   |
|   +---Admin Portal
|   |       adminportal.pfx
|   |
|   +---ARM Admin
|   |       adminmanagement.pfx
|   |
|   +---ARM Public
|   |       management.pfx
|   |
|   +---KeyVault
|   |       vault.pfx
|   |
|   +---KeyVaultInternal
|   |       adminvault.pfx
|   |
|   +---Public Extension Host
|   |       hosting.pfx
|   |
|   \---Public Portal
|           portal.pfx
|
+---AppServices
|   +---API
|   |       api.pfx
|   |
|   +---DefaultDomain
|   |       wappsvc.pfx
|   |
|   +---Identity
|   |       sso.pfx
|   |
|   \---Publishing
|           ftp.pfx
|
+---DBAdapter
|       DBAdapter.pfx
|
+---EventHubs
|       eventhub.pfx
|
\---IoTHub
        mgmtiothub.pfx

  

I have written an example script new-AzsHubcertificatesexample.ps1 so that you can generate the correct parameter types, or create a new, unique Pfx password for use by the main script. Change the variables according to your environment. You could take this further by storing the pfx password in a KeyVault; maybe I’ll write a further post on how to do this … :)

Danny McDermott 5/20/20 Danny McDermott 5/20/20

ARM Limits on Azure Stack Hub

I came across an interesting error when I was doing some maintenance work on an Azure Stack Hub stamp. I was updating App Service to version 2020 Q2 and also installing the Event Hubs Public Release preview. I got the following error message:

‘Number of read requests for subscription ‘‘ exceeded the limit of ‘15000’ for time interval ‘01:00:00’. Please try again after ‘‘ seconds. (Code: SubscriptionRequestsThrottled)’

I’d certainly not come across this before and was intrigued to see if I could find out any more information about this. I checked the Azure Stack Hub documentation and couldn’t find any reference to this limit, nor could I find anyone else who had come across this before.

What I did find related to Azure, so I thought it should be relevant. Details of the limits applied can be found in the article Throttling Resource Manager requests.

It turns out that Azure Stack Hub also applies limits to requests via ARM, although the limits are slightly different. Whereas for Azure the Read limit for a subscription is set to 12000, for Azure Stack Hub, it is set to 15000, per the error message I received.

I don’t know what the write / delete limits are, but assume they are similar to the ones set for Azure, give or take a thousand :)

So, how can you check how many read operations are remaining before hitting the limit?

The Throttling Resource Manager requests article details the methods you can use, by inspecting headers, or by using a script that was referenced in the doc: Check Resource Manager Limits for a Subscription.

This script doesn’t quite work for Azure Stack Hub, also, it doesn’t work if you’re the the AZ PowerShell Module, so I made some modifications and made them available here.

There are two versions, one for AzureRM and the other for Az PowerShell modules

Get-AzArmLimits.ps1

Get-AzureRMArmLimits.ps1

You need to be logged in to the Azure environment you want to check, but it will work for Azure and Azure Stack Hub

Here are some examples of the output:

Danny McDermott 3/9/20 Danny McDermott 3/9/20

Azure Stack Hub: When you can't view/add permissions in a tenant subscription from the portal

I have noticed on a few occasions that for a tenant subscription hosted in an Azure Stack region that I am either unable to view the IAM permissions, or add a user/service principal/group for the subscription. I have needed to do this to assign a service principal as a contributor to a subscription so that I can deploy a Kubernetes cluster via AKSE.

Typically for viewing the permissions, clicking on the Refresh button does the trick. More problematic is adding permissions via the portal. Doing so renders a screen like below:

As highlighted, the animated ‘dots’ show the blade constantly trying to retrieve the data but can’t. It is actually a known issue and is highlighted in the release notes.

The remediation it offers is to use PowerShell to verify the permissions, and gives a link to the Get-AzureRmRoleAssignment CmdLet. Not helpful if you want to set permissions, so here’s a step-by-step description of what you’ll need to do.

Pre-Reqs:

Your user account is the owner of the subscriptions when they were set up.
You have the correct PowerShell for Azure Stack Hub installed

For the example shown, I am using Azure AD identities, I have more than one tenant subscription assigned to my user account and I am adding a service principal. At the end of the post, I will show the commands for adding a group or a user .

Step-By-Step

From PowerShell, connect to the tenant subscription you want to change the permissions on. The documentation is for connecting as an operator, so here’s how to do it as a user:

$Region = '<yourRegion>'
$FQDN = '<yourFQDN>'
$AADTenantName = '<yourAADTenantName>'
$envName = "AzureStack$region" 

 # Register an Azure Resource Manager environment that targets your Azure Stack instance. Get your Azure Resource Manager endpoint value from your service provider.
Add-AzureRMEnvironment -Name $envName -ArmEndpoint "https://management.$Region.$FQDN" `
    -AzureKeyVaultDnsSuffix vault.$Region.$FQDN `
    -AzureKeyVaultServiceEndpointResourceId https://vault.$Region.$FQDN

# Set your tenant name.
$AuthEndpoint = (Get-AzureRmEnvironment -Name $envName).ActiveDirectoryAuthority.TrimEnd('/')
$TenantId = (invoke-restmethod "$($AuthEndpoint)/$($AADTenantName)/.well-known/openid-configuration").issuer.TrimEnd('/').Split('/')[-1]

# After signing in to your environment, Azure Stack cmdlets
# can be easily targeted at your Azure Stack instance.
Add-AzureRmAccount -EnvironmentName $envName -TenantId $TenantId

As I have more than one subscription, let’s get a list so we can select the correct one:

Get-AzureRmSubscription | ft Name

I got the following output:

I want to modify the AKSTest subscription:

#Set the context
$ctx = set-azurermcontext -Subscription AKSTest
$ctx

Running that command, I get :

Cool; I’ve got the correct subscription now, so let’s list the permissions assigned to it:

Get-AzureRmRoleAssignment

OK, so those are the standard permissions assigned when the subscription is created. Let’s add a service principal.

First, let’s get the service principal object

Get-AzureRmADServicePrincipal | ? {$_.DisplayName -like '*something*'} |ft DisplayName

I want to assign AKSEngineDemo

$spn = Get-AzureRmADServicePrincipal -SearchString AKSEngineDemo

Lets assign the service principal to the tenant subscription:

New-AzureRmRoleAssignment -ObjectId $spn.Id -RoleDefinitionName Contributor -scope "/subscriptions/$($ctx.Subscription.Id)"

When I take a look in the portal now (after doing a refresh :) ), I see the following:

Here’s the commands for adding an Azure AD User:

#Find the User object you want to add
Get-AzureRmADUser | ? {$_.DisplayName -like '*something*'} |ft DisplayName

#Assign the object using the displayname of the user
$ADUser = Get-AzureRmADUser -SearchString <UserName>
New-AzureRmRoleAssignment -ObjectId $ADUser.Id -RoleDefinitionName Contributor -scope "/subscriptions/$($ctx.Subscription.Id)"

Finally, how to add an Azure AD Group:

#Find the Group object you want to add
Get-AzureRmADGroup | ? {$_.DisplayName -like '*something*'} |ft DisplayName

#Assign the object using the displayname of the user
$ADGroup = Get-AzureRmADGroup -SearchString <GroupName>
New-AzureRmRoleAssignment -ObjectId $ADGroup.Id -RoleDefinitionName Contributor -scope "/subscriptions/$($ctx.Subscription.Id)"

Matt Quickenden 1/9/20 Matt Quickenden 1/9/20

Adding external load balancer IP address to an existing Kubernetes Service to update <pending>

This is one of those little things. I’m sure the veteran Kube admins out there are saying “well of course” but sometimes trying to sort through forums of Q&A on the internet is problematic even with the help of your favorite search engine. I thought I’d add one more page that might turn up and help someone else.

So you have created a self-managed K8s cluster and have you load balancer setup for your service and you have a bunch of <pending> when you look at your services. You can simply patch your service.

kubectl patch svc <svc-name> -n <namespace> -p '{"spec": {"type": "LoadBalancer", "externalIPs":["x.x.x.x"]}}

Danny McDermott 1/2/20 Danny McDermott 1/2/20

Fixing the SQL / MySQL RP update procedure on Azure Stack Hub 1910

Anyone who has been running and operating Azure Stack Hub for a while is hopefully aware that there are a number of activities that need to take place to keep it up to date and supported. The most obvious one is updating the core infrastructure using the integrated update process. It’s made easier in that the main dashboard on the admin portal for your stamp informs you of the update status - if it’s up to date or if there’s an update to apply. When it comes to additional Resource Providers, you have to do the work your self. This post details how to fix the update process for the SQL and MySQL RP’s for systems running version 1910.

I’m making the assumption that you have already deployed the RP’s previously, and have access to the pfx file that contains the sqladapter certificate for your stamp.

Version 1910 of Azure Stack Hub requires version 1.1.47.0 of the SQL & MySQL Resource Providers.

The first thing you will need to do is download the respective RP install binaries:

SQL RP can be found here: https://aka.ms/azurestacksqlrp11470
MySQL RP can be found here: https://aka.ms/azurestackmysqlrp11470

Extract the contents of the downloaded file(s) on a system that has access to the PEP and also has the compatible Azure Stack Hub PowerShell modules installed for Azure Satck Hub 1910: https://docs.microsoft.com/en-us/azure-stack/operator/azure-stack-powershell-install?view=azs-1910

Create a directory in the location where you extracted the RP files called cert. You will need to copy the pfx file containing the certificate for the SQL adapter into this directory. This step is the same for both SQL and MySQL.

We need to modify the common.psm1 file in the Prerequisites folder, similar to an earlier blog post on deploying the RP (based on 1.1.33.0)

Taking a look at the file shows us the prerequisites for the function:

It expects the Azure RM module version to be 2.3.0 and that the installation path for the modules to be \SqlMyslPsh.
If following the official instructions for deploying the Azure Stack PowerShell module compatible with 1910, this will never work, so we need to modify the Test-AzureStackPowerShell function.

Change the $AzPshInstallFolder on line 82 from "SqlMySqlPsh" to "WindowsPowerShell\Modules"

Change the $AzureRmVersion on line 84 from "2.3.0" to "5.8.3"

We need some further changes to the function to ensure that the correct path for the module is evaluated:

Locate line 118/119 in the module.

We need to modify the path so it reflects that of the loaded module. Change the "AzureRM" to AzureRM.Profile", as highlighted in the images below.

N.B. - The instructions for the fix are also applicable for new SQL & MySQL Resource Provider deployments, as they use the common.psm1 module too.

Save the file and using the update script example tailored to your environment, the update should now work.

You should see status messages like below:

Overall, it takes approximately 50 minutes to run through the update.

Topic Search

Posts by Date