Adding your custom images to MaaS

In my last two posts, we covered creating custom Windows and RHEL images for MaaS. Now we'll deal with uploading them for use. The process is relatively simple:

1. Upload your image to the MaaS server

2. Import the image to MaaS

The first step is easy enough, just use SCP (or a program like WinSCP in Windows) to upload the file:

To Import the image in MaaS, use the following command:

maas $profile boot-resources create name=custom/$imagedisplayname architecture=amd64/generic content=@$tgzfilepath

In the code, I've used variables to make explanation simpler. In real life, I'd just input the actual values. Following is an explanation of each value:

$profile - When using the maas command, you must first log in. This variable represents your maas profile name.

$imagedisplayname - You can put any value here. Preceding it by custom/ means that the image will show up under custom images in MaaS with the name specified following the /

$tgzfilepath - This is the full path to the image file on the MaaS server. If you are admin2 and you kept an image namede my-image.tgz in a directory called images under your home directory, this value would be /home/admin2/images/my-image.tgz

Creating a custom Windows image for MaaS

Continuing on the Maas theme from yesterday, I thought I'd put up a post about my experience with Windows imaging for MaaS. The Windows Openstack Imaging Tools are great for creating Windows images for Maas. They provide support for creating gold images, custom drivers, and pretty much anything else you'd want for custom image creation.

Included is support for UEFI or BIOS boot options. This is where I ran into issues. I found that Hyper-V wasn't having secureboot turned off for the VM, causing BIOS builds to fail. I found the problem and fixed it relatively easily. After successfully creating a BIOS image, and finding it wouldn't install on most of our equipment due to it being configured for UEFI, I decided to create a UEFI image.

Creating a UEFI image errored out, and so the hunt for the issue began. I won't bore you with the tedium of digging through code to figure out where the problem lay, but there are two changes that need to be made to the PS module that does most of the heavy lifting in the solution.

In the install directory, there is a file called WinImageBuilder.psm1. The first fix is to insert the following code right after line 902.

if ($DiskLayout -eq "UEFI")         {         $Drive = $Drive[1]         }

The solution doesn't treat the disk layout as an array if UEFI is selected, so naturally the disk optomization phase fails.

Following that, the next change is after line 1006. Insert the following code:

if ($DiskLayout -ne "UEFI")         {         Set-VMFirmware -VMName (Get-VM).Name -EnableSecureBoot Off         }

Turning off secureboot enables BIOS builds to succeed.

Once these fixes are in place, you can set your variables and have a successful build .

Creating a custom RHEL image for MAAS

A little change from our typical programming, today's post is about how to add an image to an Ubuntu MaaS instance. So what is a Sr Consultant, specializing in Azure and containerization doing working with Linux in an on-premises environment?

As far-removed as it seems from the norm of what we do at Avanade, there is still a lot of demand for on-prem solutions. Many cases are like ours; we have a large lab full of equipment we've already paid for and it makes fiscal sense to do much of our dev work there.

In our lab, we utilize MAAS (Metal as a Service) to manage our hardware. Out of the box (for the free version of MAAS at least), MAAS comes with images for Ubuntu and CENTOS. With the paid version of MAAS, you get Windows and RHEL images as well.

A brief glance around the internet showed that there used to be a tool called MAAS-image-builder that allows for the custom creation of RHEL images. Ubuntu seems to have burried many of the references to this once they started using images as a differentiation between free and paid versions. There are actually a few branches of this, none of which worked out of the box.

The version we eventually got to work was this one: https://code.launchpad.net/~ltrager/maas-image-builder/update_ks

To make it work, you must build it as per the instructions, and once installed, make a few changes:

1. Add the following to /usr/lib/maas-image-builder/contrib/rhel/rhel7-amd64.ks in the %packages section:

Without the python2-oauthlib.noarch and python2-requests-oauthlib.noarch packages, the install will complete, but MAAS will register it as a failed deployment.

Additionally, as anyone familiar with kickstart files will probably tell you, it's advisable to create a standard user account, and register your subscription in addition to any other customization you want to add to your image.

One caveat however: Keep in mind that no matter what you do with the part command in your kickstart, it won't actually make a difference to the final install process in MAAS. MAAS manages storage layouts on its own, and won't allow a custom storage layout for any OS other than Ubuntu.

AD FS identity integration on Azure Stack – filling in the gaps

One of the clients I’ve been engaged with use AD FS as the identity provider for their Azure Stack integrated system. All well and good, as setting that up using the instructions provided here is *fairly* straightforward: https://docs.microsoft.com/en-us/azure/azure-stack/azure-stack-integrate-identity. Here’s a high level of the tasks that need to be performed:

On Azure Stack (by the operator via Privileged Endpoint PowerShell session):

1. Setup Graph integration (configure to point to on-premises AD Domain, so user / group searches can be performed, used by IAM/RBAC)

2. Setup AD FS integration (Create federation data metafile and use automation to configure claims provider trust with on-premises AD FS)

3. Set Service Admin Owner to user in the on-premises AD Domain

On customer AD FS server by an admin with correct permissions:

1. Configure claims provider trust (either by helper script provided in Azure Stack tools, or manually)

2. If performing manually:

   1. Enable Windows forms-based authentication

   2. Add the relying party trust

   3. Configure AD FS properties to ignore token ring bindings (If using IE / Edge browsers and AD FS is running on WS 2016)

   4. Set AD FS Relying party trust with Token lifetime of 1440

   5. If performing by helper script:

      1. From Azure Stack tools directory, navigate to \DatacenterIntegration\Identity and run setupadfs.ps1

The only gotcha with the instructions that I encountered was that the certificate chain for AD FS was different than was provisioned for Azure Stack endpoints, so I tried to follow the instructions for this scenario provided in the link above, but they didn’t work.

It turns out that there was a problem with me running the provided PowerShell code:

[XML]$Metadata = Invoke-WebRequest -URI https:///federationmetadata/2007-06/federationmetadata.xml -UseBasicParsing

$Metadata.outerxml|out-file c:\metadata.xml

$federationMetadataFileContent = get-content c:\metadata.cml

$creds=Get-Credential

Enter-PSSession -ComputerName  -ConfigurationName PrivilegedEndpoint -Credential $creds

Register-CustomAdfs -CustomAdfsName Contoso -CustomADFSFederationMetadataFileContent $using:federationMetadataFileContent

View fullsize

…and here’s one that is correctly configured:

View fullsize

The things to check for are that the correct FQDN to the AD domain are provided and the user / password combination is correct. Graph just needs a ‘normal’ user account with no special permissions. Make sure the password for the user is set to not expire!

You can re-run the command from the PEP without having to run Reset-DatacenterIntegationConfiguration. Only run this when AD FS integration is broken.

If you want to use AD Groups via Graph for RBAC control, keep in mind that they need to be Universal, otherwise they will not appear.

Hopefully this information will help some of you out.

In my next blog post, I’ll fill in the gaps on creating AD FS SPN’s for use by automation / Azure CLI on Azure Stack.

Azure to Azure Stack site-to-site IPSec VPN tunnel failure... after 8 hours

We had a need to create a site-to-site VPN tunnel for a POC from Azure Stack to Azure.  It seemed pretty straight forward.  Spoiler alert, obviously I'm writing this because it wasn't. The tunnel was created okay, but each morning it would no longer allow traffic to travel across it.  The tunnel would show connected in Azure and in Azure Stack but traffic just wouldn't flow; ping, SSH, RDP, DNS and AD all wouldn't work.  After some tinkering we found we would have to change the connection's sharedkey value to something random, save it, then change it back to the correct key.  This only worked from the Azure Stack side of the connection, to re-initiate successfully and allow traffic to flow again (or recreate the connection from scratch).  It would work for another 8 hours and then fail to pass traffic again.

My suspicion was the re-keying, as this would explain why it worked at first and would fail the next day (everyday, for the last 5 days).  I tried using VPN diagnostics on the Azure side, as they don't currently support VPN diagnostics on Azure Stack (we are on update 1805).  After reviewing the IKE log there were some errors, but it was hard to find something to tell me what was going wrong, more specifically something I could do to fix it.  Below is the IKE log file I collected through the VPN diagnostics from Azure.

I logged a case with Microsoft support.  The first support person did their best.  While Microsoft can identify the endpoints they are connecting to, from Azure, they do not have permission to dig any deeper and look into the contents of our subscriptions hosted on Azure Stack.  I was asked to change the local VPN gateways from specific subnets to be the entire vnet address space.  While it worked initially, again it failed after 8 hours.

The support engineer collected some network traffic and other logs and forwarded the case to an Azure Stack support engineer. Once the call was assigned they asked me to connect to the privileged endpoint (PEP) and we proceeded with breaking the glass to Azure Stack to troubleshoot.  The engineer gave me a few PowerShell commands to run to investigate what was going on.

#First find out which of the VPN gateways is active. icm Azs-gwy01,Azs-gwy02 { get-vpns2sinterface }

#Check Quick Mode Key Exchange icm Azs-gwy01 { get-netIpsecQuickModeSA } 

#Check Main Mode Key Exchange icm Azs-gwy01 { get-netIpsecMainModeSA }

The Microsoft engineer had a hunch of exactly what he was looking for and was on point. The commands showed that the Quick mode key exchange had failed to complete the refresh, yet the Main Mode had succeeded.  This explained why the tunnel was up but no traffic could flow across it.

We rebooted the active VPN gateway so the tunnels would fail-over to the second gateway.  Logging was on by default so we just had to wait for the next timeout to occur.  When it did I was given the task of collecting and uploading the logs from the PEP.

These logs are a series of ETL files that need to be processed by Microsoft to make sense of them.  Fortunately it turned up the following log entries.

As commented above, the root cause was that the PFS and CipherType setting were incorrect on the Azure VPN gateway.  I was given a few PowerShell commands to run against the Azure Subscription to reconfigure the IPSec policy for the connection on the Azure side to match the policy of the VPN gateway and connection on Azure Stack.

$RG1 = 'RESOURCE GROUP NAME' $CONN = 'CONNECTION NAME' $GWYCONN = Get-AzureRmVirtualNetworkGatewayConnection -Name $CONN -ResourceGroupName $RG1 $newpolicy  = New-AzureRmIpsecPolicy -IkeEncryption AES256 -IkeIntegrity SHA256 -DhGroup DHGroup2 -IpsecEncryption GCMAES256 -IpsecIntegrity GCMAES256 -PfsGroup PFS2048 -SALifeTimeSeconds 27000 -SADataSizeKilobytes 33553408 Set-AzureRmVirtualNetworkGatewayConnection -VirtualNetworkGatewayConnection $GWYCONN -IpsecPolicies $newpolicy 

Almost there.  When I tried to run the command, the basic Sku doesn't allow for custom IPSec policies.  Once I changed the Sku from basic to standard the command worked and the tunnel has been up and stable.

While this is any easy fix that anyone can run against their Azure subscription without opening a support ticket, this does incur a cost difference.  Hopefully in the future these policies will match out-of-the-box between Azure and Azure Stack so every consumer can use the basic VPN Sku to connect Azure Stack to Azure over a secure tunnel.