My previous two posts have been relatively trivial modifications to modify the constraints and parameters exposed in Azure Stack Poc TP1 installer. There was a contrived underlying narrative leading us to here. As TP1 made it in to the wild, my team was tasked with being able to rapidly stand up PoC environments. We eventually landed on an LTI (Lite Touch Installation) type deployment solution which allows us to deploy and re-deploy fresh PoC environments within a couple of hours. This requirement, as expected evolved to exposing the public surfaces of Azure Stack (ARM, The Portal, Storage Accounts, etc.). I’ll attempt to outline the steps required in as concise a manner as possible (hat-tip to AzureStack.eu).
The way Azure Stack is currently glued together has the expectation that the Active Directory DNS Name of your deployment will match the DNS suffix used for the publicly accessible surfaces (e.g. portal.azurestack.local). The primary reasons for this is certificates and DNS resolution by the service fabric and clients alike. I’ll expound on this a little later. It was relatively simple to automate the exposing a new environment by allowing configuration to follow consistent convention. As a practical example, We have a number of blade chassis within our datacenters. In each chassis where we host PoC environments, each blade is on a different subnet with a specific set of IP addresses that will be exposed to the internet (if needed) via 1-to-1 NAT.
Each blade is on a different vlan and subnet for a reason. Azure Stack's use of VXLAN to talk between hosts in a scale out configuration means that if you try to stand up two hosts on the same layer 2 segment, you will have IP address conflicts between them on the private azure stack range. There are ways in the config to change the VXLAN IDs and avoid this, but it's simpler to just use separate VLANs. Each blade is on a dedicated private space VLAN, and is then NAT'd to the public internet thru a common router/firewall with 1:1 NATs on the appropriate addresses. This avoids having to put the blade directly on the internet, and a Static NAT avoids any sort of port address translation.
Prerequisites
You will need at least 5 public IP addresses and access to manage the DNS zone for the domain name you will be using. Our domain used for the subsequent content will be yourdomain.com (replace with your domain name) and I will assume that the ‘Datacenter’ (Considered the public network for Stack) network for your Azure Stack installation is will apply 1-to-1 NAT on the appropriate IP addresses exposed to the public internet.
We’ll use a hypothetical Internal IP address set of 172.20.10.33-172.20.10.39 and External IP address set of 38.x.x.30-38.x.x.34 for this example. The process I will describe will require static IP addressing on the NAT public interface. These options are required parameters of the installation along with the desired custom Active Directory DNS Domain Name.
Networking
Azure Stack Virtual IP Addresses
| Role |
Internal VIP |
‘Datacenter’ Network IP |
External IP |
| Portal and API |
192.168.133.74 |
172.20.10.33 |
38.x.x.30 |
| Gallery |
192.168.133.75 |
172.20.10.34 |
38.x.x.31 |
| CRP, NRP, SRP |
192.168.133.31 |
172.20.10.35 |
38.x.x.32 |
| Storage Account |
192.168.133.72 |
172.20.10.36 |
38.x.x.33 |
| Extensions |
192.168.133.71 |
172.20.10.37 |
38.x.x.34 |
Azure Stack VM IP Addresses
| Role |
Internal IP |
‘Datacenter’ Network IP |
| ADVM |
192.168.100.2 |
172.20.10.38 |
| ClientVM |
192.168.200.101 |
172.20.10.39 |
DNS
We will need to start a new deployment with our desired domain name. Please see my previous post Changing Azure Stack’s Active Directory DNS Domain to something other than AzureStack.local on how to make this an installation parameter. Please note, modifications to the additional resource provider (e.g. Sql and Web Apps) installers will also be required. We just changed instances of azurestack.local to $env:USERDNSDOMAIN as they are primarily executed within the Azure Stack Active Directory context.
To be honest, I probably should have addressed this in the previous post. If the Active Directory DNS domain name you wish to use will be resolvable by the host while deploying the stack I recommend you simply add an entry to the HOSTS file prior to initiating installation. We ended up adding this to our deployment process as we wanted to leave public DNS entries in place for certain environments. In this version we know the resultant IP address of the domain controller that will be created.
Add an entry like such:
192.168.100.2 yourdomain.com
You will need to add a number of A records within your public DNS zone.
| A Record |
IP Address |
| api.yourdomain.com |
38.x.x.30 |
| portal.yourdomain.com |
38.x.x.30 |
| gallery.yourdomain.com |
38.x.x.31 |
| srp.yourdomain.com |
38.x.x.32 |
| nrp.yourdomain.com |
38.x.x.32 |
| srp.yourdomain.com |
38.x.x.32 |
| xrp.yourdomain.com |
38.x.x.32 |
| *.compute.yourdomain.com |
38.x.x.32 |
| *.storage.yourdomain.com |
38.x.x.32 |
| *.network.yourdomain.com |
38.x.x.32 |
| *.blob.yourdomain.com |
38.x.x.33 |
| *.queue.yourdomain.com |
38.x.x.33 |
| *.table.yourdomain.com |
38.x.x.33 |
| *.adminextensions.yourdomain.com |
38.x.x.34 |
| *.tenantextensions.yourdomain.com |
38.x.x.34 |
Installation
The NAT IP addressing and Active Directory DNS domain name need to be supplied to the PoC installation. A rough script running in the context of the installation folder will look something like this:
[powershell]
$ADDomainName="yourdomain.com"
$AADTenant="youraadtenant.com"
$AADUserName="yourStackUser@$AADTenant"
$AADPassword=ConvertTo-SecureString -String "ThisIsYourAADPassword!" -AsPlainText -Force
$AdminPassword=ConvertTo-SecureString -String "ThisIsYourStackPassword!" -AsPlainText -Force
$NATVMStaticIP="172.20.10.30/24"
$NATVMStaticGateway="172.20.10.1"
$AADCredential=New-Object pscredential($AADUserName,$AADPassword)
$DeployScript= Join-Path $PSScriptRoot "DeployAzureStack.ps1"
&$DeployScript -ADDomainName $ADDomainName -AADTenant $AADTenant `
-AdminPassword $AdminPassword -AADCredential $AADCredential `
-NATVMStaticIP $NATVMStaticIP -NATVMStaticGateway $NATVMStaticGateway -Verbose -Force
[/powershell]
Configuring Azure Stack NAT
The public internet access, both ingress and egress, is all controlled via Routing and Remote Access and networking within the NATVM virtual machine. There are effectively three steps you will need to undertake that will require internal VIP addresses within the stack and the public IP addresses you wish to associate with your deployment. This process can be completed through the UI, however, given the nature of our lab, doing this in a scriptable fashion was an imperative. If you must do this by hand, start by opening the Routing and Remote Access MMC on the NATVM in order to modify the public NAT settings.
Create a Public IP Address Pool within the RRAS NAT configuration.
Create IP Address reservations for the all of the IP addresses for which the public DNS entries were created within the RRAS NAT configuration. Note incoming sessions are required for all the Azure Stack public surfaces.
Associate the additional Public IP addresses with the NATVM Public Interface in the NIC settings. (This is a key step that isn't covered elsewhere)
I told you that we needed to script this and unfortunately, as far as I can tell, the easiest way is still through the network shell netsh.exe. The parameters for our script are the same for every deployment and a script file for netsh is dynamically created. If you already have a large amount of drool on your keyboard from reading this tome, I'll cut to the chase and give you a sample to drive netsh. In the following example Ethernet 2 is the 'Public' Adapter on the NATVM and our public IP Addresses are in the 172.20.10.0/24 network and will be mapped according to the previous table
[text]
#Set the IP Addresses
pushd interface ipv4
add address name="Ethernet 2" address=172.20.27.38 mask=255.255.255.0
add address name="Ethernet 2" address=172.20.27.31 mask=255.255.255.0
add address name="Ethernet 2" address=172.20.27.33 mask=255.255.255.0
add address name="Ethernet 2" address=172.20.27.35 mask=255.255.255.0
add address name="Ethernet 2" address=172.20.27.34 mask=255.255.255.0
add address name="Ethernet 2" address=172.20.27.37 mask=255.255.255.0
add address name="Ethernet 2" address=172.20.27.36 mask=255.255.255.0
add address name="Ethernet 2" address=172.20.27.32 mask=255.255.255.0
add address name="Ethernet 2" address=172.20.27.39 mask=255.255.255.0
popd
#Configure NAT
pushd routing ip nat
#NAT Address Pool
add addressrange name="Ethernet 2" start=172.20.27.31 end=172.20.27.39 mask=255.255.255.0
#NAT Pool Reservations
add addressmapping name="Ethernet 2" public=172.20.27.38 private=192.168.100.2 inboundsessions=disable
add addressmapping name="Ethernet 2" public=172.20.27.31 private=192.168.5.5 inboundsessions=disable
add addressmapping name="Ethernet 2" public=172.20.27.33 private=192.168.133.74 inboundsessions=enable
add addressmapping name="Ethernet 2" public=172.20.27.35 private=192.168.133.31 inboundsessions=enable
add addressmapping name="Ethernet 2" public=172.20.27.34 private=192.168.133.75 inboundsessions=enable
add addressmapping name="Ethernet 2" public=172.20.27.37 private=192.168.133.71 inboundsessions=enable
add addressmapping name="Ethernet 2" public=172.20.27.36 private=192.168.133.72 inboundsessions=enable
add addressmapping name="Ethernet 2" public=172.20.27.32 private=192.168.200.101 inboundsessions=disable
add addressmapping name="Ethernet 2" public=172.20.27.39 private=192.168.100.12 inboundsessions=disable
popd
[/text]
If I haven’t lost you already and you would like to achieve this with PowerShell, the following script uses PowerShell direct to configure NAT in the previously demonstrated manner taking a PSCredential (an Adminstrator on NATVM) and an array of very simple objects representing the (for now) constant NAT mappings.
They can easily be stored in JSON (and I insist that is pronounced like the given name Jason) for easy serialization/de-serialization
[powershell]
#requires -Modules NetAdapter,NetTCPIP -Version 5.0
[CmdletBinding()]
param
(
[Parameter(Mandatory=$true)]
[Object[]]
$Mappings,
[Parameter(Mandatory=$false)]
[Int32]
$PoolSize=8,
[Parameter(Mandatory=$false)]
[String]
$NatVmInternalIP="192.168.200.1",
[Parameter(Mandatory=$true)]
[pscredential]
$Credential,
[Parameter(Mandatory=$false)]
[String]
$NatVmName="NATVM"
)
$ActivityId=33
$NatSetupScriptBlock=
{
[CmdletBinding()]
param()
$VerbosePreference=$Using:VerbosePreference
$ParentId=$Using:ActivityId
$NatVmInternalIP=$Using:NatVmInternalIP
$Mappings=@($Using:Mappings)
$AddressPoolSize=$Using:PoolSize
#region Helper Methods
<# .SYNOPSIS Converts a CIDR or Prefix Length as string to a Subnet Mask .PARAMETER CIDR The CIDR or Prefix Length to convert #>
Function ConvertTo-SubnetMaskFromCIDR
{
[CmdletBinding()]
[OutputType([String])]
param
(
[Parameter(Mandatory=$true,ValueFromPipeline=$true)]
[String]
$CIDR
)
$NetMask="0.0.0.0"
$CIDRLength=[System.Convert]::ToInt32(($CIDR.Split('/')|Select-Object -Last 1).Trim())
Write-Debug "Converting Prefix Length $CIDRLength from input $CIDR"
switch ($CIDRLength) {
{$_ -gt 0 -and $_ -lt 8}
{
$binary="$( "1" * $CIDRLength)".PadRight(8,"0")
$o1 = [System.Convert]::ToInt32($binary.Trim(),2)
$NetMask = "$o1.0.0.0"
break
}
8 {$NetMask="255.0.0.0"}
{$_ -gt 8 -and $_ -lt 16}
{
$binary="$( "1" * ($CIDRLength - 8))".PadRight(8,"0")
$o2 = [System.Convert]::ToInt32($binary.Trim(),2)
$NetMask = "255.$o2.0.0"
break
}
16 {$NetMask="255.255.0.0"}
{$_ -gt 16 -and $_ -lt 24}
{
$binary="$("1" * ($CIDRLength - 16))".PadRight(8,"0")
$o3 = [System.Convert]::ToInt32($binary.Trim(),2)
$NetMask = "255.255.$o3.0"
break
}
24 {$NetMask="255.255.255.0"}
{$_ -gt 24 -and $_ -lt 32}
{
$binary="$("1" * ($CIDRLength - 24))".PadRight(8,"0")
$o4 = [convert]::ToInt32($binary.Trim(),2)
$NetMask= "255.255.255.$o4"
break
}
32 {$NetMask="255.255.255.255"}
}
return $NetMask
}
Function Get-ExternalNetAdapterInfo
{
[CmdletBinding()]
param
(
[Parameter(Mandatory=$true)]
[String]
$ExcludeIP
)
$AdapterInfos=@()
$NetAdapters=Get-NetAdapter -Physical
foreach ($NetAdapter in $NetAdapters)
{
if($NetAdapter.Status -eq 'Up')
{
$AdapIpAddress=$NetAdapter|Get-NetIPAddress -AddressFamily IPv4
$AdapterInfo=New-Object PSObject -Property @{
Adapter=$NetAdapter;
IpAddress=$AdapIpAddress;
}
$AdapterInfos+=$AdapterInfo
}
}
$DesiredAdapter=$AdapterInfos|Where-Object{$_.IPAddress.IPAddress -ne $ExcludeIP}
return $DesiredAdapter
}
Function Get-NetshScriptContent
{
[CmdletBinding()]
param
(
[Parameter(Mandatory=$true)]
[Object]
$ExternalAdapter,
[Parameter(Mandatory=$true)]
[Object[]]
$NatMappings,
[Parameter(Mandatory=$true)]
[Int32]
$PoolSize
)
#Retrieve the Network Adapters
$ExternalAddress=$ExternalAdapter.IPAddress.IPAddress
$ExternalPrefixLength=$ExternalAdapter.IPAddress.PrefixLength
$ExternalAdapterName=$ExternalAdapter.Adapter.Name
Write-Verbose "External Network Adapter [$ExternalAdapterName] $($ExternalAdapter.Adapter.InterfaceDescription) - $ExternalAddress"
$IpPieces=$ExternalAddress.Split(".")
$LastOctet=[System.Convert]::ToInt32(($IpPieces|Select-Object -Last 1))
$IpFormat="$([String]::Join(".",$IpPieces[0..2])).{0}"
$PublicCIDR="$IpFormat/{1}" -f 0,$ExternalPrefixLength
$AddressPoolStart=$IpFormat -f ($LastOctet + 1)
$AddressPoolEnd=$IpFormat -f ($LastOctet + 1 + $PoolSize)
$ExternalNetMask=ConvertTo-SubnetMaskFromCIDR -CIDR $PublicCIDR
Write-Verbose "Public IP Address Pool Start:$AddressPoolStart End:$AddressPoolEnd Mask:$ExternalNetMask"
$TargetIpEntries=@()
$ReservationEntries=@()
foreach ($Mapping in $NatMappings)
{
Write-Verbose "Evaluating Mapping $($Mapping.name)"
$TargetPublicIp=$IpFormat -f $Mapping.publicIP
$TargetIpEntry="add address name=`"{0}`" address={1} mask={2}" -f $ExternalAdapterName,$TargetPublicIp,$ExternalNetMask
$TargetIpEntries+=$TargetIpEntry
if($Mapping.allowInbound)
{
$InboundEnabled="enable"
}
else
{
$InboundEnabled="disable"
}
$ReservationEntry="add addressmapping name=`"{0}`" public={1} private={2} inboundsessions={3}" -f $ExternalAdapterName,$TargetPublicIp,$Mapping.internalIP,$InboundEnabled
$ReservationEntries+=$ReservationEntry
}
$NetshScriptLines=@()
#IP Addresses
$NetshScriptLines+="#Set the IP Addresses"
$NetshScriptLines+="pushd interface ipv4"
$TargetIpEntries|ForEach-Object{$NetshScriptLines+=$_}
$NetshScriptLines+="popd"
#NAT
$NetshScriptLines+="#Configure NAT"
$NetshScriptLines+="pushd routing ip nat"
$NetshScriptLines+="#NAT Address Pool"
$NetshScriptLines+="add addressrange name=`"{0}`" start={1} end={2} mask={3}" -f $ExternalAdapterName,$AddressPoolStart,$AddressPoolEnd,$ExternalNetMask
$NetshScriptLines+="#NAT Pool Reservations"
$ReservationEntries|ForEach-Object{$NetshScriptLines+=$_}
$NetshScriptLines+="popd"
return $NetshScriptLines
}
#endregion
#region Execution
Write-Progress -ParentId $ParentId -Activity "Publishing Azure Stack" -Status "Detecting Public Adapter" -PercentComplete 5
$ExternalNetAdapter=Get-ExternalNetAdapterInfo -ExcludeIP $NatVmInternalIP
Write-Progress -Activity "Publishing Azure Stack" `
-Status "External Network Adapter [$($ExternalNetAdapter.Adapter.Name)] $($ExternalNetAdapter.Adapter.InterfaceDescription) - $($ExternalNetAdapter.IPAddress.IPAddress)" -PercentComplete 10
if($ExternalNetAdapter -eq $null)
{
throw "Unable to resolve the public network adapter!"
}
Write-Progress -ParentId $ParentId -Activity "Publishing Azure Stack" -Status "Creating Network Script" -PercentComplete 25
$NetshScript=Get-NetshScriptContent -ExternalAdapter $ExternalNetAdapter -NatMappings $Mappings -PoolSize $AddressPoolSize
#Save the file..
$NetShScriptPath=Join-Path $env:TEMP "configure-nat.txt"
$NetShScriptExe= Join-Path $env:SystemRoot 'System32\netsh.exe'
#Run NetSh
Set-Content -Path $NetShScriptPath -Value $NetshScript -Force
Write-Progress -ParentId $ParentId -Activity "Publishing Azure Stack" -Status "Created Network Script $NetShScriptPath" -PercentComplete 50
Write-Progress -ParentId $ParentId -Activity "Publishing Azure Stack" -Status "Configuring NAT $NetShScriptExe -f $NetShScriptPath" -PercentComplete 70
$NetShProcess=Start-Process -FilePath $NetShScriptExe -ArgumentList "-f $NetShScriptPath" -PassThru -Wait
Write-Progress -ParentId $ParentId -Activity "Publishing Azure Stack" -Status "Restarting RRAS.." -PercentComplete 90
Restart-Service -Name RemoteAccess
Write-Progress -ParentId $ParentId -Activity "Publishing Azure Stack" -PercentComplete 100 -Completed
#endregion
$ConfigureResult=New-Object PSObject -Property @{
NetSHProcess=$NetShProcess;
NetSHScript=$NetShScript;
}
return $ConfigureResult
}
Write-Progress -Id $ActivityId -Activity "Configuring NAT" -Status "Publishing Azure Stack TP1 PoC $NatVmName as $($Credential.UserName)..." -PercentComplete 10
$result=Invoke-Command -ScriptBlock $NatSetupScriptBlock -Credential $Credential -VMName $NatVmName
Write-Progress -Id $ActivityId -Activity "Configuring NAT" -PercentComplete 100 -Completed
return $result
[/powershell]
Certificates
You will need to export the root certificate from the CA for your installation for importing on any clients that will access your deployment. I will leave the use of publicly trusted certificates outside the scope of this post, nor will I address the use of a CA other than the one deployed with Active Directory. The expanse of aforementioned DNS entries should give you an idea of how many certificates you will need to obtain to avoid this step with any clients. In the case of operating public Azure, this is one place where it must be very nice being a Trusted Root Certificate Authority.
Exporting the root certificate is very simple as the PoC host system is joined to the domain which hosts the Azure Stack CA, both roles reside on the ADVM. To export the Root certificate to your desktop run this simple one-liner in the PowerShell console of your Host system (the same command will work from the ClientVM).
[powershell]
Get-ChildItem -Path Cert:\LocalMachine\Root| `
Where-Object{$_.Subject -like "CN=AzureStackCertificationAuthority*"}| `
Export-Certificate -FilePath "$env:USERPROFILE\Desktop\$($env:USERDOMAIN)RootCA.cer" -Type CERT
[/powershell]
The process for importing this certificate on your client will vary depending on the OS version; as such I will avoid giving a scripted method.
Right click the previously exported certificate.
Choose Current User for most use-cases.
Select Browse for the appropriate store.
Select Trusted Root Certificate Authorities
Confirm the Import Prompt
Enjoying the Results
If we now connect to the portal, we should be prompted to login via Azure Active Directory and upon authentication you should be presented with a familiar portal.
You can also connect with the Azure Cmdlets if that is more your style. We'll make a slight modification to the snippet from the post by Mike De Luca How to Connect to Azure Stack via PowerShell.
[powershell]
$ADDomainName="yourdomain.com"
$AADTenant="youraadtenant.com"
$AADUserName="yourStackUser@$AADTenant"
$AADPassword=ConvertTo-SecureString -String "ThisIsYourAADPassword!" -AsPlainText -Force
$AADCredential=New-Object PSCredential($AADUserName,$AADPassword)
$StackEnvironmentName="Azure Stack ($ADDomainName)"
$StackEnvironment=Add-AzureRmEnvironment -Name $StackEnvironmentName `
-ActiveDirectoryEndpoint ("https://login.windows.net/$AADTenant/") `
-ActiveDirectoryServiceEndpointResourceId "https://$ADDomainName-api/" `
-ResourceManagerEndpoint ("Https://api.$ADDomainName/") `
-GalleryEndpoint ("Https://gallery.$($ADDomainName):30016/") `
-GraphEndpoint "https://graph.windows.net/"
Add-AzureRmAccount -Environment $StackEnvironment -Credential $AADCredential
[/powershell]
Your Azure Stack TP1 PoC deployment should now be available to serve all clients from the public internet in an Azure Consistent manner.
[Unsupported]
