ARM

Replace/Fix Unicode characters created by ConvertTo-Json in PowerShell for ARM Templates

unicode-escape-charactes.png

Often there will be instances where you may want to make a programmatic change to an ARM template using a scripting language like PowerShell. Parsing XML or JSON type files in the past wasn’t always the easiest thing via some scripting languages. Those of us who remember the days of parsing XML files using VBScript still occasionally wake up in the middle of the night in a cold sweat.

Thankfully, PowerShell makes this relatively easy for even non-developers through the use of some native conversion cmdlets that allow you to quickly take a JSON file and convert it to and from a custom PowerShell object that you can easily traverse and modify. You can do this with the ConvertFrom-Json cmdlet like so:

[powershell]$myJSONobject = Get-Content -raw "c:\myJSONfile.txt" | ConvertFrom-Json[/powershell]

Without digging into all the different ways that you can use this to tweak an ARM template to your heart’s satisfaction, it’s reasonable to assume that at some point you’d like to convert your newly modified object back into a plain ole’ JSON file with the intention of submitting it back into Azure, and that’s when things go awry. When you use the intuitively named ConvertTo-Json cmdlet to convert your object back into a flat JSON text block, your gorgeous block of JSON content that once looked like this:

Now looks like this:

See the problem? Here it is again with some helpful highlighting:

All your special characters get replaced by Unicode escape characters. While PowerShell’s ConvertFrom-Json cmdlet and other parsers handle these codes just fine when converting from text, Azure will have none of this nonsense and your template will fail to import/deploy/etc. Fortunately, the fix is very simple. The Regex class in the .NET framework has an “unescape” method:

https://msdn.microsoft.com/en-us/library/system.text.regularexpressions.regex(v=vs.110).aspx

So instead of using:

[powershell]$myOutput = $myJSONobject | ConvertTo-Json -Depth 50[/powershell]

Use:

[powershell]$myOutput = $myJSONobject | ConvertTo-Json -Depth 50 | % { [System.Text.RegularExpressions.Regex]::Unescape($_) }[/powershell]

The output from the second command should generate a perfectly working ARM template. That is, of course, assuming that you correctly built your ARM template in the first place, but I take no responsibility for that. ;)

ExpressRoute Migration from ASM to ARM and legacy ASM Virtual Networks

word-image9.png

I recently ran into an issue where an ExpressRoute had been migrated from Classis (ASM) to the new portal (ARM), however legacy Classic Virtual Networks (VNets) were still in operation. These VNets refused to be deleted by either portal or PowerShell. Disconnecting the old VNet’s Gateway through the Classic portal would show success, but it would stay connected.

There’s no option to disconnect an ASM gateway in the ARM portal, only a delete option. Gave this a shot and predictably, this was the result:

C:\Users\will.van.allen\AppData\Local\Microsoft\Windows\INetCache\Content.Word\FailedDeleteGW.PNG

Ok, let’s go to PowerShell and look for that obstinate link. Running Get-AzureDedicatedCircuitLink resulted in the following error:

PS C:\> get-AzureDedicatedCircuitLink -ServiceKey $ServiceKey -VNetName $Vnet

get-AzureDedicatedCircuitLink : InternalError: The server encountered an internal error. Please retry the request.

At line:1 char:1

+ get-AzureDedicatedCircuitLink -ServiceKey xxxxxx-xxxx-xxxx-xxxx-xxx...

+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

+ CategoryInfo          : CloseError: (:) [Get-AzureDedicatedCircuitLink], CloudException

+ FullyQualifiedErrorId : Microsoft.WindowsAzure.Commands.ExpressRoute.GetAzureDedicatedCircuitLinkCommand

I couldn’t even find the link. Not only was modifying the circuit an issue, but reads were failing, too.

Turned out to be a simple setting change. When the ExpressRoute was migrated, as there were still Classic VNets, a final step of enabling the circuit for both deployment models was needed. Take a look at the culprit setting here, after running Get-AzureRMExpressRouteCircuit:

"serviceProviderProperties": {

"serviceProviderName": "equinix",

"peeringLocation": "Silicon Valley",

"bandwidthInMbps": 1000

},

"circuitProvisioningState": "Disabled",

"allowClassicOperations": false,

"gatewayManagerEtag": "",

"serviceKey": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx",

"serviceProviderProvisioningState": "Provisioned"

AllowClassicOperations set to “false” blocks ASM operations from any access, including a simple “get” from the ExpressRoute circuit. Granting access is straightforward:

# Get details of the ExpressRoute circuit

$ckt = Get-AzureRmExpressRouteCircuit -Name "DemoCkt" -ResourceGroupName "DemoRG"

#Set "Allow Classic Operations" to TRUE

$ckt.AllowClassicOperations = $true

More info on this here.

But we still weren’t finished. I could now get a successful response from this:

get-AzureDedicatedCircuitLink -ServiceKey $ServiceKey -VNetName $Vnet

However this still failed:

Remove-AzureDedicatedCircuitLink -ServiceKey $ServiceKey -VNetName $Vnet

So reads worked, but no modify. Ah—I remembered the ARM portal lock feature, and sure enough, a Read-Only lock on the Resource Group was inherited by the ExpressRoute (more about those here). Once the lock was removed, voila, I could remove the stubborn VNets no problem.

# Remove the Circuit Link for the Vnet

Remove-AzureDedicatedCircuitLink -ServiceKey $ServiceKey -VNetName $Vnet

# Disconnect the gateway

Set-AzureVNetGateway -Disconnect –VnetName $Vnet –LocalNetworkSiteName <LocalNetworksitename>

# Delete the gateway

Remove-AzureVNetGateway –VNetName $Vnet

There’s still no command to remove a single Vnet, you have to use the portal (either will work) or you can use PowerShell to edit the NetworkConfig.xml file, then import it.

Once our legacy VNets were cleaned up, I re-enabled the Read-Only lock on the ExpressRoute.

In summary, nothing was “broken”, just an overlooked setting. I would recommend cleaning up your ASM/Classic Vnets before migrating your ExpressRoute, it’s so much easier and cleaner, but if you must leave some legacy virtual networks in place, remember to set the ExpressRoute setting “allowclassicoperations” setting to “True” after the migration is complete.

And don’t forget those pesky ARM Resource Group locks.

Automating OMS Searches, Schedules, and Alerts in Azure Resource Manager Templates

OMS-Icon.png

Recently, I had an opportunity to produce a proof of concept around leveraging Operations Management Suite (OMS) as a centralized point of monitoring, analytics, and alerting.  As in any good Azure project, codifying your Azure resources in an Azure Resource Manager template should be table stakes. While the official ARM schema doesn't yet have all the children for the Microsoft.OperationalInsights/workspaces type, the Log Analytics documentation does document samples for interesting use cases that demonstrate data sources and saved searches.  Notably absent from this list, however, is anything about the /schedules and /action types. Now, a clever Azure Developer will realize that an ARM template is basically an orchestration and expression wrapper around the REST APIs.  In fact, I've yet to see a case where an ARM template supports a type that's not provided by those APIs.  With that knowledge and the alert API samples, we can adapt the armclient put <url> examples to an ARM template.

The Resources

What you see as an alert in the OMS portal is actually 2 to 3 distinct resources in the ARM model:

A mandatory schedules type, which defines the period and time window for the alert

{
  "name": "[concat(parameters('Workspace-Name'), '/', variables('Search-Name'), '/', variables('Schedule-Name'))]",
  "type": "Microsoft.OperationalInsights/workspaces/savedSearches/schedules",
  "apiVersion": "2015-03-20",
  "location": "[parameters('Workspace-Location')]",
  "dependsOn": [
    "[variables('Search-Id')]"
  ],
  "properties": {
    "Interval": 5,
    "QueryTimeSpan": 5,
    "Active": "true"
  }
}

A mandatory actions type, defining the trigger and email notification settings

{
  "name": "[concat(parameters('Workspace-Name'), '/', variables('Search-Name'), '/', variables('Schedule-Name'), '/', variables('Alert-Name'))]",
  "type": "Microsoft.OperationalInsights/workspaces/savedSearches/schedules/actions",
  "apiVersion": "2015-03-20",
  "location": "[parameters('Workspace-Location')]",
  "dependsOn": [
    "[variables('Schedule-Id')]"
  ],
  "properties": {
    "Type": "Alert",
    "Name": "[parameters('Search-DisplayName')]",
    "Threshold": {
      "Operator": "gt",
      "Value": 0
    },
    "Version": 1
  }
}

An optional, secondary, actions type, defining the alert's webhook settings

{
  "name": "[concat(parameters('Workspace-Name'), '/', variables('Search-Name'), '/', variables('Schedule-Name'), '/', variables('Action-Name'))]",
  "type": "Microsoft.OperationalInsights/workspaces/savedSearches/schedules/actions",
  "apiVersion": "2015-03-20",
  "location": "[parameters('Workspace-Location')]",
  "dependsOn": [
    "[variables('Schedule-Id')]"
  ],
  "properties": {
    "Type": "Webhook",
    "Name": "[variables('Action-Name')]",
    "WebhookUri": "[parameters('Action-Uri')]",
    "CustomPayload": "[parameters('Action-Payload')]",
    "Version": 1
  }
}

The Caveats

The full template adds a few eccentricities that are necessary for fully functioning resources.

First, let's talk about the search.  At the OMS interface, when you create a search, it is named <Category>|<Name> behind the scenes.  It is also lower-cased.  Now, the APIs will not downcase any name you send in, but if you use the interface to update that API-deployed search, you will duplicate it rather than being prompted to overwrite.  This tells us that for whatever reason, OMS' identifiers are case-sensitive.  This is why the template accepts a search category and name in parameters, then use variables to lower them and match the interface's naming convention.

Second, a schedule's name must be globally unique to your workspace, though the actions need not be.  Again, the OMS UI will take over here and generate GUIDs for the resources behind the scenes.  Unfortunately for us, ARM Templates don't support random number or GUID generation - a necessary evil of being deterministic.  While the template could accept a GUID by parameter, I find that asking my users for GUIDs is impolite - a uniquestring() function on the already-unique search name should suffice for most use cases.  Since the actions don't require uniqueness, static names will do.

Lastly, this template is not idempotent.  Idempotency is a characteristic of a desired state technology that focuses on ensuring the state (in our case, the template), rather than prescribing process (create this, update that, delete the other thing).  Each of Azure's types is responsible for ensuring that they implement idempotency as much as possible.  For some resources like the database extensions type, which imports .bacpac files, the entire type opts out of idempotency (once data is in the database, no more importing).  For others (such as Azure VMs), only changes in certain properties like administrator name/password break idempotency.  In the OMS search and schedule's case, if either one exists when you deploy the template, the deployment will fail with a code of BadRequest.  To add insult to injury, deleting the saved search from the interface does not delete the schedule.  Neither does deleting the alert from the UI - it simply flips the scheduled to "Enabled": false.  There is, in fact, no way to delete the schedule via the UI - you must do so via the REST API, similar to the following:

armclient delete /{Schedule ResourceID}?api-version=2015-03-20

This means that, for now, any versioned solution which desires to deploy ARM template updates to its searches and alerts will need to either:

  1. Delete the existing search/schedule, then redeploy OR
  2. Deploy the updated version side-by-side (named differently), then clean up the prior version's items.

Luckily ours was a POC, so deleting and re-creating was very much an option.  I look forward to days when search and alerting types become first-class, idempotent citizens.  Until then, I hope this approach proves useful!

The Template

The template below deploys a search, a schedule, a threshold action (the alert), and a webhook action (the action) that includes a custom JSON payload (remember to escape your JSON!).  With a little work, this template could be reworked to segregate the alert-specific components from the search so that multiple alerts could be deployed to the same search or adapted for use with Azure Automation Runbooks.  Enjoy!

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "Action-Payload": {
      "type": "string"
    },
    "Action-Uri": {
      "type": "string"
    },
    "Alert-DisplayName": {
      "type": "string"
    },
    "Search-Category": {
      "type": "string"
    },
    "Search-DisplayName": {
      "type": "string"
    },
    "Search-Query": {
      "type": "string"
    },
    "Workspace-Name": {
      "type": "string"
    },
    "Workspace-Location": {
      "type": "string"
    }
  },
  "variables": {
    "Action-Name": "webhook",
    "Alert-Name": "alert",
    "Schedule-Name": "[uniquestring(variables('Search-Name'))]",
    "Search-Name": "[toLower(concat(variables('Search-Category'), '|', parameters('Search-DisplayName')))]",
    "Search-Id": "[resourceId('Microsoft.OperationalInsights/workspaces/savedSearches', parameters('Workspace-Name'), variables('Search-Name'))]",
    "Schedule-Id": "[resourceId('Microsoft.OperationalInsights/workspaces/savedSearches/schedules', parameters('Workspace-Name'), variables('Search-Name'), variables('Schedule-Name'))]",
    "Alert-Id": "[resourceId('Microsoft.OperationalInsights/workspaces/savedSearches/schedules/actions', parameters('Workspace-Name'), variables('Search-Name'), variables('Schedule-Name'), variables('Alert-Name'))]",
    "Action-Id": "[resourceId('Microsoft.OperationalInsights/workspaces/savedSearches/schedules/actions', parameters('Workspace-Name'), variables('Search-Name'), variables('Schedule-Name'), variables('Action-Name'))]"
  },
  "resources": [
    {
      "name": "[concat(parameters('Workspace-Name'), '/', variables('Search-Name'))]",
      "type": "Microsoft.OperationalInsights/workspaces/savedSearches",
      "apiVersion": "2015-03-20",
      "location": "[parameters('Workspace-Location')]",
      "dependsOn": [],
      "properties": {
        "Category": "[parameters('Search-Category')]",
        "DisplayName": "[parameters('Search-DisplayName')]",
        "Query": "[parameters('Search-Query')]",
        "Version": "1"
      }
    },
    {
      "name": "[concat(parameters('Workspace-Name'), '/', variables('Search-Name'), '/', variables('Schedule-Name'))]",
      "type": "Microsoft.OperationalInsights/workspaces/savedSearches/schedules",
      "apiVersion": "2015-03-20",
      "location": "[parameters('Workspace-Location')]",
      "dependsOn": [
        "[variables('Search-Id')]"
      ],
      "properties": {
        "Interval": 5,
        "QueryTimeSpan": 5,
        "Active": "true"
      }
    },
    {
      "name": "[concat(parameters('Workspace-Name'), '/', variables('Search-Name'), '/', variables('Schedule-Name'), '/', variables('Alert-Name'))]",
      "type": "Microsoft.OperationalInsights/workspaces/savedSearches/schedules/actions",
      "apiVersion": "2015-03-20",
      "location": "[parameters('Workspace-Location')]",
      "dependsOn": [
        "[variables('Schedule-Id')]"
      ],
      "properties": {
        "Type": "Alert",
        "Name": "[parameters('Search-DisplayName')]",
        "Threshold": {
          "Operator": "gt",
          "Value": 0
        },
        "Version": 1
      }
    },
    {
      "name": "[concat(parameters('Workspace-Name'), '/', variables('Search-Name'), '/', variables('Schedule-Name'), '/', variables('Action-Name'))]",
      "type": "Microsoft.OperationalInsights/workspaces/savedSearches/schedules/actions",
      "apiVersion": "2015-03-20",
      "location": "[parameters('Workspace-Location')]",
      "dependsOn": [
        "[variables('Schedule-Id')]"
      ],
      "properties": {
        "Type": "Webhook",
        "Name": "[variables('Action-Name')]",
        "WebhookUri": "[parameters('Action-Uri')]",
        "CustomPayload": "[parameters('Action-Payload')]",
        "Version": 1
      }
    }
  ],
  "outputs": {}
}

Ruling Your Resource Groups with an Iron Fist

grim-reaper4.png

If it is not obvious by now, we deploy a lot of resources to Azure.  The rather expensive issue we encounter is that people rarely remember to clean up after themselves; it goes without saying we have encountered some staggeringly large bills.  To remedy this, we enforced a simple, yet draconian policy around persisting Resource Groups.  If your Resource Group does not possess values for our required tag set (Owner and Solution), it can be deleted at any time. At the time the tagging edict went out we had well over 1000 resource groups.  As our lab manager began to script the removal using the Azure Cmdlets we encountered a new issue, the synchronous operations of the Cmdlets just took too long.  We now use a little script we call "The Reaper" to "fire and forget".

If you read my previous post about Azure AD and Powershell, you may have noticed my predilection for doing things via the REST API.  "The Reaper" simply uses the module from that post to obtain an authorization token and uses the REST API to evaluate Resource Groups for tag compliance and delete the offenders asynchronously. There are a few caveats to note; it will only work with organizational accounts and it will not delete any resources which have a lock.

It is published on the PowerShell gallery, so if you can obtain it like so:

[powershell] #Just download it Save-Script -Name thereaper -Path "C:\myscripts" #Install the script (with the module dependency) Install-Script -Name thereaper [/powershell]

The required parameters are a PSCredential and an array of strings for the tags to inspect. Notable Switch parameters are AllowEmptyTags (only checks for presence of the required tags) and DeleteEmpty (removes Resource Groups with no Resources even if tagged properly). There is also a SubscriptionFilters parameter taking an array of Subscription id's to limit the scope, otherwise all Subscriptions your account has access to will be evaluated. If you would simply like to see what the results would be, use the WhatIf Switch. Usage is as follows:

[powershell] $Credential=New-Object PSCredential("username@tenant.com",("YourPassword"|ConvertTo-SecureString -AsPlainText -Force)) $results=.\thereaper.ps1 -Credential $Credential ` -RequiredTags "Owner","Solution" -DeleteEmpty ` -SubscriptionFilters "49f4ba3e-72ec-4621-8e9e-89d312eafd1f","554503f6-a3aa-4b7a-a5a9-641ac65bf746" [/powershell]

A standard liability waiver applies; I hold no responsibility for the Resource Groups you destroy.

Azure, Azure Active Directory, and PowerShell. The Hard Way

poshoauth.png

In my opinion, a fundamental shift for Windows IT professionals occurred with the release of Exchange 2007.  This established PowerShell as the tool for managing and configuring Microsoft enterprise products and systems going forward.  I seem to remember hearing a story at the time that a mandate was established for every enterprisey product going forward; each GUI action would have a corresponding PowerShell execution.  If anyone remembers the Exchange 2007 console, you could see that in action.  I won’t bother corroborating this story, because the end results are self-evident.  I can’t stress how important this was.  Engineers and administrators with development and advanced scripting skills were spared the further indignity of committing crimes against Win32 and COM+ across a hodgepodge of usually awful languages.  Windows administrators for whom automation and scripting only meant batch files, a clear path forward was presented.

PowerShell and Leaky Abstractions

For roughly two years now, the scope of my work has been mostly comprised of Azure integration and automation.  Azure proved to be no exception to the PowerShell new world order. I entered with wide-eyed optimism and I quickly discovered a great deal of things, usually of a more advanced nature, that could not be done in the portal and purportedly only via PowerShell. As I continue to receive product briefings, I have developed a bit of a pedantic pet-peeve.  PowerShell is always front and center in the presentations when referencing management, configuration, and automation.  However, I continue to see a general hand wave given as to the underlying technologies (e.g. WMI/CIM, REST API) and requirements.  I absolutely understand the intent, PowerShell has always been meant to provide a truly powerful environment in a manner that was highly accessible and friendly to the IT professional.  It has been a resounding success in that regard.  A general concern, I have, is that of too much abstraction.  There is a direct correlation between your frustration level and how far your understanding of what is going on is when an inevitable edge case is hit and the abstraction leaks.

Getting Back to the Point

All of that is a really long preface to the actual point of this post. I’ve never been a fan of the Azure Cmdlets for a number of reasons, most of which I don’t necessarily impugn the decisions made by Microsoft. To be honest, I think  both Switch-AzureMode (for those that remember) and the rapid release cadence that has introduced many understandably unavoidable breaking changes has really prejudiced me; as a result I tend to use the REST API almost exclusively. The fact is, modern systems and especially all of the micro-service architectures being touted are all powered by REST API. In the case of the Microsoft cloud, with only a few notable exceptions, authentication and authorization is handled via Azure Active Directory. It behooves the engineer or developer focused on Microsoft technologies to have a cursory understanding.  Azure Active Directory, Azure, and Office 365 are intrinsically linked. Every Azure and/or Office 365 Subscription is linked with an Azure AD tenant as the primary identity provider. The modern web seems to have adopted OAuth as an authorization standard and Azure AD can greatly streamline the authorization of web applications and API. The management and other API surfaces of Azure (and Azure Stack) and Office 365 have always taken advantage of this. The term you’ve likely heard thrown around is Bearer Token. That is more accurately described as an authorization header on the HTTP request containing a JWT (JSON Web Token).  My largest issue with the Azure and PowerShell automation has been the necessity to jump through hoops to simply obtain that token via PowerShell.  In 2016 a somewhat disingenuously Cmdlet named Get-AzureStackToken in the AzureRM.AzureStackAdmin module finally appeared.  I’m certain a large portion of the potential reading audience has used a tool like Fiddler, Postman, or even more recently resources.azure.com to either inspect or interact with these services.  Those who have can feel free to skip the straight to where this applies to PowerShell.

There are two types of applications you can create within Azure AD, each of with are identified with a unique Client Id and valid redirect URI(s) as the most relevant properties we’ll focus on.

Web Applications

  • Web applications in Azure Active Directory are OAuth2 confidential clients and likely the most appropriate option for modern (web) use cases.

  • Tokens are obtained on behalf of a user using the OAuth2 authorization grant flow. An authorization code or id token will be supplied to the specified redirect URI.

  • If needed, client credentials (a rolling secret key) can be used to obtain tokens on behalf of the user or on it’s own from the web application itself.

Native Applications

  • Native applications in Azure Active Directory are OAuth2 public clients (e.g. an application on a desktop or mobile device).

  • These applications can obtain a token directly (with managed organizational accounts) or use the authorization grant flow, but application level permissions are not applicable.

Getting to the PowerShell

I will focus primarily on the Native application type as it is most relevant to PowerShell. Most of the content will use Cmdlets from a module that will be available with this post.   The module is heavily derived/inspired by the ADAL libraries, has no external dependencies and accept a friendly PSCredential (with the appropriate rights) for any user authentication.  The Azure Cmdlets use a Native application with a Client Id of 1950a258-227b-4e31-a9cf-717495945fc2 and a redirect URI of urn:ietf:wg:oauth:2.0:oob (the prescribed default for native applications).   We’ll use this for our first attempt at obtaining a token for use against Azure Resource Manager or the legacy Service Management API.  A peculiar detail of Azure management is that this one of the scenarios a token is fungible for disparate endpoints. I always use https://management.core.windows.net as my audience regardless of whether I will be working with ARM or SM.  A token obtained from that audience will work the same as one from https://management.azure.com .

If all you would like is a snippet to obtain a token using the Azure, I’ll offer you a chance to bail out now:


$Resource='https://management.core.windows.net'
$PoshClientId="1950a258-227b-4e31-a9cf-717495945fc2"
$TenantId="yourdomain.com"
$UserName="username@$TenantId"
$Password="asecurepassword"|ConvertTo-SecureString -AsPlainText -Force
$Credential=New-Object pscredential($UserName,$Password)
Get-AzureStackToken -Resource $Resource -AadTenantId $TenantId -ClientId $PoshClientId -Credential $Credential -Authority "https://login.microsoftonline.com/$TenantId" 

A good deal of the functionality around provisioning applications and service principals has come to the Azure Cmdlets.  You can now create applications, service principals from the applications, and role assignments to the service principals. To create an application, in this case one that would own a subscription, you would write something like this:


$ApplicationSecret="ASuperSecretPassword!"
$TenantId='e05b8b95-8c85-49af-9867-f8ac0a257778'
$SubscriptionId='bc3661fe-08f5-4b87-8529-9190f94c163e'
$AppDisplayName='The Subscription Owning Web App'
$HomePage='https://azurefieldnotes.com'
$IdentifierUris=@('https://whereeveryouwant.com')
$NewWebApp=New-AzureRmADApplication -DisplayName $AppDisplayName -HomePage $HomePage `
    -IdentifierUris $IdentifierUris -StartDate (Get-Date) -EndDate (Get-Date).AddYears(1) `
    -Password $ApplicationSecret
$WebAppServicePrincipal=New-AzureRmADServicePrincipal -ApplicationId $NewWebApp.ApplicationId
$NewRoleAssignment=New-AzureRmRoleAssignment -ObjectId $NewWebApp.Id -RoleDefinitionName 'owner' -Scope "/subscriptions/$SubscriptionId"
$ServicePrincipalCred=New-Object PScredential($NewWebApp.ApplicationId,($ApplicationSecret|ConvertTo-SecureString -AsPlainText -Force))
Add-AzureRmAccount -Credential $ServicePrincipalCred -TenantId $TenantId -ServicePrincipal 

For those that stuck around, let’s take a look at obtaining JWT(s), inspecting them, and putting them to use.

I added a method for decoding the tokens, so we will have a look at the access token.  A JWT is comprised of a header, payload, and signature.  I will leave explaining the claims within the payload to identity experts.

Now that we have a token, let's use it for something useful, in this case we will ask Azure (ARM) for our associated subscriptions.

Examining the OAuth2 Flow

If you are not interested in what is going on behind the scenes feel free to skip ahead.  Each application exposes a standard set of endpoints and I will not discuss the v2.0 endpoint as I do not have enough experience using it.  There are two endpoints in particular to make note of, https://login.microsoftonline.com/{tenantid}/oauth2/authorize and https://login.microsoftonline.com/{tenantid}/oauth2/token, where {tenantid} represents the tenant id (guid or domain name) e.g. yourcompany.com or common for multi-tenant applications.  Azure AD obviously supports federation and the directing traffic to the appropriate authorization endpoint is guided by a user realm detection API of various versions at https://login.microsoftonline.com/common/UserRealm.  If we inspect the result for a fully managed Azure AD account we see general tenant detail.

If we take a look at a federated user we will see a little difference, the AuthURL property.

userrealm federated

This show us the location of our federated authentication endpoint. The token will actually be requested via a SAML user assertion that is received from an STS, in this case ADFS.

The OAuth specification uses the request parameter collection for token and authorization code responses. A username and password combination can be used to directly request a token in the fully managed scenario public client scenario.

A POST request can go directly to the Token endpoint with the following query parameters:

client_id

The Application Id

resource

The Resource URI to access

grant_type

password

username

The username

password

The password

The ADFS/WSTrust will entail sending a SOAP request to the WSTrust endpoint to authenticate and use that response to create the assertion that is exchanged for an access token.  Through user realm detection we can find the ADFS username/password endpoint.  A SOAP envelope can be sent to  endpoint to receive a security token response, containing the assertions needed.

A POST request is sent to the Username/Password endpoint for ADFS with the following envelope with noteable values encased in {}:

<s:Envelope xmlns:s='http://www.w3.org/2003/05/soap-envelope' 
    xmlns:a='http://www.w3.org/2005/08/addressing' 
    xmlns:u='http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd'>
    <s:Header>
        <a:Action s:mustUnderstand='1'>http://docs.oasis-open.org/ws-sx/ws-trust/200512/RST/Issue</a:Action>
        <a:messageID>urn:uuid:{Unique Identifier for the Request}</a:messageID>
        <a:ReplyTo>
            <a:Address>http://www.w3.org/2005/08/addressing/anonymous</a:Address>
        </a:ReplyTo>        <!-- The Username Password WSTrust Endpoint -->
        <a:To s:mustUnderstand='1'>{Username/Password Uri}</a:To>
        <o:Security s:mustUnderstand='1' 
            xmlns:o='http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd'>            <!-- The token length requested -->
            <u:Timestamp u:Id='_0'>
                <u:Created>{Token Start Time}</u:Created>
                <u:Expires>{Token Expiry Time}</u:Expires>
            </u:Timestamp>            <!-- The username and password used -->
            <o:UsernameToken u:Id='uuid-{Unique Identifier for the Request}'>
                <o:Username>{UserName to Authenticate}</o:Username>
                <o:Password>{Password to Authenticate}</o:Password>
            </o:UsernameToken>
        </o:Security>
    </s:Header>
    <s:Body>
        <trust:RequestSecurityToken xmlns:trust='http://docs.oasis-open.org/ws-sx/ws-trust/200512'>
            <wsp:AppliesTo xmlns:wsp='http://schemas.xmlsoap.org/ws/2004/09/policy'>
                <a:EndpointReference>
                    <a:Address>urn:federation:MicrosoftOnline</a:Address>
                </a:EndpointReference>
            </wsp:AppliesTo>
            <trust:KeyType>http://docs.oasis-open.org/ws-sx/ws-trust/200512/Bearer</trust:KeyType>
            <trust:RequestType>http://docs.oasis-open.org/ws-sx/ws-trust/200512/Issue</trust:RequestType>
        </trust:RequestSecurityToken>
    </s:Body>
</s:Envelope>

The token response is inspected for SAML assertion types (urn:oasis:names:tc:SAML:1.0:assertion or urn:oasis:names:tc:SAML:2.0:assertion) to find the matching token used for the OAuth token request.

<s:Envelope xmlns:s="http://www.w3.org/2003/05/soap-envelope" 
    xmlns:a="http://www.w3.org/2005/08/addressing" 
    xmlns:u="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">
    <s:Header>
        <a:Action s:mustUnderstand="1">http://docs.oasis-open.org/ws-sx/ws-trust/200512/RSTRC/IssueFinal</a:Action>
        <o:Security s:mustUnderstand="1" 
            xmlns:o="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd">
            <u:Timestamp u:Id="_0">
                <u:Created>2016-01-03T01:34:41.640Z</u:Created>
                <u:Expires>2016-01-03T01:39:41.640Z</u:Expires>
            </u:Timestamp>
        </o:Security>
    </s:Header>
    <s:Body>
        <trust:RequestSecurityTokenResponseCollection xmlns:trust="http://docs.oasis-open.org/ws-sx/ws-trust/200512">            <!-- Our Desired Token Response -->
            <trust:RequestSecurityTokenResponse>
                <trust:Lifetime>
                    <wsu:Created xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">2016-01-03T01:34:41.622Z</wsu:Created>
                    <wsu:Expires xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd">2016-01-03T02:34:41.622Z</wsu:Expires>
                </trust:Lifetime>
                <wsp:AppliesTo xmlns:wsp="http://schemas.xmlsoap.org/ws/2004/09/policy">
                    <wsa:EndpointReference xmlns:wsa="http://www.w3.org/2005/08/addressing">
                        <wsa:Address>urn:federation:MicrosoftOnline</wsa:Address>
                    </wsa:EndpointReference>
                </wsp:AppliesTo>
                <trust:RequestedSecurityToken>                    <!-- The Assertion -->
                    <saml:Assertion MajorVersion="1" MinorVersion="1" AssertionID="_e3b09f2a-8b57-4350-b1e1-20a8f07b3d3b" Issuer="http://adfs.howtopimpacloud.com/adfs/services/trust" IssueInstant="2016-08-03T01:34:41.640Z" 
                        xmlns:saml="urn:oasis:names:tc:SAML:1.0:assertion">
                        <saml:Conditions NotBefore="2016-01-03T01:34:41.622Z" NotOnOrAfter="2016-01-03T02:34:41.622Z">
                            <saml:AudienceRestrictionCondition>
                                <saml:Audience>urn:federation:MicrosoftOnline</saml:Audience>
                            </saml:AudienceRestrictionCondition>
                        </saml:Conditions>
                        <saml:AttributeStatement>
                            <saml:Subject>
                                <saml:NameIdentifier Format="urn:oasis:names:tc:SAML:1.1:nameid-format:unspecified">130WEAH65kG8zfGrZFNlBQ==</saml:NameIdentifier>
                                <saml:SubjectConfirmation>
                                    <saml:ConfirmationMethod>urn:oasis:names:tc:SAML:1.0:cm:bearer</saml:ConfirmationMethod>
                                </saml:SubjectConfirmation>
                            </saml:Subject>
                            <saml:Attribute AttributeName="UPN" AttributeNamespace="http://schemas.xmlsoap.org/claims">
                                <saml:AttributeValue>chris@howtopimpacloud.com</saml:AttributeValue>
                            </saml:Attribute>
                            <saml:Attribute AttributeName="ImmutableID" AttributeNamespace="http://schemas.microsoft.com/LiveID/Federation/2008/05">
                                <saml:AttributeValue>130WEAH65kG8zfGrZEFlBQ==</saml:AttributeValue>
                            </saml:Attribute>
                        </saml:AttributeStatement>
                        <saml:AuthenticationStatement AuthenticationMethod="urn:oasis:names:tc:SAML:1.0:am:password" AuthenticationInstant="2016-08-03T01:34:41.607Z">
                            <saml:Subject>
                                <saml:NameIdentifier Format="urn:oasis:names:tc:SAML:1.1:nameid-format:unspecified">130WEAH65kG8sfGrZENlBQ==</saml:NameIdentifier>
                                <saml:SubjectConfirmation>
                                    <saml:ConfirmationMethod>urn:oasis:names:tc:SAML:1.0:cm:bearer</saml:ConfirmationMethod>
                                </saml:SubjectConfirmation>
                            </saml:Subject>
                        </saml:AuthenticationStatement>
                        <ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#">
                            <ds:SignedInfo>
                                <ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#" />
                                <ds:SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1" />
                                <ds:Reference URI="#_e3b09f2a-8b57-4350-b1e1-20a8f07b3d3b">
                                    <ds:Transforms>
                                        <ds:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature" />
                                        <ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#" />
                                    </ds:Transforms>
                                    <ds:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1" />
                                    <ds:DigestValue>itvzbQhlzA8CIZsMneHVR15FJlY=</ds:DigestValue>
                                </ds:Reference>
                            </ds:SignedInfo>
                            <ds:SignatureValue>gBCGUmhQrJxVpCxVsy2L1qh1kMklVVMoILvYJ5a8NOlezNUx3JNlEP7wZ389uxumP3sL7waKYfNUyVjmEpPkpqxdxrxVu5h1BDBK9WqzOICnFkt6JPx42+cyAhj3T7Nudeg8CP5A9ewRCLZu2jVd/JEHXQ8TvELH56oD5RUldzm0seb8ruxbaMKDjYFuE7X9U5sCMMuglU3WZDC3v6aqmUxpSd9Kelhddleu33XEBv7CQNw84JCud3B+CC7dUwtGxwv11Mk/P0t1fGbfs+I6aSMTecKq9YmscqP9tB8ZouD42jhjhYysOQSdulStmUi6gVzQz+c2l2taa5Amd+JCPg==</ds:SignatureValue>
                            <KeyInfo xmlns="http://www.w3.org/2000/09/xmldsig#">
                                <X509Data>
                                    <X509Certificate>MIIC4DCDAcigAwIBAgIQaYQ6QyYqcrBBmOHSGy0E1DANBgkqhkiG9w0BAQsFADArMSkwJwYDVQQDEyBBREZTIFNpZ25pbmcgLSBhZGZzLmNpLmF2YWhjLmNvbTAgFw0xNjA2MDQwNjA4MDdaGA8yMTE2MDUxMTA2MDgwN1owKzEpMCcGA1UEAxMgQURGUyBTaWduaW5nIC0gYWRmcy5jaS5hdmFoYy5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDH9J6/oWYAR8Y98QnacNouKyIBdtZbosEz0HyJVyrxVqKq2AsPvCEO3WFm9Gmt/xQN9PuLidZpgICAe8Ukuv4h/NldgmgtD64mObFNuEM5pzAPRXUv6FWlVE4fnUpIiD1gC0bbQ7Tzv/cVgfUChCDpFu3ePDTs/tv07ee22jXtoyT3N7tsbIX47xBMKgF9ItN9Oyqi0JyQHZghVQ1ebNOMH3/zNdl0WcZ+Pl+osD3iufoH6H+qC9XY09B5YOWy8fJoqf+HFeSWZCHH5vJJfsPTsSilvLHCpMGlrMFaTBKqmv+m9Z3FtbzOcnKHS5PJVAymqLctkH+HbFzaDblaSRhhAgMBAAEwDQYJKoZIhvcNAQELBQADggEBAFB0E2Cj+O24aPM61JsCXLIAB28q4h4qLxMwV+ypYjFxxcQ5GzgqaPJ7BARCnW1gm3PyvNfUut9RYrT9wTJlBVY9WDBoX33jsS87riMj+JONXJ7lG/zAozxs0xIiW+PNlFdOt7xyvYstrFgPJS1E05jhiZ2PR8MS20uSlMNkVPinpz4seyyMQeM+1GbpbDE1EwwtEVKgatJN7t6nAn9mw8cHIk1et7CYOGeWCnMA9EljzNiD8wEwsG51aKfuvGrPK8Q8N/G89SPgstpe0Te5+EtWT6latXfpCwdNWxvinH49SKKa25l1VoLLNwKiQF6vK1Iw0F7dP7QkO5YdE7/MTDU=</X509Certificate>
                                </X509Data>
                            </KeyInfo>
                        </ds:Signature>
                    </saml:Assertion>
                </trust:RequestedSecurityToken>
                <trust:RequestedAttachedReference>
                    <o:SecurityTokenReference k:TokenType="http://docs.oasis-open.org/wss/oasis-wss-saml-token-profile-1.1#SAMLV1.1" 
                        xmlns:o="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd" 
                        xmlns:k="http://docs.oasis-open.org/wss/oasis-wss-wssecurity-secext-1.1.xsd">
                        <o:KeyIdentifier ValueType="http://docs.oasis-open.org/wss/oasis-wss-saml-token-profile-1.0#SAMLAssertionID">_e3b09f2a-8b57-4350-b1e1-20a8f07b3d3b</o:KeyIdentifier>
                    </o:SecurityTokenReference>
                </trust:RequestedAttachedReference>
                <trust:RequestedUnattachedReference>
                    <o:SecurityTokenReference k:TokenType="http://docs.oasis-open.org/wss/oasis-wss-saml-token-profile-1.1#SAMLV1.1" 
                        xmlns:o="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd" 
                        xmlns:k="http://docs.oasis-open.org/wss/oasis-wss-wssecurity-secext-1.1.xsd">
                        <o:KeyIdentifier ValueType="http://docs.oasis-open.org/wss/oasis-wss-saml-token-profile-1.0#SAMLAssertionID">_e3b09f2a-8b57-4350-b1e1-20a8f07b3d3b</o:KeyIdentifier>
                    </o:SecurityTokenReference>
                </trust:RequestedUnattachedReference>
                <trust:TokenType>urn:oasis:names:tc:SAML:1.0:assertion</trust:TokenType>
                <trust:RequestType>http://docs.oasis-open.org/ws-sx/ws-trust/200512/Issue</trust:RequestType>
                <trust:KeyType>http://docs.oasis-open.org/ws-sx/ws-trust/200512/Bearer</trust:KeyType>
            </trust:RequestSecurityTokenResponse>
        </trust:RequestSecurityTokenResponseCollection>
    </s:Body>
</s:Envelope>

A POST request is sent to the Token endpoint with the following query parameters:

client_id

The Application Id

resource

The Resource URI to access

assertion

The base64 encoded SAML token

grant_type

urn:ietf:params:oauth:grant-type:saml1_1-bearer urn:ietf:params:oauth:grant-type:saml2-bearer

scope

openid

A GET request is sent to the Authorize endpoint with some similar query parameters:

client_id

The Application Id

redirect_uri

The location within the application to handle the authorization code

response_type

code

prompt

login consent admin_consent

scope

optional scope for access (app uri or openid scope)

The endpoint should redirect you to the appropriate login screen via user realm detection.  Once the user login is completed, the code is added to the redirect address as either query parameters (default) or a form POST.  Once the code is retrieved it can be exchanged for a token. A POST request is sent to the Token endpoint as demonstrated before with some slightly different parameters:

client_id

The Application Id

resource

The Resource URI to access

code

The authorization code

grant_type

authorization_code

scope

previous scope

client_secret

required if confidential client

Tying it All Together

To try to show some value for your reading time, lets explore how this can be used as the solutions you support and deploy become more tightly integrated with the Microsoft cloud.  We'll start by creating a new Native application in the legacy portal.

appnative1
appnative2

I used https://itdoesnotmatter here, but you might as well follow the guidance of using urn:ietf:wg:oauth:2.0:oob.  We will now grant permissions to Azure Active Directory and Azure Service Management (for ARM too).

ADPermissions
ADServiceMgmt

I will avoid discussing configuring the application to be multi-tenant as the processes I outline are identical, it is simply a matter of the targeted tenant.  You should end up with something looking like this.

Native

Let's now try to go get a token for our new application and put it to use.  This should look exactly the same as retrieving the previous token.


$AuthCode=Approve-AzureADApplication -ClientId $NewClientId -RedirectUri 'https://itdoesnotmatter/' -TenantId sendthewolf.com -AdminConsent

nativefirstattempt

Epic failure!  Unfortunately we run into a common annoyance, the application must be consented to interactively.  I do not know of any tooling that exists to make this easy.  I added a function to make this a little easier and it supports a switch of AdminConsent to approve the application for all users within the tenant.  And step through the consent process to receive an authorization code.


$AuthCode=Approve-AzureADApplication -ClientId $NewClientId -RedirectUri 'https://itdoesnotmatter/' -TenantId sendthewolf.com -AdminConsent

Approve
Approve App

Once the authorization code is obtained it can be exchanged for a token, for which I provided another function.  That token can now be used in the exact same manner as the Azure Cmdlet application.


$TokenResult=Get-AzureADAccessTokenFromCode 'https://management.core.windows.net/' -ClientId $NewClientId -RedirectUri 'https://itdoesnotmatter/' -TenantId sendthewolf.com -AuthorizationCode $AuthCode

Authorize2

If you wanted to handle some Azure Active Directory objects, we can target a different audience, and execute actions appropriate to the account's privilege level.   In the following example we will create a new user.


$GraphBaseUri="https://graph.windows.net/"
$GraphUriBuilder=New-Object System.UriBuilder($GraphBaseUri)
$GraphUriBuilder.Path="$TenantId/users"
$GraphUriBuilder.Query="api-version=1.6"
$NewUserJSON=@"
{
    "accountEnabled": true, 
    "displayName": "Johnny Law", 
    "mailNickName" : "thelaw", 
    "passwordProfile": { 
        "password": "Password1234!", 
        "forceChangePasswordNextLogin": false 
    }, 
    "userPrincipalName": "johhny.law@$TenantId" 
}
"@
$AuthResult=Get-AzureADUserToken -Resource $GraphBaseUri -ClientId $NewClientId -Credential $Credential -TenantId $TenantId
$AuthHeaders=@{Authorization="Bearer $($AuthResult.access_token)"}
$NewUser=Invoke-RestMethod -Uri $GraphUriBuilder.Uri -Method Post -Headers $AuthHeaders -Body $NewUserJSON -ContentType "application/json"

If we want to continue the “fun” with Office 365 we can apply the exact sample approach with the Office 365 Sharepoint Online application permissions.  In the interest of moving along and with no regard for constraining access, we will configure the permissions in the following manner.

sharepoint

We’ll now do some querying of the Office 365 SharePoint video API with some more script.


$SharepointUri='https://yourdomain.sharepoint.com/'
$SpUriBuilder=New-Object System.UriBuilder($SharepointUri)
$SpUriBuilder.Path="_api/VideoService.Discover"
$AuthResult=Get-AzureADUserToken -Resource $SharepointUri -ClientId $NewClientId -Credential $Credential
$Headers=@{Authorization="Bearer $($AuthResult.access_token)";Accept="application/json";}
$VideoDisco=Invoke-RestMethod -Uri $SpUriBuilder.Uri -Headers $Headers $VideoDisco|Format-List
$VideoChannelId="306488ae-5562-4d3e-a19f-fdb367928b96"
$VideoPortalUrl=$VideoDisco.VideoPortalUrl
$ChannelUrlBuilder=New-Object System.UriBuilder($VideoPortalUrl)
$ChannelUrlBuilder.Path+="/_api/VideoService/Channels"
$ChannelOData=Invoke-RestMethod -Uri $ChannelUrlBuilder.Uri -Headers $Headers
$ChannelRoot=$ChannelUrlBuilder.Path
foreach ($Channel in $ChannelOData.Value)
{  
    $VideoUriBuilder=New-Object System.UriBuilder($Channel.'odata.id')
    $VideoUriBuilder.Path+="/Videos"
    Invoke-RestMethod -Uri $VideoUriBuilder.Uri -Headers $Headers|Select-Object -ExpandProperty value
}

We should see some output that looks like this:

spvideos

I’ve had Enough! Please Just Show me the Code.

For those who have endured or even skipped straight here, I present the following module for any use your dare apply.  The standard liability waiver applies and it is presented primarily for educational purposes.  It came from a need to access the assortment of Microsoft cloud API in environments where we could not always ensure the plethora of correct Cmdlets are installed.  Initially, being a .Net guy, I just wrapped standard use cases around ADAL .Net.  I really wanted to make sure that I really understood OAuth and OpenId Connect authorization flows as is relates to Azure Active Directory.  The entire theme of this lengthy tome is to emphasize the importance of having a relatively advanced understanding of these concepts.  Regardless of your milieu, if it has a significant Microsoft component, the demand to both integrate and support the integration(s) of numerous offerings will only grow larger.  The module is primarily targeted at the Native Client application type, however there is support for the client secret and implicit authorization flows.  There are also a few utility methods that are exposed as they may have some diagnostic use or otherwise.  The module exposes the following methods all of which support Get-Help:

  • Approve-AzureADApplication

    • Approves an Azure AD Application Interactively and returns the Authorization Code

    • ConvertFrom-EncodedJWT

      • Converts an encoded JWT to an object representation

      • Get-AzureADAccessTokenFromCode

        • Retrieves an access token from a consent authorization code

        • Get-AzureADClientToken

          • Retrieves an access token as a an OAuth confidential client

          • Get-AzureADUserToken

            • Retrieves an access token as a an OAuth public client

            • Get-AzureADImplicitFlowToken

              • Retrieves an access token interactively for a web application with OAuth implicit flow enabled

              • Get-AzureADOpenIdConfiguration

                • Retrieves the OpenId connect configuration for the specified application

                • Get-AzureADUserRealm

                  • Retrieves a the aggregate user realm data for the specified user principal name(s

                  • Get-WSTrustUserRealmDetails

                    • Retrieves the WSFederation details for a given user prinicpal name

Get it here: Azure AD Module

I hope you find it useful and remember not to fear doing things the hard way every so often.