Break an Infinite Lease on an Azure Storage Account Container
On rare occasions you may find yourself with an empty storage account container that has an infinite lease on it. This can make it near impossible to remove the container without opening a support case with Microsoft. While there are several tools and script samples available to break the lease on a blob there are few that address a lease on a container. The following should help identify if your container has an “infinite” lease and how to remove the lease.
Symptoms:
The following are symptoms of an infinite lease on a empty container:
- When viewing the properties of the container through the portal (https://portal.azure.com) you will see the lease period set to “infinite”.
- When deleting the Azure storage container though the new portal (https://portal.azure.com) you get the following error:
- “Failed to delete storage container 'MyLeasedContainer'. Error: There is currently a lease on the container and no lease ID was specified in the request.”"
- When deleting the Azure storage container though the old portal(https://manage.windowsazure.com) you get the following error:
- “There is currently a lease on the container and no lease ID was specified in the request. RequestId:5f0e6c7c-0001-00ec-7df9-0fc697000000 Time:2016-09-16T09:04:45.6778058Z”
Resolution:
The following script sample can help break this lease. Replace the property values of $StorageAccountName, $StorageAccountKey, and $ContainerName with the correct values of the container with the infinite lease.
[powershell]
$StorageAccountName = "MyStorageAccount"
$StorageAccountKey = "flmuQKvp7L6G8ga0IXLMnPJogSacsFPofKBv1g2sJhfcecVyrWtu+AMweHNwgSk6dAkOWTGx1viyQp/0aMRkBw=="
$ContainerName = "MyLeasedContainer"
$Creds = New-Object Microsoft.WindowsAzure.Storage.Auth.StorageCredentials("$StorageAccountName","$StorageAccountKey")
$CloudStorageAccount = New-Object Microsoft.WindowsAzure.Storage.CloudStorageAccount($creds, $true)
$CloudBlobClient = $CloudStorageAccount.CreateCloudBlobClient()
$BlobContainer = $CloudBlobClient.GetContainerReference($ContainerName)
$BlobContainer.BreakLease($(New-TimeSpan), $null, $null, $null)
[/powershell]
Network Security Group Rule Tags Deep Dive
Network Security Groups are a simple yet powerful tool within the Azure networking toolkit. They are an Azure resources that provides the ability to apply an Access Control List (ACL) ruleset to allow or deny network traffic to a VM. NSGs can be associated with either a subnet within a Virtual Network, or to a VM’s NIC (ARM). Think of them as a lightweight firewall. There is already some excellent documentation available on what they are (https://azure.microsoft.com/en-us/documentation/articles/virtual-networks-nsg/) and best practices on how to implement them (https://blogs.msdn.microsoft.com/igorpag/2016/05/14/azure-network-security-groups-nsg-best-practices-and-lessons-learned/) but for this post I wanted to focus on a particular part of the NSG rule set, the ‘tags’ functionality. Note, this is separate from the resource tags functionality in Resource Manager.
NSG Rule Tags: What are they?
Azure offers three ‘tags’ that can be used as a source or destination within a NSG rule. They are as follows:
- AzureLoadBalancer
- Internet
- VirtualNetwork
All three of these tags are utilised in the Default Rules created with any new Network Security Group resource:
Inbound Default Rules
Outbound Default Rules
They also can be utilised for any custom rules you wish to add to a NSG Ruleset.
Diving Deeper
Recently Microsoft has released the Diagnostics feature for Network Security Groups. This feature allows us to see Events and Rule Counts on each of the rules within the NSG. It also, however, allows us to dig a bit deeper into what these tags are and how traffic is handled by the default rules. (For how to enable Diagnostics logs see here.)
AzureLoadBalancer
Let’s start with the ‘AzureLoadBalancer’ tag. MS describes this tag as "denotes Azure’s Infrastructure load balancer. This will translate to an Azure datacenter IP where Azure’s health probes originate." Looking into the diagnostics logs we can see that this tag actually translates to the following CIDR block: 168.63.129.16/32.
This IP Address is a special IP Address within Azure and is the same for every Azure region. It maps to the physical IP address of the server machine (host node) hosting the virtual machine. Although they label as Load Balancer, it’s actually the same IP address that is used for the Azure Load Balancer health probe, DHCP, DNS and the virtual machine health probe. Any communication from this IP address should be considered as trusted and if blocked it will cause the Azure virtual machine to become unresponsive (i.e. never remove this rule).
Internet
Things get a little more interesting when we look at the ‘Internet’ tag. Microsoft’s documentation here says that this tag "denotes the IP address space that is outside the virtual network and reachable by public Internet. This range includes Azure owned public IP space as well."
When we look into the diagnostics logs, rules that use this tag have the following value for the IP Address array to match (Warning – long list):
1.0.0.0/8, 2.0.0.0/7, 4.0.0.0/6, 8.0.0.0/7, 11.0.0.0/8, 12.0.0.0/6, 16.0.0.0/5, 24.0.0.0/8, 26.0.0.0/7, 28.0.0.0/6, 32.0.0.0/3, 64.0.0.0/3, 96.0.0.0/6, 100.0.0.0/10, 100.128.0.0/9, 101.0.0.0/8, 102.0.0.0/7, 104.0.0.0/5, 112.0.0.0/5, 120.0.0.0/6, 124.0.0.0/7, 126.0.0.0/8, 128.0.0.0/3, 160.0.0.0/5, 168.0.0.0/8, 169.0.0.0/9, 169.128.0.0/10, 169.192.0.0/11, 169.224.0.0/12, 169.240.0.0/13, 169.248.0.0/14, 169.252.0.0/15, 169.255.0.0/16, 170.0.0.0/7, 172.0.0.0/12, 172.32.0.0/11, 172.64.0.0/10, 172.128.0.0/9, 173.0.0.0/8, 174.0.0.0/7, 176.0.0.0/4, 192.0.0.0/23, 192.0.3.0/24, 192.0.4.0/22, 192.0.8.0/21, 192.0.16.0/20, 192.0.32.0/19, 192.0.64.0/18, 192.0.128.0/17, 192.1.0.0/16, 192.2.0.0/15, 192.4.0.0/14, 192.8.0.0/13, 192.16.0.0/12, 192.32.0.0/11, 192.64.0.0/10, 192.128.0.0/11, 192.160.0.0/13, 192.169.0.0/16, 192.170.0.0/15, 192.172.0.0/14, 192.176.0.0/12, 192.192.0.0/10, 193.0.0.0/8, 194.0.0.0/7, 196.0.0.0/7, 198.0.0.0/12, 198.16.0.0/15, 198.20.0.0/14, 198.24.0.0/13, 198.32.0.0/11, 198.64.0.0/10, 198.128.0.0/9, 199.0.0.0/8, 200.0.0.0/5, 208.0.0.0/4
Now, if you are Rain Man you can see there are a few gaps in there but this is because these are reserved IP addresses (for reference the list of reserved IPv4 addresses is located here: http://www.iana.org/assignments/iana-ipv4-special-registry/iana-ipv4-special-registry.xhtml). Thus this tag is exactly what is advertised: public internet addresses.
Note: As stated in the Microsoft summary, this list does include the Azure public IP addresses. Therefore if a rule exists with the Internet tag and Deny action, it can result in some unexpected behaviour of virtual machine extensions (https://blogs.msdn.microsoft.com/mast/2016/04/27/vm-stuck-in-updating-when-nsg-rule-restricts-outbound-internet-connectivity/) unless those Azure Public IP Addresses are allowed with a lower priority.
VirtualNetwork
The 'VirtualNetwork' tag is the one that is a little misleading. Microsoft states that this tag "denotes all of your network address space. It includes the virtual network address space (CIDR ranges defined in Azure) as well as all connected on-premises address spaces and connected Azure VNets (local networks)." Now I’m not 100% sure, but I think it means that this tag should correspond to all of the routes the VNet’s gateway can see.
To prove this out, I created an NSG and attached it to a few different virtual networks. The first virtual network I attached it to was an isolated VNet with no VPN or Express Route connections (i.e. no VNet Gateway). When I retrieved the NSG diagnostic log, the list of addresses for the ‘Virtual Network’ tag was as follows: 10.1.0.0/16 and 168.63.129.16/32. The first CIDR block corresponds to the Address Space for the attached virtual network and the second is the same special IP Address as the ‘AzureLoadBalancer’ listed above.
The second virtual network I attached the NSG to was one that had a site to site connection to a VNet in a separate Azure Region. The list of addresses within the ‘Virtual Network’ tag and what they correspond to are as follows:
- 10.1.0.0/16 – Local Virtual Network Address Space
- 10.2.0.0/16 – Remote Virtual Network Address Space
- 10.2.0.0/24 – Remote Virtual Network first subnet (only subnet configured)
- 10.2.254.0/24 – Remote Virtual Network gateway subnet
- 168.63.129.16/32 – Special Azure VM Host IP Address
Now the ‘VirtualNetwork’ tag includes the complete address space of the combined Site to Site VPN (plus the subnets of the remote VNet curiously).
The third and final virtual network was one that had an Express Route circuit back to an on premise datacenter. To make things more interesting, this Express Route circuit also had a public peering connection to Azure (if you’re unsure what Peering connections are see here: https://azure.microsoft.com/en-us/documentation/articles/expressroute-circuit-peerings/) which means it also contained routes to Azure Public IPs. When I pulled the NSG log, the list of addresses to match contained a list of 400+ CIDR blocks.
On closer analysis, the list of subnets within this NSG log seemed to be much broader than just the virtual network, on premise and Azure public IPs. It turns out that although it’s broken down into 400+ subnets, it actually contains nearly every IPv4 address from 0.0.0.0 to 255.255.255.255. This is likely because the Gateway of the Virtual Network its attached to was a member of a BGP domain (as it’s an Express Route Gateway) which included advertised routes to on premise private and public subnets, Azure public subnets as well as public internet routes.
Summary
Overall, Network Security Groups are a great tool within the Azure security stack with the Rule Tags functionality allowing some extra capabilities for restricting traffic to and from your Virtual Networks and VMs, albeit with some considerations:
- The ‘AzureLoadBalancer’ tag is actually the Azure VM Host IP Address, and should always be permitted.
- The ‘Internet’ tag corresponds to the public IP address space, and should be allowed or denied accordingly.
- The ‘VirtualNetwork’ tag address space depends entirely on the VPN/Express Route configuration of your virtual network. The rules using this tag within a Network Security Group might be allowing a lot more traffic than first thought. If you want to make a rule to just allow or deny your virtual network’s address space, the best way would be to define that address space as a CIDR Block rather than use the ‘VirtualNetwork’ tag. If you want to have a rule that restricts traffic to your 'known' network then this is a good candidate.
Azure Subscription commercials and layout best practices - From the Vault: Part 1
First, whats an Azure subscription really?
An Azure subscription is the base container into which related resources (similar business and/or technical needs) are grouped together for usage and billing purposes. The key thing about a subscription is its a billing boundary in Azure. See, Microsoft rolls up all the resources in a subscription and sends a single bill for the entire subscription. There is no ability to sub define payment information below the subscription level. Additionally, a subscription is an administration boundary. A subscription co-admin can access any resource within the subscription, and then delegate that access through role based access control as needed.
Now, a little history. Originally a subscription was the only role based access control boundary. either everyone was an administrator of everything in the subscription, or they were a reader, or they had no access at all. This led to a school of thought where subscriptions were created, one for each working group/access boundary. This superuser or nothing access was one of the consistent early failures of the old Azure ASM console approach in a major enterprise. Having to receive 100 bills from 100 subscriptions and reconcile that, as well as have all or nothing access didn't sit well in the enterprise customer market. It didn't help that to separate resources between dev and production (and access rights) meant you separated subscriptions. Further, separate subscriptions meant new networking infrastructure and shared services setup. Each time. This didn't seem right. People started placing brokers and cloud management platforms in front of azure, looking for ways to automate subscription creation, tear down, and bill reconciliation. This waters down the platform to a least common denominator, and doesn't fit well with PaaS.
Fast forward a few years, and azure is picking up steam, and also has a new look and API. In the ARM console (the new portal), this was solved. Azure introduced granular RBAC into not just the console, but the underlying ARM API. The model chosen allows delegation of individual rights and permissions down to the lowest node in the tree (any ARM object) and still allows use of the full API, etc. Further, you can apply policy to any level of the ARM API (we'll cover this in a future post). This switch changes our guidance dramatically, and in fact makes it possible for us to go with a fewer subscription policy.
Less is more
Overall the number of subscriptions should be kept to a minimum, and in many cases a single subscription is adequate and only adding more when it makes sense. Fewer subscriptions means less complexity and overhead.
Several ARM features facilitate a reduced number of subscriptions, especially Role Based Access Control (RBAC), Tagging, NETs/VNET Peering, ARM Policy, and Azure Active Directory (AAD).
- RBAC allows creation of different access control security models within one subscription.
- Tagging permits flexibility for billing to be segregated as needed within a single subscription.
- VNETs and NSGs allow you to segregate traffic within a subscription
- VNET peering allows you to connect those segregated vnets as needed to a central vnet or an express route connection.
- ARM Policy allows you to setup restrictions for naming conventions, resource and region usage, images, etc.
- AAD can be used to grant permissions within the RBAC framework
With the new Azure Resource Model, fewer subscriptions are needed resulting in fewer billing invoices, top level administrators and lower overall complexity. Less is better.
Best Practices
What you should do:
- Keep subscriptions to a minimum to reduce complexity
- Segment bills by leveraging Tagging instead of creating more subscriptions (See our tagging post here)
- Use Resource Groups as a application life cycle container
- Use Resource Groups to define security boundaries
- Use RBAC to grant access and delegate administration
What you shouldn't do:
- Do not create a subscription for each environment Dev/Test/Prod to protect quota and enforce security.
- Instead leverage the feature Azure Dev/Test Labs capability to ring-fence pools of IaaS capacity for specific dev teams/users if necessary. (Understand the limitations of Dev/Test Labs in its current state)
- Consider if enforcing quota is necessary. Using ARM Policy, or even an advanced billing system can have the same outcome without the complexity/business pain.
- Resource limits can modified by Microsoft up to a point, but most limits are well beyond what a typical client would require. For example one current limit is 10,000 VM cores per region per subscription. At present, Storage accounts is the limit most customers hit first, at 250 per subscription. Spend some time on careful management of this resource, but don't go crazy locking it down, 250 is still a lot and that change someday.
- Do not create multiple subscriptions just to have separate bills for each department.
- Tagging can be used to separate out cost
- Separate subscriptions introduces the need for a second set of networking infrastructure, or cross subscription Vnets thru site to site VPNs. While it is possible to do it does increase complexity.
- Do not use a subscription as the primary method of delegating administration.
- Subscriptions should be as a very high level administrative container, but that’s it. (e.g. one subscription per IT department in a company with multiple IT departments might make sense).
- Avoid spanning applications across multiple subscriptions with a single app, even with multiple environments, because it reduces your ability to view and manage all the related items from one place and on one bill.
- If you must have multiple subscriptions, don't split by dev/test/prod, instead split by groups of apps with the entire app and its related apps contained within a single subscription.
- Do not proactively split subscriptions based an "eventually" needing more resources.
- Resource limits are always increasing so by the time you may get close to a limit it would have likely been increased.
Billing
A single bill per client is best, leverage tagging to segment the bill
There is a 1:1 relationship between an Azure subscription and its corresponding bill so knowing how IT related expenses are handled internally will be a major factor to consider. For most clients the IT department handles the bill from a central budget. While other clients require the bill to be broken down in order to charge back costs to each business unit. Utilize tags to achieve the desired result.
Be mindful that only the Account administrator can manage billing.
Plans and Offers/Commercials
If you have an Enterprise Agreement with Microsoft, an EA subscription will be the best choice.
There are many different types of Azure tenant types available (Enterprise, MSDN, Pay-As-You-Go, Free, Sponsorship, etc.). The recommended subscription type applicable to most customers is Enterprise.
Be aware some tenants do not have access to certain service types. For example an EA tenant can not purchase AAD Basic or Premium through the Azure portal, it must be purchased separately thru the EA portal (because it is a per user based service). The same is also true of certain third party services available only through specific tenant type. Before moving forward with a subscription, ensure the commercial construct of the subscription matches the intended use. Sponsorship subscriptions should not be used as the main subscription as you'll need to migrate the resources to another subscription at some point. This is because Sponsorship subscriptions usually have a hard usage cap or time limit. When they reach that cap/limit, they revert to Pay-As-You-Go with no discounts.
Resource Limits
Do not create additional subscriptions planning on growing beyond current resource limits.
Resource limits in Azure change frequently. Do not use current limits and your anticipated growth to determine the number of subscriptions to deploy. Always have the mindset to have as few subscriptions as possible (ideally one), for as long as possible. Defer adding complexity that may never be needed or not needed for a while.
There are hard limits on how many of a given resource can exist within a subscription. These limits can be raised by Microsoft to a point, but once this point is reached additional subscriptions are required. It is important to note that ARM limits supersede the subscription limits, for example ARM is unlimited number of cores, but the limit is 10,000 VMs per region per subscription. Be aware that many quotas for resources in Azure Resource Groups are per-region, not per-subscription (as service management quotas are).
It is important to keep current on Azure Subscription Limits, and expect them to change frequently. Azure Subscription Limit information is available here.
When you have to separate into multiple subscriptions
Each application should be wholly contained within a single subscription whenever possible. e.g. the Dev/Test/Production environments for an app, across all regions should exist in a single subscription. Next try to place adjacent applications in a single subscription, particularly if they are managed and/or billed with the same people. If you split the app between subscriptions it reduces your ability to view and manage all the related items from one place, and see them on one bill. We have seen cases where clients have separated test/dev/production into separate subscriptions, which adds complexity without value. We don't recommend this now that RBAC and tagging are available.
Again, goal is as few subscriptions as possible. Only add new subscriptions as you need them. Do not proactively split subscriptions based an "eventually" needing more resources. Odds are by the time "eventually" comes the limits will have increased.
A Quick note on Azure Administrators and Co-Administrators
They still exist. Co-Administrators (limit 200 per subscription for ASM, no limit for ARM) - This role has the same access privileges as the Service Administrator, but can’t change the association of subscriptions to Azure directories. Minimize the number of co-admins as much as possible, and use RBAC to grant access on the principle of least privilege. Think of them as the domain admins of your azure account. E.g. they should be restricted to specific tasks that need them.
A Final note on Azure Stack and Azure Pack
Windows Azure Pack uses the ASM/Co-Administrator model, and since you own the whole thing, its not that detrimental to give each group its own subscription.
Windows Azure Stack uses the ARM model. Wile you could give everyone their own subscription, we don't see a reason to deviate from the guidance here within a business unit. Azure Stack subscriptions still need networking, etc. just like an azure subscription. There is a concept of delegated subscriptions (basically nesting) in Stack, but since it doesn't currently translate to Azure, and because RBAC/Resource Group based rights management works well, we simply don't see the need. Instead use multiple subscriptions in stack to peel off resources for groups you truly don't trust, like partners and 3rd party organizations.
This is the first in a series of blog posts discussing the subscription and resource group layout we recommend for large enterprise customers. Wile smaller customers may not hit some of the limits discussed here, its certainly equally relevant and applicable.
Wile this content is primarily written by the author, some guidance is the work of a collection of people within Avanade's Azure Cloud Enablement group.
More, HTML Reporting in PowerShell – Part 5
More? Okay, Part 5.I just created some new functions for creating two columns in a sections and added functionality for optional alternating row colors for tables. If you have worked through the other 4 parts you will be up to speed. If this is all new to you, I'd suggest to start here @ Part 1, otherwise, let's jump straight in.
Pre-requisites We are going to use the same code to create the Azure VMs recordset and other session variables we built up in the other parts. If you have previously installed this module, you will need to update it to 1.0.0.12 from the PowerShell Gallery. You can run update-module -name ReportHTML or if this is new to you you can use install-module -name ReportHTML. Additionally you can contribute via github ReportHTML which also has all the Examples scripts for download Example 15 without comments and Example 15 with comments.
Section with Columns (Example 15) First we will create a section with two columns and split the array into each column.
[powershell] ####### Example 15 ######## $rpt = @() $rpt += Get-HtmlOpen -TitleText ($ReportName + "Example 15") $rpt += Get-HtmlContentOpen -BackgroundShade 2 -HeaderText "VM States"
# We will start the first column with a get-htmlColumn1of2 $rpt += get-HtmlColumn1of2
# Now add the content you want in the first column $rpt += Get-HtmlContentOpen -BackgroundShade 1 -HeaderText Deallocated $rpt += Get-HtmlContentTable ($RMVMArray | where {$_.PowerState -eq "Deallocated"} | select ResourceGroup, Name, Size, DataDiskCount, ImageSKU) $rpt += Get-HtmlContentClose
# Next we need to close this first section $rpt += get-htmlColumnClose
# Then we can start the second column with a get-htmlColumn2of2 $rpt += get-htmlColumn2of2 $rpt += Get-HtmlContentOpen -BackgroundShade 1 -HeaderText Running $rpt += Get-HtmlContentTable ($RMVMArray | where {$_.PowerState -eq "Running"} | select ResourceGroup, Name, Size, DataDiskCount, ImageSKU ) $rpt += Get-HtmlContentClose
# Now we need to close this second section $rpt += get-htmlColumnClose $rpt += Get-HtmlContentClose [/powershell]
Alternating Row Color (Example 15) This next code section show the Set-TableRowColor with the -Alternating switch. Using this switch is an either or with the expression colouring.
[powershell] $rpt += Get-HtmlContentOpen -HeaderText 'Alternating Row Color'
# For Alternating row color you can use the -Alternating switch which will add the row color definition for Odd and Even and also adds a header color. $rpt += Get-HtmlContentTable (Set-TableRowColor ($RMVMArray | select ResourceGroup, Name, Location, Size, ImageSKU ) -Alternating) $rpt += Get-HtmlContentclose [/powershell]
Report Suggestions? Next, I plan to turn my attention to creating some practical reports for use. Even possibly recreating some existing reports for distribution like this computer report. If you have a report you can share or perhaps one you would like to see created, with special attention given to Azure content please share links, requests or ideas in the comments below.
Addendum
Here is a Sample Systems Report and the script that is a recreation of the computer report linked above it could use some more work but shows you what is possible.
It looks like I have a few system errors I need to go look at :)
Topic Search
Posts by Date
- August 2025 1
- March 2025 1
- February 2025 1
- October 2024 1
- August 2024 1
- July 2024 1
- October 2023 1
- September 2023 1
- August 2023 3
- July 2023 1
- June 2023 2
- May 2023 1
- February 2023 3
- January 2023 1
- December 2022 1
- November 2022 3
- October 2022 7
- September 2022 2
- August 2022 4
- July 2022 1
- February 2022 2
- January 2022 1
- October 2021 1
- June 2021 2
- February 2021 1
- December 2020 2
- November 2020 2
- October 2020 1
- September 2020 1
- August 2020 1
- June 2020 1
- May 2020 2
- March 2020 1
- January 2020 2
- December 2019 2
- November 2019 1
- October 2019 7
- June 2019 2
- March 2019 2
- February 2019 1
- December 2018 3
- November 2018 1
- October 2018 4
- September 2018 6
- August 2018 1
- June 2018 1
- April 2018 2
- March 2018 1
- February 2018 3
- January 2018 2
- August 2017 5
- June 2017 2
- May 2017 3
- March 2017 4
- February 2017 4
- December 2016 1
- November 2016 3
- October 2016 3
- September 2016 5
- August 2016 11
- July 2016 13