"CannotChangeInventoryId is not allowed" Enabling a VMWare OS template in Azure via VMWare Arc-Connected vSphere
While working on a VMware Arc-connected instance of vSphere I had an issue enabling a template for use in Azure. I had created a template deleted it then tried to recreate it with the same name and received the following error.
{"code":"DeploymentFailed","message":"At least one resource deployment operation failed. Please list deployment operations for details. Please see https://aka.ms/DeployOperations for usage details.","details":[{"code":"CannotChangeInventoryId","message":"Resource 'template-Win2019STD' was previously configured with InventoryItemId 'vmtpl-vm-2015'. Changing InventoryItemId is not allowed."}]}
In the resource group and select show hidden items, we can see a template with that name
if we actually look at the object or JSON view we can see this is the ID from the error
simply delete the template causing the issue
Now when you try to “Enable in Azure” it should work.
This also works with the other VMware types that can be enabled.
Updating Azure CLI on WSL2
I came across an issue on my local system when attempting to update Azure CLI to the latest version so that I could check out the Azure Arc HCIBox Jumpstart (It needs at least version 2.40.0).
I’m running Windows 11, with WSL2 Ubuntu-20.04 and I installed the AZ CLI using the one line install command. At the time, it was version 2.39.0
I tried to use the ‘az upgrade’ command, but it didn’t work.
So I went to the link provided https://aka.ms/doc/InstallAzureCli
There was nothing regarding the issue I had, so I figured I had to remove the cli and re-install.
Here’s the command I ran to remove the CLI
rm -r $HOME/lib/azure-cli
rm $HOME/bin/az
sed -i '/$HOME\/lib\/azure-cli\/az.completion/d' $HOME/.bash_profile
hash -rI figured I should use apt for the re-install:
sudo apt-get update && sudo apt-get install --only-upgrade -y azure-cli
Checking the version shows I now have the latest version:
And it looks like running ‘az upgrade’ should work…
Cool! Now it’s time to check out the HCIBox ;)
Creating MAAS workload annotations for Kubernetes using kubectl output
Trying to manually keep track of servers and versions is challenging. MAAS has a workload annotation feature that allows you to create name=value using command line calls using MASS cli. I wanted to explore creating a script to pull data from kubectl and create workload annotations for machines in a Kubernetes cluster.
For this exercise, I intend to execute the script from a MAAS rack controller, on which I have installed kubectl and PowerShell. In the lab, there are three microk8s clusters. The first job was to collect the config files from each cluster and combine them into a single config so I can use contexts from within the kubectl commands. Using the command ‘kubectl config view --raw > kn.conf’ I created one file for each k8s cluster on the region controller in directory ~/.kube/clusters.
mkdir ~/.kube/clusters
mv /path/k1.conf ~/.kube/clusters
mv /path/k2.conf ~/.kube/clusters
export KUBECONFIG=$(find ~/.kube/clusters -type f | sed ':a;N;s/\n/:/;ba')
kubectl config get-clusters
kubectl config view --flatten > ~/.kube/configThere are various methods to combine and clean up k8s cluster contexts. I had to clean up the file to create unique cluster names as well as user token names. I manually edited the ~/.kube/config combined file and tweaked these settings.
This is the script I created. It does require that you have a MAAS profile created already.
# tag-maasworkloadannotations.ps1
[CmdletBinding()]
param (
$k8scontext ,
$maasProfile
)
write-verbose "executing tag kubernetes workload annotation for MAAS"
$LastUpdated = Get-Date
# Kubectl contexts
Write-Verbose "Retrieving Kubernetes cluster details from context $k8scontext"
$kcontexts = kubectl config view -o json | convertfrom-json
Write-Verbose "$(($kcontexts.contexts | Measure-Object).count) kube contexts found"
$kcontext = $kcontexts.contexts | Where-Object {$_.name -eq $k8scontext}
$kversion = kubectl version --context $k8scontext -o json | convertfrom-json
# K8s nodes
Write-Verbose "Retrieving Kubernetes nodes from context $k8scontext"
$nodes = kubectl get nodes --context $k8scontext -o json | convertfrom-json
Write-Verbose "$(($nodes.items | Measure-Object).count) k8s nodes found"
# Maas machines
Write-Verbose "Retrieving machines from MAAS using profile $maasProfile"
$machines = maas $maasProfile machines read | convertfrom-json
Write-Verbose "$($machines.count) maas machines found"
$powerParams = maas $maasProfile machines power-parameters | convertfrom-json
# Build Annotations
Write-Verbose "Building workload annotation records"
$WorkloadAnnontations = @()
foreach ($node in $nodes.items) {
$WorkloadAnnontation = @{}
$WARecord = '' | select-object systemid, hostname, WA
$machine = $machines | Where-Object {$_.hostname -eq $node.metadata.name}
if ($machine -ne ""){
$WARecord.systemid = $machine.system_id
$WARecord.hostname = $machine.hostname
#$WorkloadAnnontation.add("osImage", $node.status.nodeInfo.osImage)
#$WorkloadAnnontation.add("systemUUID", $node.status.nodeInfo.systemUUID)
#$WorkloadAnnontation.add("machineID", $node.status.nodeInfo.machineID)
$WorkloadAnnontation.add("k8scluster", $kcontext.context.cluster )
$WorkloadAnnontation.add("buildDate", $kversion.serverVersion.buildDate)
$WorkloadAnnontation.add("containerRuntimeVersion", $node.status.nodeInfo.containerRuntimeVersion)
$WorkloadAnnontation.add("kernelVersion", $node.status.nodeInfo.kernelVersion)
$WorkloadAnnontation.add("kubeProxyVersion", $node.status.nodeInfo.kubeProxyVersion)
$WorkloadAnnontation.add("kubeletVersion", $node.status.nodeInfo.kubeletVersion)
$WorkloadAnnontation.add("outofband", $powerParams.$($machine.system_id).power_address)
$WorkloadAnnontation.add("AnnotationUpdated",$LastUpdated)
if ($node.metadata.labels.'node.kubernetes.io/microk8s-controlplane' -eq "microk8s-controlplane")
{$WorkloadAnnontation.add("nodeType", 'Master')}
if ($node.metadata.labels.'node.kubernetes.io/microk8s-worker' -eq "microk8s-worker")
{$WorkloadAnnontation.add("nodeType", 'Worker')}
$WARecord.wa = $WorkloadAnnontation
}
$WorkloadAnnontations += $WARecord
}
# publish workload annotations
$i = 1
$c = $($WorkloadAnnontations.count)
Write-Verbose "Publishing $c workload annotation records"
foreach ($WA in $WorkloadAnnontations){
$KeyValueData = ""
foreach ($key in $WA.wa.keys) {
$KeyValueData += "$($key)='$($wa.wa.($key))' "
}
Write-Verbose "[$i/$c] Building command for $($WA.hostname)"
$execmd = "maas $maasProfile machine set-workload-annotations $($WA.systemID) $KeyValueData > /dev/null 2>&1"
write-debug $execmd
Invoke-Expression $execmd
Write-Verbose "[$i/$c] Command executed for $($WA.hostname)"
$i++
}
$RunTime = New-TimeSpan -Start $LastUpdated -End (get-date)
$ExecutionTime = "Execution time was {0} hours, {1} minutes, {2} seconds and {3} milliseconds." -f $RunTime.Hours, $RunTime.Minutes, $RunTime.Seconds, $RunTime.Milliseconds
write-verbose $ExecutionTimeUsing the [CmdletBinding()] allows me to leverage verbose and debug options.
./tag-maasworkloadannotations.ps1 -k8scontext k8sdemo -maasprofile mquick -Verbose -debug./tag-maasworkloadannotations.ps1 -k8scontext microk8s -maasprofile mquick -Verbose
This took 12 minutes but could probably create streamlined script with PowerShell jobs
which can allow combinations of filters on the workload annotations
There is a lot more possible here, this was helpful for me while consolidating microk8s clusters and making sure I wasn’t releasing machines that were in use in a cluster if I had tagged them incorrectly in MAAS.
External Reference
Using multiple kubeconfig files and how to merge to a single – Oueta
Deploying ASDK 2206 to an Azure VM
Azure Stack Hub version 2206 was release a couple of months ago, but anyone trying to deploy ASDK 2206 to Azure will have found that the latest version is 2108. Until the official method is updated, here’s how you can do it .
I’m using the awesome scripts by Yagmur Sahin as the basis for the solution: https://github.com/yagmurs/AzureStack-VM-PoC
Open the Azure portal and then create a PowerShell Cloud Shell.
I recommend you reset the user settings, as there can be issues with versions of the Azure PowerShell modules.
Run the following command in the new PowerShell session:
git clone https://github.com/dmc-tech/AzureStack-VM-PoC.git
Run the following, changing to meet your requirements. VirtualMachineSize can be from the following sizes:
"Standard_E32s_v3",
"Standard_E48s_v3"
cd ./AzureStack-VM-PoC/ARMv2
$ResourceGroupName = 'asdk01-uks'
$Region = 'uk south'
$VirtualMachineSize = 'Standard_E48s_v3'
$DataDiskCount = 11
./Deploy-AzureStackonAzureVM.ps1 -ResourceGroupName $ResourceGroupName -Region $Region -VirtualMachineSize $VirtualMachineSize -DataDiskCount $DataDiskCountThe configuration example above has enough resources to run an OpenShift cluster.
Running the script will initially:
Create a resource group
Create a storage account
copy the ASDK 2206 VHD image to the storage account
Create the VM using the VHD image
Create a Public IP for the VM
Note: As part of the provisioning process, the admin user account you specify gets changed to ‘Administrator’. I would Strongly recommend removing the Public IP associated with the VM and deploy Azure Bastion to protect your ASDK instance
Once the ASDK VM has been provisioned, connect to it (Bastion or RDP). The username you specified previously is ignored, so use ‘Administrator’ as the user and enter the password you defined.
Once connected, open a PowerShell window (as Administrator), and run the following as an example (I’m using ADFS as I’m simulating a disconnected environment)
C:\CloudDeployment\Setup\InstallAzureStackPOC.ps1 -TimeServer '129.6.15.28' -DNSForwarder '8.8.8.8' -UseADFSYou’ll then need to enter the AdminPassword when prompted, and then the script will do it’s magic (as long as the password is correct!) and take a number of hours to install.
The above recording shows the first few minutes of the script (sped-up! :) ).
After a few hours, the VM will reboot. If you want to check progress, you should use the following username to connect:
azurestackadmin@azurestack.local
Use the password you initially defined
Here’s some of the output you’ll see from PowerShell if you do connect as azurestackadmin (there’s still a few hours left to go!)
After 7hours 25 minutes, the install completed. You can determine this from the following log entry:
To prove that version 2206 has been installed, open the admin portal and check the properties for the region/instance.
As I used ADFS for this example, I had to login as cloudadmin@azurestack.local. If using AAD use the account you define when initially running the setup script.
Hope that helps if you want to deploy version 2206 as well as a simplified deployment tutorial.
Footnote: I tried using v5 series VM’s to deploy ASDK on, but it failed due to a network issue. I assume it is due to a different NIC/drive being used than the v3 series.
Azure Arc Connected K8s Billing usage workbook error - 'union' operator: Failed to resolve table expression named 'ContainerLogV2'
Using ‘Container Insights’ for Kubernetes can get expensive, I recently saw a customer creating 19TB of logs a month from logging container insights from an AKS cluster.
You may want to check what logs you are capturing as suggested through Microsoft guides Monitoring cost for Container insights - Azure Monitor | Microsoft Docs which suggests one of the first steps to check billing usage.
As of writing this if you enable container insights and try to view Data Usage you get an error. ‘union' operator: Failed to resolve table expression named 'ContainerLogV2’. ContainerLogV2 is in preview or if you are just aren’t ready to deploy that, here is how to fix the report manually. I also have included the Full report at the end.
However, depending on when you deploy this and how you deployed insight metrics you may hit this error.
As of writing this, ContainerLogV2 is in preview and you can follow this guide to deploy it. Configure the ContainerLogV2 schema (preview) for Container Insights - Azure Monitor | Microsoft Docs. If you want to gain some quick insight into your usage you can edit the workbook. We just need to get through the maze of edits to get to the underlying query.
Select ‘Azure Arc’ services
select ‘Kubernetes clusters’ under infrastructure
select your K8s cluster
select ‘Workbooks’ under Monitoring
select ‘Container Insights Usage’ workbook
you can either edit this one or ‘Save As’ and create a new name. I will create a copy.
Open your chosen workbook
you can now edit the group item.
by Namespace
Editing the Namespace selection Dropdown parameter query
Hopefully, you get the gist of the process. There are a few more queries to edit.
And
This will now work for Data Usage. This is likely a temporary solution until ContainerLogV2 comes out of preview. Configure the ContainerLogV2 schema (preview) for Container Insights - Azure Monitor | Microsoft Docs
Gallery Template code
{
"version": "Notebook/1.0",
"items": [
{
"type": 9,
"content": {
"version": "KqlParameterItem/1.0",
"crossComponentResources": [
"{resource}"
],
"parameters": [
{
"id": "670aac26-0ffe-4f29-81c2-d48911bc64b6",
"version": "KqlParameterItem/1.0",
"name": "timeRange",
"label": "Time Range",
"type": 4,
"description": "Select time-range for data selection",
"isRequired": true,
"value": {
"durationMs": 21600000
},
"typeSettings": {
"selectableValues": [
{
"durationMs": 300000
},
{
"durationMs": 900000
},
{
"durationMs": 3600000
},
{
"durationMs": 14400000
},
{
"durationMs": 43200000
},
{
"durationMs": 86400000
},
{
"durationMs": 172800000
},
{
"durationMs": 259200000
},
{
"durationMs": 604800000
},
{
"durationMs": 1209600000
},
{
"durationMs": 2419200000
},
{
"durationMs": 2592000000
},
{
"durationMs": 5184000000
},
{
"durationMs": 7776000000
}
],
"allowCustom": true
}
},
{
"id": "bfc96857-81df-4f0d-b958-81f96d28ddeb",
"version": "KqlParameterItem/1.0",
"name": "resource",
"type": 5,
"isRequired": true,
"isHiddenWhenLocked": true,
"typeSettings": {
"additionalResourceOptions": [
"value::1"
],
"showDefault": false
},
"timeContext": {
"durationMs": 21600000
},
"timeContextFromParameter": "timeRange",
"defaultValue": "value::1"
},
{
"id": "8d48ec94-fde6-487c-98bf-f1295f5d8b81",
"version": "KqlParameterItem/1.0",
"name": "resourceType",
"type": 7,
"description": "Resource type of resource",
"isRequired": true,
"query": "{\"version\":\"1.0.0\",\"content\":\"\\\"{resource:resourcetype}\\\"\",\"transformers\":null}",
"isHiddenWhenLocked": true,
"typeSettings": {
"additionalResourceOptions": [
"value::1"
],
"showDefault": false
},
"timeContext": {
"durationMs": 21600000
},
"timeContextFromParameter": "timeRange",
"defaultValue": "value::1",
"queryType": 8
},
{
"id": "1b826776-ab99-45ab-86db-cced05e8b36d",
"version": "KqlParameterItem/1.0",
"name": "clusterId",
"type": 1,
"description": "Used to identify the cluster name when the cluster type is AKS Engine",
"isHiddenWhenLocked": true,
"timeContext": {
"durationMs": 0
},
"timeContextFromParameter": "timeRange"
},
{
"id": "67227c35-eab8-4518-9212-1c3c3d564b20",
"version": "KqlParameterItem/1.0",
"name": "masterNodeExists",
"type": 1,
"query": "let MissingTable = view () { print isMissing=1 };\r\nlet masterNodeExists = toscalar(\r\nunion isfuzzy=true MissingTable, (\r\nAzureDiagnostics \r\n| getschema \r\n| summarize c=count() \r\n| project isMissing=iff(c > 0, 0, 1)\r\n) \r\n| top 1 by isMissing asc\r\n);\r\nprint(iif(masterNodeExists == 0, 'yes', 'no'))\r\n",
"crossComponentResources": [
"{resource}"
],
"isHiddenWhenLocked": true,
"timeContext": {
"durationMs": 0
},
"timeContextFromParameter": "timeRange",
"queryType": 0,
"resourceType": "{resourceType}"
}
],
"style": "pills",
"queryType": 0,
"resourceType": "{resourceType}"
},
"name": "pills"
},
{
"type": 1,
"content": {
"json": "Please note that the Container Insights Usage workbook for AKS Engine clusters shows your billable data usage for your cluster's entire workspace ({resource:name}), and not just the cluster itself ({clusterId}). ",
"style": "info"
},
"conditionalVisibility": {
"parameterName": "resourceType",
"comparison": "isEqualTo",
"value": "microsoft.operationalinsights/workspaces"
},
"name": "aks-engine-billable-data-shown-applies-to-entire-workspace-not-just-the-cluster-info-message",
"styleSettings": {
"showBorder": true
}
},
{
"type": 11,
"content": {
"version": "LinkItem/1.0",
"style": "tabs",
"links": [
{
"id": "3b7d39f2-38c5-4586-a155-71e28e333020",
"cellValue": "selectedTab",
"linkTarget": "parameter",
"linkLabel": "Overview",
"subTarget": "overview",
"style": "link"
},
{
"id": "7163b764-7ab2-48b3-b417-41af3eea7ef0",
"cellValue": "selectedTab",
"linkTarget": "parameter",
"linkLabel": "By Table",
"subTarget": "table",
"style": "link"
},
{
"id": "960bbeea-357f-4c07-bdd9-6f3f123d56fd",
"cellValue": "selectedTab",
"linkTarget": "parameter",
"linkLabel": "By Namespace",
"subTarget": "namespace",
"style": "link"
},
{
"id": "7fbcc5bd-62d6-4aeb-b55e-7b1fab65716a",
"cellValue": "selectedTab",
"linkTarget": "parameter",
"linkLabel": "By Log Source",
"subTarget": "logSource",
"style": "link"
},
{
"id": "3434a63b-f3d5-4c11-9620-1e731983041c",
"cellValue": "selectedTab",
"linkTarget": "parameter",
"linkLabel": "By Diagnostic Master Node ",
"subTarget": "diagnosticMasterNode",
"style": "link"
}
]
},
"name": "tabs"
},
{
"type": 1,
"content": {
"json": "Due to querying limitation on Basic Logs, if you are using Basic Logs, this workbook provides partial data on your container log usage.",
"style": "info"
},
"conditionalVisibility": {
"parameterName": "selectedTab",
"comparison": "isEqualTo",
"value": "overview"
},
"name": "basic-logs-info-text"
},
{
"type": 12,
"content": {
"version": "NotebookGroup/1.0",
"groupType": "editable",
"title": "Container Insights Billing Usage",
"items": [
{
"type": 1,
"content": {
"json": "<br/>\r\nThis workbook provides you a summary and source for billable data collected by [Container Insights solution.](https://docs.microsoft.com/en-us/azure/azure-monitor/insights/container-insights-overview)\r\n\r\nThe best way to understand how Container Insights ingests data in Log Analytics workspace is from this [article.](https://medium.com/microsoftazure/azure-monitor-for-containers-optimizing-data-collection-settings-for-cost-ce6f848aca32)\r\n\r\nIn this workbook you can:\r\n\r\n* View billable data ingested by **solution**.\r\n* View billable data ingested by **Container logs (application logs)**\r\n* View billable container logs data ingested segregated by **Kubernetes namespace**\r\n* View billable container logs data ingested segregated by **Cluster name**\r\n* View billable container log data ingested by **logsource entry**\r\n* View billable diagnostic data ingested by **diagnostic master node logs**\r\n\r\nYou can fine tune and control logging by turning off logging on the above mentioned vectors. [Learn how to fine-tune logging](https://docs.microsoft.com/en-us/azure/azure-monitor/insights/container-insights-agent-config)\r\n\r\nYou can control your master node's logs by updating diagnostic settings. [Learn how to update diagnostic settings](https://docs.microsoft.com/en-us/azure/aks/view-master-logs)"
},
"name": "workbooks-explanation-text"
},
{
"type": 1,
"content": {
"json": "`Master node logs` are not enabled. [Learn how to enable](https://docs.microsoft.com/en-us/azure/aks/view-master-logs)"
},
"conditionalVisibility": {
"parameterName": "masterNodeExists",
"comparison": "isEqualTo",
"value": "no"
},
"name": "master-node-logs-are-not-enabled-msg"
}
]
},
"conditionalVisibility": {
"parameterName": "selectedTab",
"comparison": "isEqualTo",
"value": "overview"
},
"name": "workbook-explanation"
},
{
"type": 12,
"content": {
"version": "NotebookGroup/1.0",
"groupType": "editable",
"items": [
{
"type": 3,
"content": {
"version": "KqlItem/1.0",
"query": "union withsource = SourceTable AzureDiagnostics, AzureActivity, AzureMetrics, ContainerLog, Perf, KubePodInventory, ContainerInventory, InsightsMetrics, KubeEvents, KubeServices, KubeNodeInventory, ContainerNodeInventory, KubeMonAgentEvents, ContainerServiceLog, Heartbeat, KubeHealth, ContainerImageInventory\r\n| where _IsBillable == true\r\n| project _BilledSize, TimeGenerated, SourceTable\r\n| summarize BillableDataBytes = sum(_BilledSize) by bin(TimeGenerated, {timeRange:grain}), SourceTable\r\n| render piechart\r\n\r\n\r\n",
"size": 3,
"showAnnotations": true,
"showAnalytics": true,
"title": "Billable Data from Container Insights",
"timeContextFromParameter": "timeRange",
"queryType": 0,
"resourceType": "{resourceType}",
"crossComponentResources": [
"{resource}"
],
"chartSettings": {
"xAxis": "TimeGenerated",
"createOtherGroup": 100,
"seriesLabelSettings": [
{
"seriesName": "LogManagement",
"label": "LogManagementSolution(GB)"
},
{
"seriesName": "ContainerInsights",
"label": "ContainerInsightsSolution(GB)"
}
],
"ySettings": {
"numberFormatSettings": {
"unit": 2,
"options": {
"style": "decimal"
}
}
}
}
},
"customWidth": "50",
"showPin": true,
"name": "billable-data-from-ci"
}
]
},
"conditionalVisibility": {
"parameterName": "selectedTab",
"comparison": "isEqualTo",
"value": "table"
},
"name": "by-datatable-tab"
},
{
"type": 12,
"content": {
"version": "NotebookGroup/1.0",
"groupType": "editable",
"items": [
{
"type": 3,
"content": {
"version": "KqlItem/1.0",
"query": "KubePodInventory\r\n| distinct ContainerID, Namespace\r\n| join kind=innerunique (\r\nContainerLog\r\n| where _IsBillable == true\r\n| summarize BillableDataBytes = sum(_BilledSize) by ContainerID\r\n) on ContainerID\r\n| union (\r\nKubePodInventory\r\n| distinct ContainerID, Namespace\r\n)\r\n| summarize Total=sum(BillableDataBytes) by Namespace\r\n| render piechart\r\n",
"size": 3,
"showAnalytics": true,
"title": "Billable Container Log Data Per Namespace",
"timeContextFromParameter": "timeRange",
"queryType": 0,
"resourceType": "{resourceType}",
"crossComponentResources": [
"{resource}"
],
"tileSettings": {
"showBorder": false,
"titleContent": {
"columnMatch": "Namespace",
"formatter": 1
},
"leftContent": {
"columnMatch": "Total",
"formatter": 12,
"formatOptions": {
"palette": "auto"
},
"numberFormat": {
"unit": 17,
"options": {
"maximumSignificantDigits": 3,
"maximumFractionDigits": 2
}
}
}
},
"graphSettings": {
"type": 0,
"topContent": {
"columnMatch": "Namespace",
"formatter": 1
},
"centerContent": {
"columnMatch": "Total",
"formatter": 1,
"numberFormat": {
"unit": 17,
"options": {
"maximumSignificantDigits": 3,
"maximumFractionDigits": 2
}
}
}
},
"chartSettings": {
"createOtherGroup": 100,
"ySettings": {
"numberFormatSettings": {
"unit": 2,
"options": {
"style": "decimal"
}
}
}
}
},
"name": "billable-data-per-namespace"
}
]
},
"conditionalVisibility": {
"parameterName": "selectedTab",
"comparison": "isEqualTo",
"value": "namespace"
},
"name": "by-namespace-tab"
},
{
"type": 12,
"content": {
"version": "NotebookGroup/1.0",
"groupType": "editable",
"items": [
{
"type": 9,
"content": {
"version": "KqlParameterItem/1.0",
"crossComponentResources": [
"{resource}"
],
"parameters": [
{
"id": "d5701caa-0486-4e6f-adad-6fa5b8496a7d",
"version": "KqlParameterItem/1.0",
"name": "namespace",
"label": "Namespace",
"type": 2,
"description": "Filter data by namespace",
"isRequired": true,
"query": "KubePodInventory\r\n| distinct ContainerID, Namespace\r\n| join kind=innerunique (\r\nContainerLog\r\n| project ContainerID\r\n) on ContainerID\r\n| union (\r\nKubePodInventory\r\n| distinct ContainerID, Namespace\r\n)\r\n| distinct Namespace\r\n| project value = Namespace, label = Namespace, selected = false\r\n| sort by label asc",
"crossComponentResources": [
"{resource}"
],
"typeSettings": {
"additionalResourceOptions": [
"value::1"
],
"showDefault": false
},
"timeContext": {
"durationMs": 0
},
"timeContextFromParameter": "timeRange",
"defaultValue": "value::1",
"queryType": 0,
"resourceType": "{resourceType}"
}
],
"style": "pills",
"queryType": 0,
"resourceType": "{resourceType}"
},
"name": "namespace-pill"
},
{
"type": 3,
"content": {
"version": "KqlItem/1.0",
"query": "KubePodInventory\r\n| where Namespace == '{namespace}'\r\n| distinct ContainerID, Namespace\r\n| join hint.strategy=shuffle (\r\nContainerLog\r\n| where _IsBillable == true\r\n| summarize BillableDataBytes = sum(_BilledSize) by LogEntrySource, ContainerID\r\n) on ContainerID\r\n| union (\r\nKubePodInventory\r\n| where Namespace == '{namespace}'\r\n| distinct ContainerID, Namespace\r\n)\r\n| extend sourceNamespace = strcat(LogEntrySource, \"/\", Namespace)\r\n| summarize Total=sum(BillableDataBytes) by sourceNamespace\r\n| render piechart",
"size": 0,
"showAnnotations": true,
"showAnalytics": true,
"title": "Billable Data Per Log-Source Type for Namespace Selected",
"timeContextFromParameter": "timeRange",
"queryType": 0,
"resourceType": "{resourceType}",
"crossComponentResources": [
"{resource}"
],
"chartSettings": {
"createOtherGroup": 100,
"ySettings": {
"numberFormatSettings": {
"unit": 2,
"options": {
"style": "decimal"
}
}
}
}
},
"customWidth": "35",
"showPin": true,
"name": "billable-data-per-log-source",
"styleSettings": {
"showBorder": true
}
},
{
"type": 3,
"content": {
"version": "KqlItem/1.0",
"query": "KubePodInventory\r\n| where Namespace == '{namespace}'\r\n| distinct ContainerID, Namespace\r\n| project ContainerID\r\n| join hint.strategy=shuffle ( \r\nContainerLog\r\n| project _BilledSize, ContainerID, TimeGenerated, LogEntrySource\r\n) on ContainerID\r\n| union (\r\nKubePodInventory\r\n| where Namespace == '{namespace}'\r\n| distinct ContainerID, Namespace\r\n| project ContainerID\r\n)\r\n| summarize ContainerLogData = sum(_BilledSize) by bin(TimeGenerated, {timeRange:grain}), LogEntrySource\r\n| render linechart\r\n\r\n",
"size": 0,
"showAnnotations": true,
"showAnalytics": true,
"title": "Billable Container Log Data for Namespace Selected",
"timeContextFromParameter": "timeRange",
"queryType": 0,
"resourceType": "{resourceType}",
"crossComponentResources": [
"{resource}"
],
"chartSettings": {
"xSettings": {},
"ySettings": {
"numberFormatSettings": {
"unit": 2,
"options": {
"style": "decimal"
}
}
}
}
},
"customWidth": "65",
"conditionalVisibility": {
"parameterName": "namespace:label",
"comparison": "isNotEqualTo",
"value": "<unset>"
},
"showPin": true,
"name": "billable-container-log-data-for-namespace-selected",
"styleSettings": {
"showBorder": true
}
}
]
},
"conditionalVisibility": {
"parameterName": "selectedTab",
"comparison": "isEqualTo",
"value": "logSource"
},
"name": "by-log-source-tab"
},
{
"type": 12,
"content": {
"version": "NotebookGroup/1.0",
"groupType": "editable",
"items": [
{
"type": 1,
"content": {
"json": "`Master node logs` are not enabled. [Learn how to enable](https://docs.microsoft.com/en-us/azure/aks/view-master-logs)"
},
"conditionalVisibility": {
"parameterName": "masterNodeExists",
"comparison": "isEqualTo",
"value": "no"
},
"name": "master-node-logs-are-not-enabled-msg-under-masternode-tab"
},
{
"type": 3,
"content": {
"version": "KqlItem/1.0",
"query": "AzureDiagnostics\r\n| summarize Total=sum(_BilledSize) by bin(TimeGenerated, {timeRange:grain}), Category\r\n| render barchart\r\n\r\n\r\n",
"size": 0,
"showAnnotations": true,
"showAnalytics": true,
"title": "Billable Diagnostic Master Node Log Data",
"timeContextFromParameter": "timeRange",
"queryType": 0,
"resourceType": "{resourceType}",
"crossComponentResources": [
"{resource}"
],
"chartSettings": {
"ySettings": {
"unit": 2,
"min": null,
"max": null
}
}
},
"customWidth": "75",
"conditionalVisibility": {
"parameterName": "masterNodeExists",
"comparison": "isEqualTo",
"value": "yes"
},
"showPin": true,
"name": "billable-data-from-diagnostic-master-node-logs"
}
]
},
"conditionalVisibility": {
"parameterName": "selectedTab",
"comparison": "isEqualTo",
"value": "diagnosticMasterNode"
},
"name": "by-diagnostic-master-node-tab"
}
],
"fallbackResourceIds": [
"/subscriptions/357e7522-d1ad-433c-9545-7d8b8d72d61a/resourceGroups/ArcResources/providers/Microsoft.Kubernetes/connectedClusters/sv5-k8sdemo"
],
"fromTemplateId": "community-Workbooks/AKS/Billing Usage Divided",
"$schema": "https://github.com/Microsoft/Application-Insights-Workbooks/blob/master/schema/workbook.json"
}Topic Search
-
Securing TLS in WAC (Windows Admin Center) https://t.co/klDc7J7R4G
Posts by Date
- August 2025 1
- March 2025 1
- February 2025 1
- October 2024 1
- August 2024 1
- July 2024 1
- October 2023 1
- September 2023 1
- August 2023 3
- July 2023 1
- June 2023 2
- May 2023 1
- February 2023 3
- January 2023 1
- December 2022 1
- November 2022 3
- October 2022 7
- September 2022 2
- August 2022 4
- July 2022 1
- February 2022 2
- January 2022 1
- October 2021 1
- June 2021 2
- February 2021 1
- December 2020 2
- November 2020 2
- October 2020 1
- September 2020 1
- August 2020 1
- June 2020 1
- May 2020 2
- March 2020 1
- January 2020 2
- December 2019 2
- November 2019 1
- October 2019 7
- June 2019 2
- March 2019 2
- February 2019 1
- December 2018 3
- November 2018 1
- October 2018 4
- September 2018 6
- August 2018 1
- June 2018 1
- April 2018 2
- March 2018 1
- February 2018 3
- January 2018 2
- August 2017 5
- June 2017 2
- May 2017 3
- March 2017 4
- February 2017 4
- December 2016 1
- November 2016 3
- October 2016 3
- September 2016 5
- August 2016 11
- July 2016 13