You missed my session on GPU Management with PowerCLI and Python at VMware Explore this year!?!? That’s fine – more than 50 people watched it live. Maybe you were one of them. 😉
Regardless, if you’re interested in learning more about managing GPUs in VMware VCF using either PowerShell or Python, I’ve got you covered. I’m sharing all the resources from my session right here for your enjoyment and enlightenment.
From DirectPath Profiles to GPU Performance Stats
Last year I talked about managing DirectPath profiles using PowerShell and PowerCLI (VCF and VVF 9 New GPU Management Scripting with PowerShell and Python – CODEQT1767LV). This year I went for something that seemed simpler, but isn’t: pulling GPU performance statistics.
Both VMware and NVIDIA have been working hard on adding additional stats around GPUs. This is helpful for many reasons – regardless of whether you’re doing AI, VDI, or maybe something else entirely with GPUs. The intent, of course, is to feed these stats into vSphere monitoring tools like VCF Operations.
Having all this information at your fingertips is awesome. You should also be able to build automation workflows around some of the data that comes out of it – which is great if that’s your only tool.
But wouldn’t it be even better to feed GPU stats into Grafana, or build a system that watches for hot spots in your data center and balances workloads by temperature?
That’s where the scripts I shared come into play. They’re the basic building blocks needed for any project that consumes GPU stats.
Watch the Session and Get the Slides
If you’d like to watch the session, the video can be found in the VMware Explore video library.
If you’d like the slides from the session, I made sure to share them.
Now, on to how all this stuff actually works.
What GPU Metrics Are Available in VCF 9.1?
Let’s start with what GPU metrics you can pull from a 9.1 environment. In my research I was only able to find values for clusters and hosts. Metrics should exist for VMs as well, but I was not able to get them to appear correctly in my lab environment with an NVIDIA L4 GPU and various vGPU configurations for the VM.
The performance counters are the same for both clusters and individual hosts. They are:
gpu.utilization.average– The compute utilization of a GPU in percentagegpu.mem.used.average– The amount of GPU memory used by workloads in kilobytesgpu.mem.reserved.latest– The amount of GPU memory consumed by virtual GPUs in kilobytesgpu.power.used.latest– The power used by a GPU in wattsgpu.temperature.average– The temperature of a GPU in degrees Celsiusgpu.mem.total.latest– The total amount of GPU memory in kilobytes
The Complete GPU Counter List
There are other GPU counters if you query the perfManager object. Currently, many do not appear to have values when I query for them – it may be that my GPU doesn’t support them, or they haven’t been linked and activated yet. The first six in the following list are the ones that returned values I was able to query. The complete list from my environment:
ID: 286 | Metric: gpu.utilization.average
ID: 287 | Metric: gpu.mem.used.average
ID: 288 | Metric: gpu.mem.reserved.latest
ID: 289 | Metric: gpu.power.used.latest
ID: 290 | Metric: gpu.temperature.average
ID: 291 | Metric: gpu.mem.total.latest
ID: 479 | Metric: gpu.utilization.latest
ID: 480 | Metric: gpu.mem.used.latest
ID: 481 | Metric: gpu.temperature.latest
ID: 482 | Metric: gpu.compute.clock.demand.latest
ID: 483 | Metric: gpu.compute.clock.used.latest
ID: 484 | Metric: gpu.compute.utilization.latest
ID: 485 | Metric: gpu.mem.utilization.latest
ID: 486 | Metric: gpu.mem.demand.latest
ID: 487 | Metric: gpu.encode.utilization.latest
ID: 488 | Metric: gpu.decode.utilization.latest
ID: 489 | Metric: gpu.power.limit.latest
ID: 490 | Metric: gpu.compute.clock.limit.latest
ID: 491 | Metric: gpu.shutdown.temperature.latest
ID: 492 | Metric: gpu.slowdown.temperature.latest
ID: 493 | Metric: gpu.mem.errors.ecc.corrected.latest
ID: 494 | Metric: gpu.mem.errors.ecc.uncorrected.latest
ID: 495 | Metric: gpu.tensor.utilization.latest
ID: 842 | Metric: gpu.utilization.none
ID: 843 | Metric: gpu.utilization.maximum
ID: 844 | Metric: gpu.utilization.minimum
ID: 845 | Metric: gpu.mem.used.none
ID: 846 | Metric: gpu.mem.used.maximum
ID: 847 | Metric: gpu.mem.used.minimum
ID: 848 | Metric: gpu.mem.usage.none
ID: 849 | Metric: gpu.mem.usage.average
ID: 850 | Metric: gpu.mem.usage.maximum
ID: 851 | Metric: gpu.mem.usage.minimum
ID: 852 | Metric: gpu.mem.used.gb.latest
ID: 853 | Metric: gpu.mem.reserved.gb.latest
ID: 854 | Metric: gpu.mem.total.gb.latest
Finding Available GPU Metrics with PowerCLI
You might be wondering how to know what’s available on your hosts or clusters. In PowerShell you can run the following commands to get information about your GPU metrics:
#$TargetEntity = Get-Cluster -Name "91-lab-vdi-cl01"
$TargetEntity = Get-VMHost -Name "esx04.wondernerd.lab"
Get-StatType -entity $TargetEntity | where-object {$_ -match "gpu"}
For this command set I have two different $TargetEntity values defined – one gets the cluster object, the other gets the host object. That’s passed to the Get-StatType command and then piped to an object search that matches items with “gpu” in the name. The result is the list of objects I described above.
Finding Available GPU Metrics with Python (pyVmomi)
We can do this in Python as well – it’s just a bit longer:
from pyVim.connect import SmartConnect
from vmware.vapi.vsphere.client import create_vsphere_client, VsphereClient
from pyVmomi import vim
import ServiceInstance #file with login info
si = ServiceInstance.ConnectToVC()
content = si.RetrieveContent()
perfManager = content.perfManager
counter_ids = [] #Base ID Values
counter_info = {} #Full ID and name pairs
counter_id_dict = {} #GPU ID name pairs
for counter in perfManager.perfCounter:
full_name = f"{counter.groupInfo.key}.{counter.nameInfo.key}.{counter.rollupType}"
counter_info[counter.key] = full_name
if "gpu" in full_name:
print(f"ID: {counter.key} | Metric: {full_name}")
counter_ids.append(counter.key)
counter_id_dict.update({counter.key : full_name})
This code produces the second list I provided above, with the IDs and metric names.
Reading Actual GPU Metric Values with PowerCLI
You’re probably thinking: it’s good to know what metrics exist, but how do we find out what the actual values are? In PowerCLI we do it by passing one more parameter to Get-Stat and running the results through a ForEach loop:
$HostPerf = get-stattype -Entity $TargetEntity | where-object {$_ -match "gpu"}
foreach ($HostHas in $HostPerf) {
$GPUresults = Get-Stat -Entity $TargetEntity -Stat $HostHas –Realtime -MaxSamples 1
Write-Host $GPUresults.MetricId
Write-Host $GPUresults.Description
Write-Host $GPUresults.Value
}
The output looks something like this:
gpu.utilization.average
The compute utilization of a GPU in percentage
0
gpu.mem.used.average
The amount of GPU memory used by workloads in kilobytes
8238592 ...
Reading GPU Metric Values with Python
Python, again, is a little longer:
host = content.searchIndex.FindByDnsName(dnsName="esx04.wondernerd.lab", vmSearch=False)
metrics = [vim.PerformanceManager.MetricId(counterId=cid, instance="*") for cid in counter_ids]
query = vim.PerformanceManager.QuerySpec(
entity=host,
metricId=metrics,
intervalId=20,
maxSample=1
)
perf_results = perfManager.QueryPerf(querySpec=[query])
for result in perf_results:
for metric_value in result.value:
counter_id = metric_value.id.counterId
# Fetch the most recent value from the value array
latest_val = metric_value.value[-1] if metric_value.value else 0
print(f"Description: {counter_id_dict[counter_id]} | Value: {latest_val}")
We basically set up a query for a single sample and pass that into perfManager using the QueryPerf property. This gives us an array we can work with, then we filter the results using nested for loops. The result looks something like this:
Description: gpu.power.used.latest | Value: 17
Description: gpu.mem.used.none | Value: 4633728
Description: gpu.mem.used.average | Value: 4633728
Description: gpu.mem.used.maximum | Value: 4633728
Description: gpu.mem.used.minimum | Value: 4633728
Description: gpu.mem.reserved.latest | Value: 4633728
Description: gpu.temperature.average | Value: 42
Description: gpu.utilization.none | Value: 0
Description: gpu.utilization.average | Value: 0
Description: gpu.utilization.maximum | Value: 0
Description: gpu.utilization.minimum | Value: 0
Description: gpu.mem.usage.none | Value: 1964
Description: gpu.mem.usage.average | Value: 1964
Description: gpu.mem.usage.maximum | Value: 1964
Description: gpu.mem.usage.minimum | Value: 1964
Description: gpu.mem.total.latest | Value: 23586816
Practical Use Case: Detecting Data Center Hot Spots
Being able to get the values is great and all, but how do we apply it to something practical – say, a data center, or even a single host, getting too warm?
Easy enough. Let’s pull the temperature from the GPU for the last day and run some statistics on it. With PowerCLI:
$GPUtemp = Get-Stat -Entity $TargetEntity –Stat gpu.temperature.average -Start (Get-Date).AddDays(-1) -IntervalMins 5
$CleanTempArray = @()
[system.Int32[]]$CleanTempArray = @($GPUtemp.Value)
($CleanTempArray | Measure-Object -StandardDeviation -Average -Sum -Maximum -Minimum)
We use the Get-Stat command and pass it the stat we want along with the time range. Next we create a temp array to hold our statistics from processing the GPU temp data. Then we process it using Measure-Object and ask for our statistics. The result:
Count : 288
Average : 42.5902777777778
Sum : 12266
Maximum : 44
Minimum : 41
StandardDeviation : 0.687501099847781
Property :
If you’d like to read more about standard deviation, I’d recommend the Wikipedia article on it.
The takeaway: if my GPU temperature climbs above 43.965 degrees, the cooling should be investigated because something isn’t “normal.” We could then use some of the other functions in Python or PowerShell to look at the whole environment and see if the problem is specific to a host, a rack, a cluster, or any other grouping. We can then take appropriate action for the impacted VMs, maybe that’s putting the host in maintenance mode or sending a notification. The results can help optimize and protect your workloads.
Get the Scripts
You can find all of these scripts (plus last years scripts) on my GitHub and work with them yourself. If you find a fun way to use them, please share your approach.
