Automate UCS firmware upgrade for VMware vSphere clusters

I found plenty of posts with scripts that automated the firmware upgrade for UCS servers running VMware ESXi. I did however not think they were “safe” enough, so I wrote my own.
It will update the firmware of all the hosts in the specified cluster, one host at a time.
It requires UCS Powertool 2.x and VMware PowerCLI 6.x. It logs the relevant steps with time stamps, so you’ll know whats happening.

It does the following:
1. Get the ESXi-hosts in the cluster to work in.
2. Dismounts VMware tools(would prevent putting a host in maintenance mode) on VMs running on the host.
3. Takes the first(added to the cluster, not name) host and puts it in maintenance mode.
4. Records the MAC address of vmnic0 for that host.
5. Shuts down the host.
6. Checks in UCS Manager which Service Profile matches that MAC address.
7. Verifies that the matching server is powered off.
8. Applies the firmware policy specified.
9. Acknowledges the reboot.
10. Waits until the server operational states goes from “config” to “ok”.
11. Waits for the server to reconnect in vCenter.
12. Exits the host from maintenance mode.
13. Repeats the process from step 2 with the rest of the hosts in the cluster.

PS. I always make sure I understand exactly what a script I find on the internet does before I run it, and test it in a test environment first. I urge you to do the same.

# Load Cisco UCS Modules
. "C:\Program Files (x86)\WindowsPowerShell\Modules\Cisco.UCSManager\StartUcsPS.ps1"
 
# Add the VMware Modules
if ( !(Get-Module -Name VMware.VimAutomation.Core -ErrorAction SilentlyContinue) ) {
. “C:\Program Files (x86)\VMware\Infrastructure\PowerCLI\Scripts\Initialize-PowerCLIEnvironment.ps1”
}
 
# Enter the vCenter, UCS Fabric Interconnect, ESXi cluster name and Firmware package.
$vcenter = "<vcenter ip>"
$ucsfi = "<ucs fi ip>"
$cluster = "<esxi cluster name>"
$fw = "<host firmware policy name>"
 
# Connect to Vcenter
Connect-Viserver $vcenter
 
# Connect to UCS FI
Connect-Ucs $ucsfi
 
# Get VMhosts in Cluster
$vmhosts = get-cluster $cluster |get-vmhost
 
# Begin loop
foreach($vmhostname in $vmhosts) {
 
# Notify
write-host (get-date -UFormat %H:%M) "*** Starting upgrade process of" $vmhostname.name "***"
 
# Dismount VMware tools
$toolsvms = Get-vmhost $vmhostname | get-vm
($toolsvms | Get-View | Where {$_.Runtime.ToolsInstallerMounted}) | % {Dismount-Tools -VM $_.Name}
foreach ($vm in $toolsvms) {
                             $cd = Get-CDDrive $vm | where { $_.ConnectionState.Connected -eq "true"}
                             if ( $cd.isopath -like "*tools*iso" ){
                                                          write-host $vm
                                                          set-cddrive -cd $cd -nomedia -Confirm:$false
                                                          }
}
 
# Put Host into Maintenance Mode
write-host (get-date -UFormat %H:%M) "Placing server in maintenance mode"
Set-VMHost -VMHost $vmhostname -State "Maintenance" -Evacuate:$true -confirm:$false | Out-Null
 
# Get Host Mac Address
$vmhost = Get-VMHost $vmhostname
$vmMacAddr = $VMhost.NetworkInfo.PhysicalNic | where { $_.name -ieq "vmnic0" }
 
# Shutdown VMhost
write-host (get-date -UFormat %H:%M) "Shutting down server"
Stop-VMHost $vmhost -Confirm:$false | Out-Null
 
# Wait for ESXi host to completely shut down
write-host (get-date -UFormat %H:%M) "Waiting for server to power off"
while (Get-VMHost $vmhost | where {$_.ConnectionState -eq "Maintenance"}) {sleep 10}
 
# Get UCS Service Profile that matches the MAC address
$SP = (Get-UcsServiceProfile | Get-UCSVnic | where {$_.addr -ieq $vmMacAddr.mac} | Get-UcsParent).Name
 
# Get physical blade
$UcsBlade = (Get-UcsServiceProfile $SP).PnDn
 
# Wait if its not powered off yet in UCSM
write-host (get-date -UFormat %H:%M) "Verifying that server is powered off in UCSM"
while ((Get-Ucsserver |where {$_.Dn -eq $UcsBlade}).operpower -eq "on") {sleep 10}
 
# Pause
write-host (get-date -UFormat %H:%M) "The firmware of" $SP "will be upgraded to" $fw "in 60 seconds, hit CTRL-C to abort"
sleep 60
 
# Continue only if the blade is powered off
if ((Get-Ucsserver |where {$_.Dn -eq $UcsBlade}).operpower -eq "off")
                            {
 
                             # Set Firmware Policy
                             write-host (get-date -UFormat %H:%M) "Applying firmware update..."
                             Get-UcsServiceProfile -name $SP | Set-UcsServiceProfile -HostFwPolicyName $fw -Force | Out-Null
                             sleep 5
 
                             # Acknowledge the policy change
                             Get-UcsServiceProfile -Name $SP | Get-UcsLsmaintAck | Set-UcsLsmaintAck -AdminState "trigger-immediate" -Descr "" -PolicyOwner "local" -Scheduler "" -Force | Out-Null
                             sleep 5
 
                             # Wait for server to be operational
                             while ((Get-Ucsserver |where {$_.Dn -eq $UcsBlade}).operstate -ne "ok") {sleep 60}
                             write-host (get-date -UFormat %H:%M) "Update complete"
 
                             # Wait for Server to be reconnected
                             write-host (get-date -UFormat %H:%M) "Waiting for server to reconnect in vCenter"
                             while (Get-VMHost $vmhost | where {$_.ConnectionState -eq "Disconnected" -or $_.ConnectionState -eq "NotResponding"}) {sleep 10}
                             sleep 30
 
                             # Take VMhost out of Maintenance Mode
                             write-host (get-date -UFormat %H:%M) "Exiting maintenance mode"
                             Set-VMHost -VMHost $vmhost -State "Connected" -confirm:$false | Out-Null
 
                             # Notify completion
                             write-host (get-date -UFormat %H:%M) "*** Firmware upgrade of" $SP "complete ***"
                             sleep 400
                             }
else                         {
                             write-host (get-date -UFormat %H:%M) "Error: failed to power off server. Stopping script" ; break
                             }
}
 
# Disconnect from Vcenter & UCS
Disconnect-Viserver $vcenter -confirm:$false
Disconnect-Ucs
This entry was posted in Automation, PowerCLI, VMware and tagged , , , , , , , . Bookmark the permalink.