get-admin

If you have ever changed the vCenter server certificates, you’ve experienced having all your hosts disconnected from vCenter.  I couldn’t imagine reconnecting them one at a time… You could do this all natively in PowerCLI, but that would require you to fully remove and then add the hosts.  That is very inconvenient, and almost as much trouble as doing it by hand… In this case it is both faster and easier to just use the native vSphere API.

# Get the hostsystem object for every host currently disconnected.
$VMhosts = Get-View -ViewType 'Hostsystem' `
                    -Property 'name' `
                    -Filter @{"Runtime.ConnectionState"="disconnected"}            

Foreach ($VMhost in $VMhosts)
{
    # Create a reconnect spec
    $HostConnectSpec = New-Object VMware.Vim.HostConnectSpec
    $HostConnectSpec.hostName = $VMhost.name
    $HostConnectSpec.userName = 'root'
    $HostConnectSpec.password = 'PassWord'            

    # Reconnect the host
    $taskMoRef = $VMhost.ReconnectHost_Task($HostConnectSpec)            

    # optional, but i like to return a task object, that way I can 
    # easily integrate this into a pipeline later if need be.
    Get-VIObjectByVIView -MORef $taskMoRef
}

~Glenn

While migrating a small environment to vSphere today I ran into my nemesis Host Profiles again. When are they going to Fix these things? The fact that they are incapable of even rudimentary iSCSI configuration is embarrassing. I’m sure vmware will fix it, but until then I wrote a simple one-liner that will configure iSCSI on a new host.

$VMhost = Get-VMhost 'ESX01'
$ChapUserName = 'vmware'
$ChapPassword = 'password'
$SendTargets = '192.168.1.1'            

# Enable the software ISCSI adapter if not already enabled.
$VMHostStorage = Get-VMHostStorage -VMHost $VMhost | Set-VMHostStorage -SoftwareIScsiEnabled $True            

#sleep while iSCSI starts up
Start-Sleep -Seconds 30            

# By default vSphere will set the Target Node name to iqn.1998-01.com.vmware:<HostName>-<random number> 
# This script will remove everything after the hostname, set Chap auth, and add a send Target.
#
# Example iqn.1998-01.com.vmware:esx01-165435 would become iqn.1998-01.com.vmware:esx01
$VMHostHba = Get-VMHostHba -VMHost $VMHost -Type IScsi |
    Where-object { $_.IScsiName -match "(?<IQN>iqn.1998-01.com.vmware\:[^-]+)"} |
    Set-VMHostHba -IScsiName $Matches.IQN |
    Set-VMHostHba -ChapName $ChapUserName -ChapPassword $ChapPassword -ChapType "Required" |
    New-IScsiHbaTarget -Address $SendTargets -Port "3260"                

#restart the host to make sure everything took
Restart-VMHost -VMHost $VMHost -Confirm:$false | out-null

~Glenn

I’ve ran across this particular issue myself, and submitted a bug to the PowerCLI team, but shortly after Andrew posted his ESXi 4.0 autoinstall Tim asked about this very issue.  There is a documentation error in Example #5 from the Apply-VMHostProfile cmdlet help.  Which contains the following code example.

$profile = Get-VMHostProfile -Name testProfile            

$additionalConfiguration = Apply-VMHostProfile -ApplyOnly -Profile $profile -Entity 10.23.114.166
$additionalConfiguration['network.hostPortGroup["key-vim-profile-host-HostPortgroupProfile-VMkernel"].ipConfig.IpAddressPolicy.address'] = '10.0.0.128'
$additionalConfiguration['network.hostPortGroup["key-vim-profile-host-HostPortgroupProfile-VMkernel"].ipConfig.IpAddressPolicy.subnetmask'] = '255.255.255.0'            

Apply-VMHostProfile -ApplyOnly -Profile $profile -Entity 10.23.114.166 -Variable $additionalConfiguration

Sadly if you tried to execute the above you would get the following error.

PS > $additionalConfiguration['network.hostPortGroup["key-vim-profile-host-HostPortgroupProfile-VMkernel"].ipConfig.Ip
AddressPolicy.address'] =  "10.52.8.11"
Array assignment to [network.hostPortGrou ..] failed: Cannot convert   value "network.hostPortGroup["key-vim-profile-
host-HostPortgroupProfile-VMkernel"].ipConfig.IpAddressPolicy.address" to   type "System.Int32". Error: "Input string
was not in a correct format.".
At line:1 char:26
+ $additionalConfiguration[ <<<<  '"network.hostPortGroup["key-vim-profile-host-HostPortgroupProfile-VMkernel"].ipCo
nfig.IpAddressPolicy.address'] ='10.52.8.11'
   + CategoryInfo          : InvalidOperation: (10.52.8.11:String) [],   RuntimeException
   + FullyQualifiedErrorId : ArrayAssignmentFailed 

 

At first this may appear a little cryptic, but it get’s a lot clearer once we inspect the object types in use.

PS > $additionalConfiguration.GetType()

IsPublic IsSerial Name                                     BaseType
-------- -------- ----                                     --------
True     True     Object[]                                 System.Object  

PS > $additionalConfiguration[0]

Name                           Value
----                           -----
network.hostPortGroup["key-...

PS > $additionalConfiguration[0].GetType()

IsPublic IsSerial Name                                     BaseType
-------- -------- ----                                     --------
True     True     DictionaryEntry                          System.ValueType

The example from the help docs was apparently expecting a Hashtable to be returned from apply-VMhostProfile.  Instead we found an array of DictionaryEntry objects... hence the error.

There are two possible work around's we can employ until the PowerCLI team ships a fix.  The first one is complicated, but dynamic.

$VMHostProfile = Get-VMHostProfile -Name testProfile            

$additionalConfiguration = Apply-VMHostProfile -ApplyOnly -Profile $VMHostProfile -Entity 10.23.114.166
($additionalConfiguration | Where-Object {$_.Name -eq 'network.hostPortGroup["key-vim-profile-host-HostPortgroupProfile-VMkernel"].ipConfig.IpAddressPolicy.address'}).Value = '10.0.0.128'
($additionalConfiguration | Where-Object {$_.Name -eq 'network.hostPortGroup["key-vim-profile-host-HostPortgroupProfile-VMkernel"].ipConfig.IpAddressPolicy.subnetmask'}).Value = '255.255.255.0'            

Apply-VMHostProfile -ApplyOnly -Profile $VMHostProfile -Entity 10.23.114.166 -Variable $additionalConfiguration

I actually don't like this approach even though it's a modified version of the included example.  I prefer just a simple static Hashtable.

$VMHostProfile = Get-VMHostProfile -Name testProfile            

$additionalConfiguration = @{
    'network.hostPortGroup["key-vim-profile-host-HostPortgroupProfile-VMkernel"].ipConfig.IpAddressPolicy.address'    = '10.0.0.128'
    'network.hostPortGroup["key-vim-profile-host-HostPortgroupProfile-VMkernel"].ipConfig.IpAddressPolicy.subnetmask' = '255.255.255.0'
}
Apply-VMHostProfile -ApplyOnly -Profile $VMHostProfile -Entity 10.23.114.166 -Variable $additionalConfiguration

All in all, the HostProfile cmdlets are surprisingly complete, and I think the majority of the "issues" I've ran across are a result of the SDK itself.  The Host Profiles sections of the API just don't have the same fit and finish I've come to expect in a VMware API.

I'm sure carter and team will have this fixed in the next release, untill then...

~Glenn

I wrote this script about a year ago to deal with errant SMVI snapshots, and was drafting this blog post when my rss feed caught me off guard. It appears Matt Robinson has beat me to the punch line.   He has produced a Perl script that cleans up any leftover snapshots, but if you favor a PowerShell approach… I give you Remove-SMVISnapshots.
Continue Reading »

If you don’t know already PowerCLI now has two modes single and multiple.  It stands for exactly what you think it does.  In single mode when you execute a command PowerCLI runs that command against the server you’re connected too.  Multiple mode allows you to specify multiple vCenter/ESX/vSphere host, and when you execute a command it runs that command against every server you’ve specified! This had to be one painful feature to get right, but the PowerCLI team nailed it.

I’ll admit when I first played with it I thought I would never need/use multiple mode.  That is until our vCenter server was inadvertently shutdown instead of rebooted.  Normally this would lead to one of two out comes.
A.) forcefully register vCenter on the first host I hit and power it up.
B.) A twenty minute search for the host that has vCenter.

Well today I didn’t feel like doing either… On a whim I tried this new-fangled multiple connection thing… IT WORKED!

# Set PowerCLI to multiple
Set-PowerCLIConfiguration -DefaultVIServerMode Multiple -Confirm:$false
# Connect to every vSphere host in the cluster that contains vCenter
Connect-VIServer -User root -Password password -Server esx1,esx2,esx3,esx4,esx5,esx6,esx7,esx8
# Start vCenter
Get-VM vCenter01 | Start-VM

I’ve since wrapped all this up in a batch file and added it to our playbook for a lights out recovery of virtual center!

~Glenn

I use NetApp SnapManager for Virtual Infrastructure(SMVI) to back up 95% of my environment. SMVI isn’t perfect, and it’s kinda a pain, but it gives me the ability to back up 100-250 vm’s in less than 10min! Well the actual NetApp snapshot takes seconds. The rest of the time is spent waiting on ESX snapshots. The only real downside here is from time to time SMVI will fail to delete the ESX snapshots. I have been using PowerCLI to find and delete these snapshot, but the cmdlets are just too slow, for what I’m trying to do. I’ll post my Finalized SMVI cleanup script later. Until then, I give you finding snapshots really fast!

Continue Reading »

From time to time I try and update all bios/firmware/etc. While VMotion makes this all but a cinch, evacuating every host in succession puts quite a strain on DRS.   Therefore,  I *try* and patch everything when installing uudates.  I’ve developed my own little technique where I let VUM do it’s thing.  Catching the Host in the middle of the reboot, and update everything else.  While the actual update methodology changes from OEM to OEM.  I use the same simple little script for vSphere/ESX.  Basically, when I see the remediation task start for a host I kick this function off, “watching” that host, and walk away. Then when vCenter registers a VMHost reboot on the Host this little guy let’s me know.

Function WatchVMHostReboot {
    param(
        [VMware.VimAutomation.Client20.VMHostImpl]
        $VMhost
    )
    $running = $false
    $start = ("{0:MM/dd/yyyy}" -f [datetime]::now)
    while (-Not $running){
        $events = Get-VIEvent -Entity $VMHost `
            -Username com.vmware.vcIntegrity `
            -Types info `
            -Start $start |
                Where-Object {$_.fullFormattedMessage -match "reboot"}
        if ($events) {
            Write-Host "reboot on $($vmhost.Name)`a`a`a"
            $running = $True
        }
    }
}

In case you didn’t already know `a will cause your motherboard to beep.  That loud annoying  “I can hear it two cubicles over” beep!

~Glenn

My Virtual Infrastructure was recently audited.  As part of my preparation for said audit I needed to verify that several extra configuration Parameters were set on every VM. Nothing ground breaking, this has all been covered here, and here. So why the repost, well I’m obsessed with scaling! I don’t like doing anything that I can’t use to the nth degree. Having said that I found two simple tweaks that dramatically increased the performance of these scripts.

If you ever find yourself using where-object move back up the pipeline… can you use a filter instead? Here I dramatically improved performance by leveraging the built-in filter capabilities of Get-View. I was also able to crank it up by simply switching from the ReconfigVM method to the ReconfigVM_Task method. Unless your performing some serial action, always, always use the task method. Offloading the babysitting to vCenter just makes sense! Finally, I loath text files, especially when they create a needless dependencies. Here I use a simple hashtable to embed my configuration in the script it self.

I successfully used this script to update over 500 vm’s in less than 4min!  Now that is what I call scale!  I know the security experts our there would argue that this is meaningless, b/c of this or that… all I know is I passes my audit with flying colors (didn’t have one ding on a VM’s configuration).

$ExtraOptions = @{
    "isolation.tools.copy.disable"="true";
    "isolation.tools.paste.disable"="true";
    "isolation.tools.diskShrink.disable"="true";
    "isolation.tools.diskWiper.disable"="true";
    "isolation.tools.connectable.disable"="true";
    "isolation.tools.setGUIOptions.Enable"="false";
    "log.keepOld"="10";
    "log.rotateSize"="100000"
}
 
# build our configspec using the hashtable from above.  I prefer this
# method over the use of files b/c it has one less needless dependency.
$vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec
# note we have to call the GetEnumerator before we can iterate through
Foreach ($Option in $ExtraOptions.GetEnumerator()) {
    $OptionValue = New-Object VMware.Vim.optionvalue
    $OptionValue.Key = $Option.Key
    $OptionValue.Value = $Option.Value
    $vmConfigSpec.extraconfig += $OptionValue
}
# Get all vm's not including templates
$VMs = Get-View -ViewType VirtualMachine -Property Name -Filter @{"Config.Template"="false"}
 
# Do it!
foreach($vm in $vms){
    $vm.ReconfigVM_Task($vmConfigSpec)
}

~Glenn

I can honestly say that I capitalized on a once in a lifetime opportunity.  For what ever reason Dev day was small this year.  There where only around 300 of us on the Developer track, and while the superstars of Virtualization were all looping through PTAP sessions I was attending small 15 to 1 labs with the likes of Steve Jin, Scott Herold, Carter Shanklin, LucD, and Cody Bunch…  You could say I learned a thing or two!

I started Monday with DS-13 it was suppose to be an Introduction to the vSphere Webservices SDK .  Unfortunately system errors prevented Steve from giving his full presentation!  My first lab I spent 45m trying to log into my virtual desktop.  It wasn’t a total lost though as I had Access to one of the authoritative sources on the VI API!  Shortly after Steve’s session I got a little side tracked, and picked my schedule back up with DS-16.

DS-16 Extending PowerCLI to Enterprise Applications with Virtualization EcoShell (VESI) presented by Scott Herold.  This session proved to be my favorite from Monday, and ran an hour long (in a good way)!  Good news, Scott and his team have done some fantastic work.  He is attempting to develop on demand.  Meaning as a demand for a feature/need starts to bubble up from the community either from the VESI forums or the usual places.  Scott fills that gap with a custom script extending the PowerCLI, or by modifying the user interface itself, extending VESI to better match the needs of the virtual administrator.  An example of the latter was on display where the VESI team has added the ability to transform any data set into rich charts. An important distinction with VESI is it is meant to enable the Virtualization Administrator NOT the VI Admin!  VESI will have full support for any Hypervisor/mgmt framework that the community has demand for.  It will also encompass any peripheral components of the virtual world.  Providing easy to use and context relevant access to any pain point whether it be storage, Network, AD… What ever the community needs!

The cynic out there will ask okay what does Vizioncore get out of this?  the answer, A single pane of glass that encompasses the entire virtualization ecosystem.  Oh yeah, and that pane of glass, it will one day serve as the front end for all of Vizioncores products!  The question was asked about pricing, and Scott insists that “VESI is and always will remain free”.  They need this framework for there own internal roadmap.  It’s extension to the community as a whole in my opinion will garnish them nothing but good will, and a built in user base.  Your probably asking yourself where’s the bad?

Politics… anyone from the PowerShell community will immediately recognize the VESI interface.  It’s our old friend PowerGUI, I asked Scott why something new, why not just build on top of PowerGUI.  His answer was speed, the PowerGUI team has a product roadmap, and there users need different things then Scotts.  He used the upcoming charts feature as an example.  It could take PowerGUI 18 months to get charts on there roadmap. PowerGUI is already hard at work putting out other fires.  By Scott forking PowerGUI he created a divisions but that division purchased an independent product roadmap.  It’s this roadmap that is enabling him to move with the Virtualization Community.  The sad part to me I don’t believe the division was truly necessary.  Why Scotts team couldn’t just develop those same features, injecting them into PowerGUI as needed, and thereby enhancing both products at once… that can only be political.  We all know how software works. There is no technical reason preventing this.  Alas while I think a best of both worlds super PowerGUI would have been better for everyone.  I for one am glad to have VESI in our tool belt. If your new to PowerShell or the PowerCLI check it out as Carter put it “VESI is the onramp to PowerCLI and PowerShell Scripting”… Couldn’t agree more!

Finally I ended Monday with a session on VIX.  while there is some really cool stuff coming in VIX there has been no change for the PowerShell community.  The latest version of VIX shipped just last week, and sadly 1.7 still offers no .Net/vi sdk interfaces. The .COM interface is critically crippled if you want to use it with vSphere, and overall your still forced to provide a username/Password to the guest OS.  Alas it’s not nearly as bad as I made it out to be!

The 1.7 release added full support for vSphere 4.0, and the VIX team is currently evaluating SSIP/Certification based authentication for the guest.  As for how it will ultimately be extended into powershell it looks like either a .net class, or by extending the vi api.  Either way will be a win for powershell as we can easily extend either into first-class cmdlets!  The use case for VIX is a bit nitch, but when you need it nothing else will do!

An interesting tidbit if you’re super security cautious you can disable VIX by adding

“Guest.Command.Enabled”=”False”

to either the VM or the host.  Be aware that this WILL break upgrading of VMware tools, and Guest customization as they both use vix as the underlying technology!

So that’s Developer day at VMworld all in all I had a blast, and met the superstars of the VI API/vSphere automation community.  The Food was 10x better then what they’re serving here at VMworld, and I get a free license of vSphere!  All for $249 USD, if you’re interested in more advanced automation at the vi api level I highly recommend developer day. 

VMware this was a win, win… let’s try and keep it for the future.

~Glenn

More migration/upgrade stuff… As part of a larger migration strategy, I needed to migrate whole datastores. Literally Terabytes of VM data, while sVMotion did all the heavy lifting. I still needed a control script to manage this process for me. This script ran successfully for over three days on one of my larger datastores. Faithfully, moving my VM’s two by two!

$SourceDS = Get-Datastore 'VMData1'
$DestinationDS = Get-Datastore 'VMData0'
$MaxConcurrent = 2
#Keep at least 10% freespace in the datastore
$buffer = ($DestinationDS.CapacityMB * .1) * 1048576
$DestView = Get-View -VIObject $DestinationDS -Property Info
$SourceView = Get-View -VIObject $SourceDS -Property vm
 
While ($DestView.Info.FreeSpace -gt $buffer -and $SourceView.VM.count -ge 1)
{
    $SourceDS | Get-VM | Select-Object -First $MaxConcurrent | Move-VM -Datastore $DestinationDS -RunAsync | Wait-task
    $DestView.UpdateViewData("Info")
    $SourceView.UpdateViewData("Vm")
}

Honestly I completed this over a month ago, but I consider these little scripts I write to be too simple to post… My drafts folder is full of stuff like this. On the one hand I want to post for my own use, but I can search my drafts for that. So I ask you, do you get any value out of these one off solutions? Please, don’t pad my ego… If you don’t find this kind of stuff useful don’t pretend. I have my documentation, I want to know if you would like access to it?

See ya’ll at VMworld
~Glenn