I won’t attempt to provide a feature rundown or tell you why vSphere 4.1 is the greatest thing since sliced bread.  It appears to be a solid release, but  I’ll leave that analysis to the experts…Instead I want to talk about the vSphere hypervisor (previously ESXi).

Why the name change? Simple what was previously mis-branded as a separate technology is really the hypervisors core.  Previously in ESX3.5, ESXi was a separate technology, but as of vSphere 4 they have had a unified core.   In-fact the product we like to think about as vSphere 4.0/4.1 is really just a vSphere hypervisor with a special management VM!  This is important, the only difference is the console which is nothing more than a VM!

So why the distinction, Why now?  VMware is playing it’s hand this round because that special VM is going bye, bye.  The Next release of vSphere will not have a service console.. PAINIC…. RUN IN CIRCLES THE ZOMBIES ARE COMMING!!!

Don’t Panic, Personally I applaud the move.  Over the past year and a half I’ve heard every argument against the console less hypervisor, but honestly I chalk it all up to people fear change.  There are a couple thousand admins who have invested a lot of time mastering vSphere, and VMware is about to change the whole game on them.  These guys/gals bring up several arguments against the console less hypervisor, I’ll attempt to offer my counter argument to these points.

Q. No 3rd Party agents.

A. It has been public knowledge that the console was going away, and as of vSphere 4.0 VMware shipped a new management appliance vMA.  One of the intended uses of this appliance was to install 3rd party agents.  So you see we do still have 3rd Party agents they just need to be rewritten.  In most cases this will result in a better product. Unfortunately, the vast majority of 3rd party software, could better be described as a really complex perl script running over ssh!

Q. Hardware monitors/plug-in

A. Part of the original ESXi 3.5 release was the introduction of a rudimentary CIM provider.  This provider has been fully expanded , and made extensible.  While it is a change from the traditional agent based monitors CIM does fill in this gap.

Q. Automating common tasks.

A. As of vSphere 4.1 Tech Support Mode supports SSH, but you should really be using either PowerCLI or the vCLI!  While it is true that are still a couple of things that can only be done via the console.  I’m confident VMware will fix those gaps before putting the console out to pasture.

Q. Security

A. So this is the big one, and my personal pet peeve.  I’ve heard security experts bash the vSphere hypervisor claiming it was insecure.  I just don’t understand this stance, admittedly I’m no security expert.  I only work with the federal government in some of the most secure data centers in the world, but what do I know…

Let’s break this down shall we… The only difference is a VM.   Admittedly this VM has special connections into the vmkernal, but it’s still just a VM.  How exactly does the inclusion of a VM make the hypervisor more secure?  In my opinion the exclusion of this VM instantly increased the security posture of most organizations.  The reason for this simple, it was hard to properly harden the console.  Alternatively it was all too easy to open a critical security hole, and expose ones infrastructure with the console.

Yes you still have to do several things to really lock down the console less hypervisor, but it’s not nearly the feat the console once was.  In fact it’s simple;

1. Modify the Proxy.xml (turning off all unneeded web services, and make everything use https).
2. Enable Lockdown mode.
3. Physical security.

That’s it folks, that’s all it takes to secure the hypervisor.  There are a couple hundred other little things necessary to design a secure infrastructure, but as you can see the hypervisor is easy!  In fact I’m so confident in this I’m willing to hold a Bobby Flay style throw down.  If you have the means to provide a  pair of internet facing vSphere hosts. I’ll secure the console less hypervisor, we’ll get TexiWill to harden the legacy console based hypervisor, and then we’ll release the IP’s to the world.  Have at it, folks I bet the console less hypervisor holds up at least as long as the legacy hypervisor!

Why so brash? Well it will take an exploit to get in to the console less hypervisor, and any exploit will also be present in the legacy hypervisor.  The console less vSphere hypervisor without access to the physical host or vCenter there is simply no other way in.   Remember this isn’t Linux or BSD or UNIX… it’s vSphere it’s practicality firmware, and the whole point was to remove all that crap that weaken the security , and stability to begin with!

I really want to put this to bed!  Let’s develop the to do list for VMware.  The 10-20 things they need to fix before they can finally kill the console.  Then let’s collectively shut up about it.  It’s going to happen, and complaining with arbitrary little gripes… or demanding NDA meetings with engineers isn’t going to stop any of it.  The Task at hand is simple, weed out the crap, and focus on what needs to be fixed in vSphere v.Next.

If we missed something let us know in the comments.
~Glenn

NetApp’s SnapManager for Virtual Infrastructure (SMVI) is a great product, but it’s messy. If it encounters the any error, it seemingly forgets to delete the virtual machine snapshots from the Virtual Infrastructure before dying.

To prevent many orphans (I’ve seen as many as 20 on a single virtual machine) from happening, I created a quick Nagios check that simply alerts when it sees them.

This script is very elementary. It very simply uses a regex to check for any snapshots that match the default SMVI naming convention. For each one it finds, a counter is incremented. If any are found, the script returns an error to Nagios, which causes an alert to be sent.

#!/usr/bin/perl -w
#
# check_vi_smvi_snapshots.pl - written by Andrew Sullivan, 2010-06-16
#
# Please report bugs and request improvements at http://get-admin.com/blog/?p=1059
#
# A simple script to look for snapshots that match the name pattern that smvi uses.
# We are merely pulling a list of all snapshots, searching for the string "smvi" in 
# the name, if it's found, we return a warning condition.  This could lead to a 
# "false" positive if it runs while a snapshot series is still ongoing, but since
# the smvi snaps should be very short lived the condidition will not last unless
# the snap is left.
#
# Example:
#   ./check_vi_smvi_snapshots.pl --server your.esx.host --username you --password secret
#
 
use strict;
use warnings;
 
use FindBin;
use lib "$FindBin::Bin/../";
 
use VMware::VIRuntime;
 
# substitute the location of your nagios perl library
use lib "/usr/lib64/nagios/plugins";
use utils qw(%ERRORS);
 
Opts::parse();
Opts::validate();
 
Util::connect();
 
main();
 
Util::disconnect();
 
sub main {
 
	# the number of smvi snapshots
	my $smviSnaps = 0;
 
	# for setting the type of exit we want
	my $exitCondition = "";
 
	# we need MORs for each of the VMs on the host
	my $VMs = Vim::find_entity_views( view_type => 'VirtualMachine' );
 
	foreach my $vm (@$VMs) {
		if ($vm->snapshot) {			
			foreach my $childSnapshot (@{$vm->snapshot->snapshotInfo->rootSnapshotList}) {
				$smviSnaps += getSnaps($childSnapshot);
			}
 
		} else {
			#print $vm->name . " has no snapshots\n";
		}
	}
 
	if ($smviSnaps > 0) {
		print "WARNING - " . $smviSnaps . " SMVI snapshots exist.\n";
		$exitCondition = "WARNING";
 
	} else {
		print "OK - No SMVI snapshots exist.\n";
		$exitCondition = "OK";
 
	}
 
	Util::disconnect();
	exit $ERRORS{ $exitCondition };
}
 
sub getSnaps {
	my ($snapshotTree) = @_;
	my $snapcount = 0;
 
	# uncomment for debugging
	#print "Found snap: " . $snapshotTree->{name} . "\n";
 
	if ( $snapshotTree->{name} =~ /smvi/ ) {
		$snapcount++;
	}
 
	if ($snapshotTree->childSnapshotList) {
		foreach my $childSnapshot (@{$snapshotTree->childSnapshotList}) {
			$snapcount += getSnaps($childSnapshot);
		}
	}
 
	return $snapcount;
}

I’ve set the check to execute once an hour in my environment, as I don’t feel that granularity finer than that is needed…an hour’s worth of change is ok for an SMVI snapshot for me.

I’ve been out of touch for most of this week, having only been able to be connected for an extended period of time today, and from somewhere out in left field I received an extremely surprising email from Mr. John Troyer…

I have been named a 2010 vExpert! Words can not describe how honored I am to receive this designation, I feel truly humbled by the others that have received the award and I can only hope that when I grow up I can be like them.

Thank you again to John Troyer and his team for this privilege!

Andrew

I wrote this script about a year ago to deal with errant SMVI snapshots, and was drafting this blog post when my rss feed caught me off guard. It appears Matt Robinson has beat me to the punch line.   He has produced a Perl script that cleans up any leftover snapshots, but if you favor a PowerShell approach… I give you Remove-SMVISnapshots.
Continue Reading »

If you don’t know already PowerCLI now has two modes single and multiple.  It stands for exactly what you think it does.  In single mode when you execute a command PowerCLI runs that command against the server you’re connected too.  Multiple mode allows you to specify multiple vCenter/ESX/vSphere host, and when you execute a command it runs that command against every server you’ve specified! This had to be one painful feature to get right, but the PowerCLI team nailed it.

I’ll admit when I first played with it I thought I would never need/use multiple mode.  That is until our vCenter server was inadvertently shutdown instead of rebooted.  Normally this would lead to one of two out comes.
A.) forcefully register vCenter on the first host I hit and power it up.
B.) A twenty minute search for the host that has vCenter.

Well today I didn’t feel like doing either… On a whim I tried this new-fangled multiple connection thing… IT WORKED!

# Set PowerCLI to multiple
Set-PowerCLIConfiguration -DefaultVIServerMode Multiple -Confirm:$false
# Connect to every vSphere host in the cluster that contains vCenter
Connect-VIServer -User root -Password password -Server esx1,esx2,esx3,esx4,esx5,esx6,esx7,esx8
# Start vCenter
Get-VM vCenter01 | Start-VM

I’ve since wrapped all this up in a batch file and added it to our playbook for a lights out recovery of virtual center!

~Glenn

My Virtual Infrastructure was recently audited.  As part of my preparation for said audit I needed to verify that several extra configuration Parameters were set on every VM. Nothing ground breaking, this has all been covered here, and here. So why the repost, well I’m obsessed with scaling! I don’t like doing anything that I can’t use to the nth degree. Having said that I found two simple tweaks that dramatically increased the performance of these scripts.

If you ever find yourself using where-object move back up the pipeline… can you use a filter instead? Here I dramatically improved performance by leveraging the built-in filter capabilities of Get-View. I was also able to crank it up by simply switching from the ReconfigVM method to the ReconfigVM_Task method. Unless your performing some serial action, always, always use the task method. Offloading the babysitting to vCenter just makes sense! Finally, I loath text files, especially when they create a needless dependencies. Here I use a simple hashtable to embed my configuration in the script it self.

I successfully used this script to update over 500 vm’s in less than 4min!  Now that is what I call scale!  I know the security experts our there would argue that this is meaningless, b/c of this or that… all I know is I passes my audit with flying colors (didn’t have one ding on a VM’s configuration).

$ExtraOptions = @{
    "isolation.tools.copy.disable"="true";
    "isolation.tools.paste.disable"="true";
    "isolation.tools.diskShrink.disable"="true";
    "isolation.tools.diskWiper.disable"="true";
    "isolation.tools.connectable.disable"="true";
    "isolation.tools.setGUIOptions.Enable"="false";
    "log.keepOld"="10";
    "log.rotateSize"="100000"
}
 
# build our configspec using the hashtable from above.  I prefer this
# method over the use of files b/c it has one less needless dependency.
$vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec
# note we have to call the GetEnumerator before we can iterate through
Foreach ($Option in $ExtraOptions.GetEnumerator()) {
    $OptionValue = New-Object VMware.Vim.optionvalue
    $OptionValue.Key = $Option.Key
    $OptionValue.Value = $Option.Value
    $vmConfigSpec.extraconfig += $OptionValue
}
# Get all vm's not including templates
$VMs = Get-View -ViewType VirtualMachine -Property Name -Filter @{"Config.Template"="false"}
 
# Do it!
foreach($vm in $vms){
    $vm.ReconfigVM_Task($vmConfigSpec)
}

~Glenn

I can honestly say that I capitalized on a once in a lifetime opportunity.  For what ever reason Dev day was small this year.  There where only around 300 of us on the Developer track, and while the superstars of Virtualization were all looping through PTAP sessions I was attending small 15 to 1 labs with the likes of Steve Jin, Scott Herold, Carter Shanklin, LucD, and Cody Bunch…  You could say I learned a thing or two!

I started Monday with DS-13 it was suppose to be an Introduction to the vSphere Webservices SDK .  Unfortunately system errors prevented Steve from giving his full presentation!  My first lab I spent 45m trying to log into my virtual desktop.  It wasn’t a total lost though as I had Access to one of the authoritative sources on the VI API!  Shortly after Steve’s session I got a little side tracked, and picked my schedule back up with DS-16.

DS-16 Extending PowerCLI to Enterprise Applications with Virtualization EcoShell (VESI) presented by Scott Herold.  This session proved to be my favorite from Monday, and ran an hour long (in a good way)!  Good news, Scott and his team have done some fantastic work.  He is attempting to develop on demand.  Meaning as a demand for a feature/need starts to bubble up from the community either from the VESI forums or the usual places.  Scott fills that gap with a custom script extending the PowerCLI, or by modifying the user interface itself, extending VESI to better match the needs of the virtual administrator.  An example of the latter was on display where the VESI team has added the ability to transform any data set into rich charts. An important distinction with VESI is it is meant to enable the Virtualization Administrator NOT the VI Admin!  VESI will have full support for any Hypervisor/mgmt framework that the community has demand for.  It will also encompass any peripheral components of the virtual world.  Providing easy to use and context relevant access to any pain point whether it be storage, Network, AD… What ever the community needs!

The cynic out there will ask okay what does Vizioncore get out of this?  the answer, A single pane of glass that encompasses the entire virtualization ecosystem.  Oh yeah, and that pane of glass, it will one day serve as the front end for all of Vizioncores products!  The question was asked about pricing, and Scott insists that “VESI is and always will remain free”.  They need this framework for there own internal roadmap.  It’s extension to the community as a whole in my opinion will garnish them nothing but good will, and a built in user base.  Your probably asking yourself where’s the bad?

Politics… anyone from the PowerShell community will immediately recognize the VESI interface.  It’s our old friend PowerGUI, I asked Scott why something new, why not just build on top of PowerGUI.  His answer was speed, the PowerGUI team has a product roadmap, and there users need different things then Scotts.  He used the upcoming charts feature as an example.  It could take PowerGUI 18 months to get charts on there roadmap. PowerGUI is already hard at work putting out other fires.  By Scott forking PowerGUI he created a divisions but that division purchased an independent product roadmap.  It’s this roadmap that is enabling him to move with the Virtualization Community.  The sad part to me I don’t believe the division was truly necessary.  Why Scotts team couldn’t just develop those same features, injecting them into PowerGUI as needed, and thereby enhancing both products at once… that can only be political.  We all know how software works. There is no technical reason preventing this.  Alas while I think a best of both worlds super PowerGUI would have been better for everyone.  I for one am glad to have VESI in our tool belt. If your new to PowerShell or the PowerCLI check it out as Carter put it “VESI is the onramp to PowerCLI and PowerShell Scripting”… Couldn’t agree more!

Finally I ended Monday with a session on VIX.  while there is some really cool stuff coming in VIX there has been no change for the PowerShell community.  The latest version of VIX shipped just last week, and sadly 1.7 still offers no .Net/vi sdk interfaces. The .COM interface is critically crippled if you want to use it with vSphere, and overall your still forced to provide a username/Password to the guest OS.  Alas it’s not nearly as bad as I made it out to be!

The 1.7 release added full support for vSphere 4.0, and the VIX team is currently evaluating SSIP/Certification based authentication for the guest.  As for how it will ultimately be extended into powershell it looks like either a .net class, or by extending the vi api.  Either way will be a win for powershell as we can easily extend either into first-class cmdlets!  The use case for VIX is a bit nitch, but when you need it nothing else will do!

An interesting tidbit if you’re super security cautious you can disable VIX by adding

“Guest.Command.Enabled”=”False”

to either the VM or the host.  Be aware that this WILL break upgrading of VMware tools, and Guest customization as they both use vix as the underlying technology!

So that’s Developer day at VMworld all in all I had a blast, and met the superstars of the VI API/vSphere automation community.  The Food was 10x better then what they’re serving here at VMworld, and I get a free license of vSphere!  All for $249 USD, if you’re interested in more advanced automation at the vi api level I highly recommend developer day. 

VMware this was a win, win… let’s try and keep it for the future.

~Glenn

So there I was in-between sessions just after lunch at VMworld when someone on twitter mentioned a VCP button.  Being a big fan of badge bling I showed the post to Andrew, and asked some guys around us If any of them knew anything about it.  One had seen the button, but didn’t know where you got one or why.  This is where my Monday got a little sidetracked.  At first we headed down to the materials pickup where I caught John Troyer stealing buttons!  But they directed us to the customer service desk.

Andrew and I proceeded to the customer services desk to inquire, and were quickly informed that we needed to be VCP4.  Honestly, that should have been the end of it, but for whatever reason we decided to walk over and look for ourselves.  Next thing I know I’ve slapped down my AMEX and I’m sitting behind a person vue terminal!  Why is this significant?

I’ve never touched vSphere, no beta, no RC access… Nothing.. Heck I didn’t even review the configuration maximums before hand!  I took it totally off the cuff no prep at all, and  passed!  I have my VCP on VI 3, and I’ve kept up with vSphere via our wonderful bloggers.  In a month or so I should be the proud owner of a VCP 4 certification.  At this point I need throw a huge thank you over to John Troyer, because while I’ve never touched these features I have listened to extensive interviews/discussions covering every aspect or vSphere.  In many ways the VMware Communities Roundtable Podcast was my “what’s new what’s changed” course.  In addition  to that overview our community in particular Duncan. Scott, Boche, etc provided me with the real life reference implementations.

What I don’t have is the hands on experience, I lack the practical knowledge that comes from performing a real implementation (A.K.A. the part that matters).  So there you have it, everyone complained about the need for the class, and VMware listened.  My question to you is… should I be able pass that test.  Knowing the whole background would YOU accept my updated credentials?  Something to think about next time we start complaining about the need for a class.  Personally, I never would have upgraded if it weren’t for the grace period, so I for one am grateful, but is this what we really want?

Perhaps we can agree that I’m just that good..? I would say that I benefited from the world class engineers I studied by proxy.  Personally I believe all of the above to be true.  Perhaps we could agree that the VCP isn’t that big of a deal, that It’s not a VCDX…I passed the “I know what these do in theory” test. Not the “I know what these do in practice” test. we would then agree that we would test for those skills further up the stack. 

As a Microsoft Professional I’m very familiar with that type of system, but is that what we want? In theory it sounds like a better system, but there is no end once you start down that road.  Microsoft recently had to enact a master certification to find the real masters.  Think about that for a second… why do you need a test to prove ‘this guy really really knows what he’s doing’ shouldn’t the engineer certification handle that?   I don’t know the answer I just wanted to ask the question?

~Glenn

Last year Andrew talked our employer into sending us to VMworld.   We followed up that amazing week with several whitepapers, and a year of being ahead of the curve.  Apparently we did something right because VMworld 2009 here we come!  Last year my goal for the show was to gather a better understanding for how other VI Administrators worked.  To assess where we were in both process, and execution.  This year I have a stack of stuff I want to get into.

Processes:
I want to find what others use for VM Request/Provisioning?  Lab manager Is great but that only covers my developers.  What about joe admin or a new customer? Is there anything that competes with lifecycle manager(Quest had to buy something ).  Additionally, we’ve seem to reach a critical mass, and charge back (at least the math behind it) will be of interest to me in particular.  While I don’t think we’ll ever charge per division, I would like to be able to show the cost in datacenter capacity every VM is consuming.

Technology:
I chose to skip the first round of VDI… It’s been a year time to look at VMware View.  What’s the report card is VDI ready for primetime, or is Citrix still my best bang for the buck, are there any new players I never heard of? Sticking with VDI, anything new in thin clients that I care about (doubtful), but then again you never know.  I also need to take a fresh look at I/O virtualization. I spoke with the NextIO guys last year, and left excited, but I never got to that part of our datacenter… Maybe this year.  All of that isn’t even addressing any storage, network, but I have a separate list for Cisco, and NetApp!

Dev Day:
I’m really looking forward to Monday.  You can keep your NDA, and partner track.  I want to spend a day with the other SDK/Automation geeks.  Perhaps I can find some fellow PowerShell enthusiast in the crowd.

People:
Finally, I’m hoping to meet you People!  Last year I was fairly new to VMware, and was rather uneasy about the whole thing.   A year later I’m much more familiar with the Virtualization Community, and I hope to get to meet you in person.  Especially, the PowerCLI enthusiast our there… not to be too cocky, but I called it, and while I’m no LucD… I know a thing or two about PowerShell, and would love to talk!

See ya at the show,
~Glenn

I recently assisted with a major hardware life cycle. We Migrated from Dell 1950′s to IBM BladeCenter H with H21XM blades. While the processors were comparable the RAM upgrade was tremendous. Going from 8/16GB to 32GB. This increase enabled much greater consolidation ratios, and with such we found ourselves with an abundance of hardware. After shutting down several hosts, and collapsing a few clusters we still weren’t driving our infrastructure. Shortly thereafter we realized that the resource limitations we had previously set. Were no longer necessary, and in some cases where decreasing performance (memory ballooning). A few minutes with the VI Toolkit for Windows, and we were screaming again!

# we'll use the same object to remove the limit
$ResourceAllocationInfo = New-Object VMware.Vim.ResourceAllocationInfo
$ResourceAllocationInfo.Limit = -1
 
# Get the Managed object for each VM
Foreach ($VM in get-vm | get-view) {
    # Create a fresh VirtualMachineConfigSpec
    $VirtualMachineConfigSpec = New-object VMware.Vim.VirtualMachineConfigSpec
    $i = 0
    # If the CPU Allocation is not unlimited add CPU to our spec.
    IF ($VM.ResourceConfig.CpuAllocation.Limit -ne -1) {
        $VirtualMachineConfigSpec.cpuAllocation = $ResourceAllocationIn
    	$i++
    }
    # If the Memory Allocation is not unlimited add Memory to our spec.
    IF ($VM.ResourceConfig.MemoryAllocation.Limit -ne -1) {
        $VirtualMachineConfigSpec.memoryAllocation = $ResourceAllocationInfo
        $i++
    }
    # If $I is gt 0 then we have a VM to fix... Trigger the reconfig task.
    IF ($I -gt 0) {
        write-verbose "Removing limits on $($VM.Name)"
        [void]$VM.ReconfigVM_Task($VirtualMachineConfigSpec)
    }
}

Now all that is left is for Andrew to translate this to perl/rCLI.
~Glenn