get-admin

VMware is a facilitator. I know, you’re thinking “yeah, they facilitate my power/space/cooling savings, they facilitate infrastructure consolidation, IT agility, high availability, etc.”, but really, they facilitate me being lazy (which for sysadmins is a good thing…a lazy admin will only want to do a task once, then automate the sh*t out of it).

I’ve already documented how I hacked the ESXi 4.0 installer to have it do the installation without interaction. However, VMware has one-upped me and integrated kickstart into their installer. This makes things VASTLY easier, requires no tomfoolery with the ISO, and is significantly more capable.

This blog post will be just a short one to demonstrate how easy it is now to have the install be “touch-free”. I am working on some more complex examples in the coming days.

So without further blithering from me, on with the install! Put the CD in the drive (or mount the iso remotely), boot the server. When it reaches the boot options menu:

Press tab to append options to the boot line. Append the following after the vmkboot.gz, but before the --- after it.

ks=file://etc/vmware/weasel/ks.cfg

It is VERY IMPORTANT that you place the kickstart file location after vmkboot.gz, but before the next boot module. It should not be at the end.

mboot.c32 vmkboot.gz ks=file://etc/vmware/weasel/ks.cfg --- vmkernel.gz ---sys.vgz ---cim.vgz --- ienviron.vgz --- install.vgz

Here is an example:

When you’re done, press enter. It will begin to load the data off the CD, and when the different install modules are done it should simply begin to install ESXi just like how I had hacked it together previously…

The only thing left will be to press “Enter” when it’s done (why?!).

A word of caution…the kickstart that VMware has provided will automatically select and format the first disk that it finds, regardless of it being local or “remote” (i.e. a SAN LUN). I would assume that the vast majority of the time it’s going to find the local disks first, however……..

Hopefully in the next few days I’ll have some more time to play with the new kickstart features and post some more examples. VMware has really done some great things with this process and it is now possible to have the entire process be automated…
1) use DHCP to provide the “permanent” IP
2) use a network PXE boot for the media and to provide a KS file
3) use the --post section of the kickstart file to have the server reach out and touch a vCLI or PowerCLI configuration host and provide permanent configuration.

The reason that step one should provide the actual IP is that it provides an easy way of having your configuration host (vCLI or PowerCLI) know what IP (and potentially hostname) to assign to the host.

Good luck, and thanks to VMware for (finally) integrating kickstart with ESXi!

I won’t attempt to provide a feature rundown or tell you why vSphere 4.1 is the greatest thing since sliced bread.  It appears to be a solid release, but  I’ll leave that analysis to the experts…Instead I want to talk about the vSphere hypervisor (previously ESXi).

Why the name change? Simple what was previously mis-branded as a separate technology is really the hypervisors core.  Previously in ESX3.5, ESXi was a separate technology, but as of vSphere 4 they have had a unified core.   In-fact the product we like to think about as vSphere 4.0/4.1 is really just a vSphere hypervisor with a special management VM!  This is important, the only difference is the console which is nothing more than a VM!

So why the distinction, Why now?  VMware is playing it’s hand this round because that special VM is going bye, bye.  The Next release of vSphere will not have a service console.. PAINIC…. RUN IN CIRCLES THE ZOMBIES ARE COMMING!!!

Don’t Panic, Personally I applaud the move.  Over the past year and a half I’ve heard every argument against the console less hypervisor, but honestly I chalk it all up to people fear change.  There are a couple thousand admins who have invested a lot of time mastering vSphere, and VMware is about to change the whole game on them.  These guys/gals bring up several arguments against the console less hypervisor, I’ll attempt to offer my counter argument to these points.

Q. No 3rd Party agents.

A. It has been public knowledge that the console was going away, and as of vSphere 4.0 VMware shipped a new management appliance vMA.  One of the intended uses of this appliance was to install 3rd party agents.  So you see we do still have 3rd Party agents they just need to be rewritten.  In most cases this will result in a better product. Unfortunately, the vast majority of 3rd party software, could better be described as a really complex perl script running over ssh!

Q. Hardware monitors/plug-in

A. Part of the original ESXi 3.5 release was the introduction of a rudimentary CIM provider.  This provider has been fully expanded , and made extensible.  While it is a change from the traditional agent based monitors CIM does fill in this gap.

Q. Automating common tasks.

A. As of vSphere 4.1 Tech Support Mode supports SSH, but you should really be using either PowerCLI or the vCLI!  While it is true that are still a couple of things that can only be done via the console.  I’m confident VMware will fix those gaps before putting the console out to pasture.

Q. Security

A. So this is the big one, and my personal pet peeve.  I’ve heard security experts bash the vSphere hypervisor claiming it was insecure.  I just don’t understand this stance, admittedly I’m no security expert.  I only work with the federal government in some of the most secure data centers in the world, but what do I know…

Let’s break this down shall we… The only difference is a VM.   Admittedly this VM has special connections into the vmkernal, but it’s still just a VM.  How exactly does the inclusion of a VM make the hypervisor more secure?  In my opinion the exclusion of this VM instantly increased the security posture of most organizations.  The reason for this simple, it was hard to properly harden the console.  Alternatively it was all too easy to open a critical security hole, and expose ones infrastructure with the console.

Yes you still have to do several things to really lock down the console less hypervisor, but it’s not nearly the feat the console once was.  In fact it’s simple;

1. Modify the Proxy.xml (turning off all unneeded web services, and make everything use https).
2. Enable Lockdown mode.
3. Physical security.

That’s it folks, that’s all it takes to secure the hypervisor.  There are a couple hundred other little things necessary to design a secure infrastructure, but as you can see the hypervisor is easy!  In fact I’m so confident in this I’m willing to hold a Bobby Flay style throw down.  If you have the means to provide a  pair of internet facing vSphere hosts. I’ll secure the console less hypervisor, we’ll get TexiWill to harden the legacy console based hypervisor, and then we’ll release the IP’s to the world.  Have at it, folks I bet the console less hypervisor holds up at least as long as the legacy hypervisor!

Why so brash? Well it will take an exploit to get in to the console less hypervisor, and any exploit will also be present in the legacy hypervisor.  The console less vSphere hypervisor without access to the physical host or vCenter there is simply no other way in.   Remember this isn’t Linux or BSD or UNIX… it’s vSphere it’s practicality firmware, and the whole point was to remove all that crap that weaken the security , and stability to begin with!

I really want to put this to bed!  Let’s develop the to do list for VMware.  The 10-20 things they need to fix before they can finally kill the console.  Then let’s collectively shut up about it.  It’s going to happen, and complaining with arbitrary little gripes… or demanding NDA meetings with engineers isn’t going to stop any of it.  The Task at hand is simple, weed out the crap, and focus on what needs to be fixed in vSphere v.Next.

If we missed something let us know in the comments.
~Glenn

While migrating a small environment to vSphere today I ran into my nemesis Host Profiles again. When are they going to Fix these things? The fact that they are incapable of even rudimentary iSCSI configuration is embarrassing. I’m sure vmware will fix it, but until then I wrote a simple one-liner that will configure iSCSI on a new host.

$VMhost = Get-VMhost 'ESX01'
$ChapUserName = 'vmware'
$ChapPassword = 'password'
$SendTargets = '192.168.1.1'            

# Enable the software ISCSI adapter if not already enabled.
$VMHostStorage = Get-VMHostStorage -VMHost $VMhost | Set-VMHostStorage -SoftwareIScsiEnabled $True            

#sleep while iSCSI starts up
Start-Sleep -Seconds 30            

# By default vSphere will set the Target Node name to iqn.1998-01.com.vmware:<HostName>-<random number> 
# This script will remove everything after the hostname, set Chap auth, and add a send Target.
#
# Example iqn.1998-01.com.vmware:esx01-165435 would become iqn.1998-01.com.vmware:esx01
$VMHostHba = Get-VMHostHba -VMHost $VMHost -Type IScsi |
    Where-object { $_.IScsiName -match "(?<IQN>iqn.1998-01.com.vmware\:[^-]+)"} |
    Set-VMHostHba -IScsiName $Matches.IQN |
    Set-VMHostHba -ChapName $ChapUserName -ChapPassword $ChapPassword -ChapType "Required" |
    New-IScsiHbaTarget -Address $SendTargets -Port "3260"                

#restart the host to make sure everything took
Restart-VMHost -VMHost $VMHost -Confirm:$false | out-null

~Glenn

NetApp’s SnapManager for Virtual Infrastructure (SMVI) is a great product, but it’s messy. If it encounters the any error, it seemingly forgets to delete the virtual machine snapshots from the Virtual Infrastructure before dying.

To prevent many orphans (I’ve seen as many as 20 on a single virtual machine) from happening, I created a quick Nagios check that simply alerts when it sees them.

This script is very elementary. It very simply uses a regex to check for any snapshots that match the default SMVI naming convention. For each one it finds, a counter is incremented. If any are found, the script returns an error to Nagios, which causes an alert to be sent.

#!/usr/bin/perl -w
#
# check_vi_smvi_snapshots.pl - written by Andrew Sullivan, 2010-06-16
#
# Please report bugs and request improvements at http://get-admin.com/blog/?p=1059
#
# A simple script to look for snapshots that match the name pattern that smvi uses.
# We are merely pulling a list of all snapshots, searching for the string "smvi" in 
# the name, if it's found, we return a warning condition.  This could lead to a 
# "false" positive if it runs while a snapshot series is still ongoing, but since
# the smvi snaps should be very short lived the condidition will not last unless
# the snap is left.
#
# Example:
#   ./check_vi_smvi_snapshots.pl --server your.esx.host --username you --password secret
#
 
use strict;
use warnings;
 
use FindBin;
use lib "$FindBin::Bin/../";
 
use VMware::VIRuntime;
 
# substitute the location of your nagios perl library
use lib "/usr/lib64/nagios/plugins";
use utils qw(%ERRORS);
 
Opts::parse();
Opts::validate();
 
Util::connect();
 
main();
 
Util::disconnect();
 
sub main {
 
	# the number of smvi snapshots
	my $smviSnaps = 0;
 
	# for setting the type of exit we want
	my $exitCondition = "";
 
	# we need MORs for each of the VMs on the host
	my $VMs = Vim::find_entity_views( view_type => 'VirtualMachine' );
 
	foreach my $vm (@$VMs) {
		if ($vm->snapshot) {			
			foreach my $childSnapshot (@{$vm->snapshot->snapshotInfo->rootSnapshotList}) {
				$smviSnaps += getSnaps($childSnapshot);
			}
 
		} else {
			#print $vm->name . " has no snapshots\n";
		}
	}
 
	if ($smviSnaps > 0) {
		print "WARNING - " . $smviSnaps . " SMVI snapshots exist.\n";
		$exitCondition = "WARNING";
 
	} else {
		print "OK - No SMVI snapshots exist.\n";
		$exitCondition = "OK";
 
	}
 
	Util::disconnect();
	exit $ERRORS{ $exitCondition };
}
 
sub getSnaps {
	my ($snapshotTree) = @_;
	my $snapcount = 0;
 
	# uncomment for debugging
	#print "Found snap: " . $snapshotTree->{name} . "\n";
 
	if ( $snapshotTree->{name} =~ /smvi/ ) {
		$snapcount++;
	}
 
	if ($snapshotTree->childSnapshotList) {
		foreach my $childSnapshot (@{$snapshotTree->childSnapshotList}) {
			$snapcount += getSnaps($childSnapshot);
		}
	}
 
	return $snapcount;
}

I’ve set the check to execute once an hour in my environment, as I don’t feel that granularity finer than that is needed…an hour’s worth of change is ok for an SMVI snapshot for me.

I’ve been out of touch for most of this week, having only been able to be connected for an extended period of time today, and from somewhere out in left field I received an extremely surprising email from Mr. John Troyer…

I have been named a 2010 vExpert! Words can not describe how honored I am to receive this designation, I feel truly humbled by the others that have received the award and I can only hope that when I grow up I can be like them.

Thank you again to John Troyer and his team for this privilege!

Andrew

Being, first and foremost, lazy and getting my paychecks for being a system administrator, I felt that the amount of work involved in loading ESXi 4.0 on my blades was entirely too much. I have well over 100 blades, each one needing to have vSphere loaded onto it, configured, and added to vCenter. Even using the directions scattered across the internet about reducing the amount of effort involved in loading vSphere was too much for me.

Others have documented how to PXE boot ESXi elsewhere on the internet, however I wasn’t interested in having a “stateless” install…I merely wanted to automate installing ESXi to the local hard drive. My blades have a single hard drive, a single generation one SSD or two SAS drives in a RAID 1 depending on the vendor, and I simply want the installer to always install to that drive without bothering me. Loading from the “remote media” functionality of the DRAC/iLO for the blades takes forever, so I wanted to be able to install using PXE and push the media over that medium.

So, having been a developer for several years I decided to dive further into the the install process than others had detailed. Turns out that eliminating all input from an administrator to load the operating system was pretty simple.

The end result is that I am able to power on a blade, hit F12 to have it PXE boot and walk away. Some time later, we can use PowerShell and the PowerCLI to find the hosts (they will be somewhere in the DHCP scope of the provisioning LAN), give them a permanent IP and hostname, then configure them and add them to vCenter. By using PXE and the interactionless (yes, I did make up that word) install, I cut the time to load ESXi from about 45 minutes (using the remote media function takes FOREVER!) to less than 10.

Continue Reading »

I use NetApp SnapManager for Virtual Infrastructure(SMVI) to back up 95% of my environment. SMVI isn’t perfect, and it’s kinda a pain, but it gives me the ability to back up 100-250 vm’s in less than 10min! Well the actual NetApp snapshot takes seconds. The rest of the time is spent waiting on ESX snapshots. The only real downside here is from time to time SMVI will fail to delete the ESX snapshots. I have been using PowerCLI to find and delete these snapshot, but the cmdlets are just too slow, for what I’m trying to do. I’ll post my Finalized SMVI cleanup script later. Until then, I give you finding snapshots really fast!

Continue Reading »

From time to time I try and update all bios/firmware/etc. While VMotion makes this all but a cinch, evacuating every host in succession puts quite a strain on DRS.   Therefore,  I *try* and patch everything when installing uudates.  I’ve developed my own little technique where I let VUM do it’s thing.  Catching the Host in the middle of the reboot, and update everything else.  While the actual update methodology changes from OEM to OEM.  I use the same simple little script for vSphere/ESX.  Basically, when I see the remediation task start for a host I kick this function off, “watching” that host, and walk away. Then when vCenter registers a VMHost reboot on the Host this little guy let’s me know.

Function WatchVMHostReboot {
    param(
        [VMware.VimAutomation.Client20.VMHostImpl]
        $VMhost
    )
    $running = $false
    $start = ("{0:MM/dd/yyyy}" -f [datetime]::now)
    while (-Not $running){
        $events = Get-VIEvent -Entity $VMHost `
            -Username com.vmware.vcIntegrity `
            -Types info `
            -Start $start |
                Where-Object {$_.fullFormattedMessage -match "reboot"}
        if ($events) {
            Write-Host "reboot on $($vmhost.Name)`a`a`a"
            $running = $True
        }
    }
}

In case you didn’t already know `a will cause your motherboard to beep.  That loud annoying  “I can hear it two cubicles over” beep!

~Glenn

My Virtual Infrastructure was recently audited.  As part of my preparation for said audit I needed to verify that several extra configuration Parameters were set on every VM. Nothing ground breaking, this has all been covered here, and here. So why the repost, well I’m obsessed with scaling! I don’t like doing anything that I can’t use to the nth degree. Having said that I found two simple tweaks that dramatically increased the performance of these scripts.

If you ever find yourself using where-object move back up the pipeline… can you use a filter instead? Here I dramatically improved performance by leveraging the built-in filter capabilities of Get-View. I was also able to crank it up by simply switching from the ReconfigVM method to the ReconfigVM_Task method. Unless your performing some serial action, always, always use the task method. Offloading the babysitting to vCenter just makes sense! Finally, I loath text files, especially when they create a needless dependencies. Here I use a simple hashtable to embed my configuration in the script it self.

I successfully used this script to update over 500 vm’s in less than 4min!  Now that is what I call scale!  I know the security experts our there would argue that this is meaningless, b/c of this or that… all I know is I passes my audit with flying colors (didn’t have one ding on a VM’s configuration).

$ExtraOptions = @{
    "isolation.tools.copy.disable"="true";
    "isolation.tools.paste.disable"="true";
    "isolation.tools.diskShrink.disable"="true";
    "isolation.tools.diskWiper.disable"="true";
    "isolation.tools.connectable.disable"="true";
    "isolation.tools.setGUIOptions.Enable"="false";
    "log.keepOld"="10";
    "log.rotateSize"="100000"
}
 
# build our configspec using the hashtable from above.  I prefer this
# method over the use of files b/c it has one less needless dependency.
$vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec
# note we have to call the GetEnumerator before we can iterate through
Foreach ($Option in $ExtraOptions.GetEnumerator()) {
    $OptionValue = New-Object VMware.Vim.optionvalue
    $OptionValue.Key = $Option.Key
    $OptionValue.Value = $Option.Value
    $vmConfigSpec.extraconfig += $OptionValue
}
# Get all vm's not including templates
$VMs = Get-View -ViewType VirtualMachine -Property Name -Filter @{"Config.Template"="false"}
 
# Do it!
foreach($vm in $vms){
    $vm.ReconfigVM_Task($vmConfigSpec)
}

~Glenn

So there I was in-between sessions just after lunch at VMworld when someone on twitter mentioned a VCP button.  Being a big fan of badge bling I showed the post to Andrew, and asked some guys around us If any of them knew anything about it.  One had seen the button, but didn’t know where you got one or why.  This is where my Monday got a little sidetracked.  At first we headed down to the materials pickup where I caught John Troyer stealing buttons!  But they directed us to the customer service desk.

Andrew and I proceeded to the customer services desk to inquire, and were quickly informed that we needed to be VCP4.  Honestly, that should have been the end of it, but for whatever reason we decided to walk over and look for ourselves.  Next thing I know I’ve slapped down my AMEX and I’m sitting behind a person vue terminal!  Why is this significant?

I’ve never touched vSphere, no beta, no RC access… Nothing.. Heck I didn’t even review the configuration maximums before hand!  I took it totally off the cuff no prep at all, and  passed!  I have my VCP on VI 3, and I’ve kept up with vSphere via our wonderful bloggers.  In a month or so I should be the proud owner of a VCP 4 certification.  At this point I need throw a huge thank you over to John Troyer, because while I’ve never touched these features I have listened to extensive interviews/discussions covering every aspect or vSphere.  In many ways the VMware Communities Roundtable Podcast was my “what’s new what’s changed” course.  In addition  to that overview our community in particular Duncan. Scott, Boche, etc provided me with the real life reference implementations.

What I don’t have is the hands on experience, I lack the practical knowledge that comes from performing a real implementation (A.K.A. the part that matters).  So there you have it, everyone complained about the need for the class, and VMware listened.  My question to you is… should I be able pass that test.  Knowing the whole background would YOU accept my updated credentials?  Something to think about next time we start complaining about the need for a class.  Personally, I never would have upgraded if it weren’t for the grace period, so I for one am grateful, but is this what we really want?

Perhaps we can agree that I’m just that good..? I would say that I benefited from the world class engineers I studied by proxy.  Personally I believe all of the above to be true.  Perhaps we could agree that the VCP isn’t that big of a deal, that It’s not a VCDX…I passed the “I know what these do in theory” test. Not the “I know what these do in practice” test. we would then agree that we would test for those skills further up the stack. 

As a Microsoft Professional I’m very familiar with that type of system, but is that what we want? In theory it sounds like a better system, but there is no end once you start down that road.  Microsoft recently had to enact a master certification to find the real masters.  Think about that for a second… why do you need a test to prove ‘this guy really really knows what he’s doing’ shouldn’t the engineer certification handle that?   I don’t know the answer I just wanted to ask the question?

~Glenn