get-admin

First off, Happy VMworld everyone! It’s finally here, so let the socialization, learning and exhaustion begin.

I had a bit of free time this morning so I decided to do a lab. I didn’t arrive early enough to get in on the preview like some others, however even after seeing their posts about it the labs floor is quite impressive. From the seats arranged so that everyone can see the projectors, to the apparent “control center” in the middle, it’s an impressive setup.

I chose lab 20…logging in was painless and provisioning of the virtual machines was extremely quick. Access was also very good, initially anyway…more on that in a bit. Progress through the lab was quick, directions were good, screen shots were accurate and quickly identified the key fields that needed to have data entry done.

Overall, I really enjoyed the lab. In the last couple of years I haven’t done very many of them simply because most of the inability to get in. This year I had zero wait, and there were a few free seats around me. The dual monitor thin clients work well and PCoIP is amazing…

On that not so subtle transition, apparently the section I was in was running off the DC cloud. For the first 75% of the lab, I couldn’t tell…in fact, if the WAN (I assume it was the WAN) hadn’t of had some issues, I would have never known. Even when things started to go south and I could tell that latency was through the roof, the session was still usable, slow, but usable. At one point the client lost connectivity, however it quickly regained and started the session exactly where I left off.

I’m incredibly impressed with the labs…it’s my first real work done on a thin client, over PCoIP using a “cloud” infrastructure. If the provisioning of virtual machines works in the private cloud like it did for me today on VMware’s lab cloud, then we have a lot to look forward to in the future!

Outstanding work as always Labs Team, thank you!

This morning while standing up a new vScan A/V server I wanted to look up our McAfee service account.  I knew the account would be a domain account, and I knew it would be a member of the backup operators group on the filer.  With that in mind I ran the following.

[0:4]> Get-NaDomainUser -Group "Backup Operators"

SID
---
S-1-5-21-XXXXXXXX-XXXXXXXXX-XXXXXXXXX-112477
S-1-5-21-XXXXXXXX-XXXXXXXXX-XXXXXXXXX-111419
S-1-5-21-XXXXXXXX-XXXXXXXXX-XXXXXXXXX-146727

Well that’s rather useless… Unfortunately, the OnTAP API doesn’t provide a means to convert a SID to a NTAccount.  This is normally accomplished via the “cifs lookup” command on the Ontap CLI, but that doesn’t help us much from the toolkit.  Fortunately .Net provides a native means to perform this conversion.  This isn’t new to anyone who has been following PowerShell for a while (/\/\o\/\/ first posted these function way back in the Monad days), but that doesn’t make them any less useful!

Function ConvertTo-NTAccount
{
    Param(
        [Parameter(Mandatory=$true,
            HelpMessage="Enter the Sid to translate",
            ValueFromPipeLine=$true,
            ValueFromPipelineByPropertyName=$true
        )]
        [string]
        $SID
    )
    Process {
        $SIDObject = New-Object system.security.principal.securityidentifier($SID)
        write-output $SIDObject.translate([system.security.principal.ntaccount])
    }
}
Function ConvertTo-SID
{
    Param(
        [Parameter(Mandatory=$true,
            HelpMessage="Enter the NTAccount to translate in the form of domain\account",
            ValueFromPipeLine=$true,
            ValueFromPipelineByPropertyName=$true
        )]
        [string]
        $NTAccount
    )
    Process {
        $NTAccountObject = New-Object system.security.principal.NtAccount($NTaccount)
        write-output $NTAccountObject.translate([system.security.principal.securityidentifier])
    }
}

Armed with my trusty functions Let's try this again!

[0:15]> Get-NaDomainUser -Group "Backup Operators" | ConvertTo-NTAccount

Value
-----
GetAdmin\svcAV
GetAdmin\svcBackup
GetAdmin\svcMNV

Now that’s more like it!  This is what I Love about powershell.  In the past I would have had to push back on my sales rep, who would have inturn pushed back on the development team.  fast forward a year, and maybe I would have a workaround.  Or I would have had to try and glue a couple third party exe together (yuck). With PowerShell if I don’t like something I simply extend it in script.  No development, nothing complicated, just a couple line of PowerShell.  Best of all I can then provide this to the vendor as a concreate example of what I want in the next release (hint hint NetApp cifs lookup needs to be in the SDK!)

It really is just great stuff.
~Glenn

I have made no secret that I use two applications daily to monitor my infrastructure: Nagios and Cacti. I have created a fair number of scripts (and hopefully publishing more soon) to help Nagios monitor the different parts of the infrastructure, however I haven’t published many of my Cacti scripts previously.

One of the most useful is the config that I use to monitor the different protocol stats for volumes. I created an indexed query so that the single script, and accompanying XML file, are capable of monitoring all the volumes, and I can select which graphs to create for each volume. The polling script is loosely based off of the multi-protocol realtime volume statistics script that I created some time ago.

Download the template and script(s) here.

Some examples…

Total Operations, Latency
  

CIFS Operations, Latency
  

NFS Operations, Latency
  

iSCSI Operations, Latency
  

My philosophy with Nagios checks, especially with the NetApp, is that unless there are extenuating circumstances then I want all volumes (or whatever may be being checked) to be checked equally and at the same time. This means I don’t want to have to constantly add and remove checks from Nagios as volumes are added, deleted and modified. I would much rather have one check that checks all of the volumes and reports on them en masse. This means I don’t have to think about the check itself, but rather, only what it’s checking.

One of the many things that I regularly monitor on our multitude of NetApp systems is snapshots. We have had issues, especially with LUNs, where the snapshots have gotten out of control.

In order to prevent this, or at least hope that someone is watching the screen…, I wrote a quick script that checks to see if the total size of snapshots on a volume exceed the snap reserve. Since not all of our volumes have a snap reserve, I also put in the ability to check the size of the snaps against the percentage of free space left in the volume.

This last measure is a little strange, but I think it works fairly well. Take, for example, a 100GB volume. If it is 50% full (50GB), there is no snap reserve and the alert percentage is left at the default of 40% free space, then the alert will happen when snapshots exceed about 15GB. “But that’s not 40% of the free space”, I hear you saying. Ahhh, but it is…you see as the snapshot(s) grow, there is less free space, which means that it takes a larger percentage as the free space shrinks. So at 15GB of snapshots, there would be 35GB of free space, and 40% of 35GB is 14GB.

This causes the alerts to happen earlier than you may expect at first. You can adjust this number to be a percentage of the total space in the volume if you like…however, why not just set a snap reserve at that point? I chose to make the script this way in order to attempt to keep a little more free space in the volume, while not making a snap reserve mandatory.

One last word…please keep in mind this script does not check for a volume being filled, you should have other checks for that. This merely checks to see if snapshots have exceeded a threshold of space in the volume to prevent them from taking up too much space.

Bring on the Perl…

Continue Reading »

VMware is a facilitator. I know, you’re thinking “yeah, they facilitate my power/space/cooling savings, they facilitate infrastructure consolidation, IT agility, high availability, etc.”, but really, they facilitate me being lazy (which for sysadmins is a good thing…a lazy admin will only want to do a task once, then automate the sh*t out of it).

I’ve already documented how I hacked the ESXi 4.0 installer to have it do the installation without interaction. However, VMware has one-upped me and integrated kickstart into their installer. This makes things VASTLY easier, requires no tomfoolery with the ISO, and is significantly more capable.

This blog post will be just a short one to demonstrate how easy it is now to have the install be “touch-free”. I am working on some more complex examples in the coming days.

So without further blithering from me, on with the install! Put the CD in the drive (or mount the iso remotely), boot the server. When it reaches the boot options menu:

Press tab to append options to the boot line. Append the following after the vmkboot.gz, but before the --- after it.

ks=file://etc/vmware/weasel/ks.cfg

It is VERY IMPORTANT that you place the kickstart file location after vmkboot.gz, but before the next boot module. It should not be at the end.

mboot.c32 vmkboot.gz ks=file://etc/vmware/weasel/ks.cfg --- vmkernel.gz ---sys.vgz ---cim.vgz --- ienviron.vgz --- install.vgz

Here is an example:

When you’re done, press enter. It will begin to load the data off the CD, and when the different install modules are done it should simply begin to install ESXi just like how I had hacked it together previously…

The only thing left will be to press “Enter” when it’s done (why?!).

A word of caution…the kickstart that VMware has provided will automatically select and format the first disk that it finds, regardless of it being local or “remote” (i.e. a SAN LUN). I would assume that the vast majority of the time it’s going to find the local disks first, however……..

Hopefully in the next few days I’ll have some more time to play with the new kickstart features and post some more examples. VMware has really done some great things with this process and it is now possible to have the entire process be automated…
1) use DHCP to provide the “permanent” IP
2) use a network PXE boot for the media and to provide a KS file
3) use the --post section of the kickstart file to have the server reach out and touch a vCLI or PowerCLI configuration host and provide permanent configuration.

The reason that step one should provide the actual IP is that it provides an easy way of having your configuration host (vCLI or PowerCLI) know what IP (and potentially hostname) to assign to the host.

Good luck, and thanks to VMware for (finally) integrating kickstart with ESXi!

I won’t attempt to provide a feature rundown or tell you why vSphere 4.1 is the greatest thing since sliced bread.  It appears to be a solid release, but  I’ll leave that analysis to the experts…Instead I want to talk about the vSphere hypervisor (previously ESXi).

Why the name change? Simple what was previously mis-branded as a separate technology is really the hypervisors core.  Previously in ESX3.5, ESXi was a separate technology, but as of vSphere 4 they have had a unified core.   In-fact the product we like to think about as vSphere 4.0/4.1 is really just a vSphere hypervisor with a special management VM!  This is important, the only difference is the console which is nothing more than a VM!

So why the distinction, Why now?  VMware is playing it’s hand this round because that special VM is going bye, bye.  The Next release of vSphere will not have a service console.. PAINIC…. RUN IN CIRCLES THE ZOMBIES ARE COMMING!!!

Don’t Panic, Personally I applaud the move.  Over the past year and a half I’ve heard every argument against the console less hypervisor, but honestly I chalk it all up to people fear change.  There are a couple thousand admins who have invested a lot of time mastering vSphere, and VMware is about to change the whole game on them.  These guys/gals bring up several arguments against the console less hypervisor, I’ll attempt to offer my counter argument to these points.

Q. No 3rd Party agents.

A. It has been public knowledge that the console was going away, and as of vSphere 4.0 VMware shipped a new management appliance vMA.  One of the intended uses of this appliance was to install 3rd party agents.  So you see we do still have 3rd Party agents they just need to be rewritten.  In most cases this will result in a better product. Unfortunately, the vast majority of 3rd party software, could better be described as a really complex perl script running over ssh!

Q. Hardware monitors/plug-in

A. Part of the original ESXi 3.5 release was the introduction of a rudimentary CIM provider.  This provider has been fully expanded , and made extensible.  While it is a change from the traditional agent based monitors CIM does fill in this gap.

Q. Automating common tasks.

A. As of vSphere 4.1 Tech Support Mode supports SSH, but you should really be using either PowerCLI or the vCLI!  While it is true that are still a couple of things that can only be done via the console.  I’m confident VMware will fix those gaps before putting the console out to pasture.

Q. Security

A. So this is the big one, and my personal pet peeve.  I’ve heard security experts bash the vSphere hypervisor claiming it was insecure.  I just don’t understand this stance, admittedly I’m no security expert.  I only work with the federal government in some of the most secure data centers in the world, but what do I know…

Let’s break this down shall we… The only difference is a VM.   Admittedly this VM has special connections into the vmkernal, but it’s still just a VM.  How exactly does the inclusion of a VM make the hypervisor more secure?  In my opinion the exclusion of this VM instantly increased the security posture of most organizations.  The reason for this simple, it was hard to properly harden the console.  Alternatively it was all too easy to open a critical security hole, and expose ones infrastructure with the console.

Yes you still have to do several things to really lock down the console less hypervisor, but it’s not nearly the feat the console once was.  In fact it’s simple;

1. Modify the Proxy.xml (turning off all unneeded web services, and make everything use https).
2. Enable Lockdown mode.
3. Physical security.

That’s it folks, that’s all it takes to secure the hypervisor.  There are a couple hundred other little things necessary to design a secure infrastructure, but as you can see the hypervisor is easy!  In fact I’m so confident in this I’m willing to hold a Bobby Flay style throw down.  If you have the means to provide a  pair of internet facing vSphere hosts. I’ll secure the console less hypervisor, we’ll get TexiWill to harden the legacy console based hypervisor, and then we’ll release the IP’s to the world.  Have at it, folks I bet the console less hypervisor holds up at least as long as the legacy hypervisor!

Why so brash? Well it will take an exploit to get in to the console less hypervisor, and any exploit will also be present in the legacy hypervisor.  The console less vSphere hypervisor without access to the physical host or vCenter there is simply no other way in.   Remember this isn’t Linux or BSD or UNIX… it’s vSphere it’s practicality firmware, and the whole point was to remove all that crap that weaken the security , and stability to begin with!

I really want to put this to bed!  Let’s develop the to do list for VMware.  The 10-20 things they need to fix before they can finally kill the console.  Then let’s collectively shut up about it.  It’s going to happen, and complaining with arbitrary little gripes… or demanding NDA meetings with engineers isn’t going to stop any of it.  The Task at hand is simple, weed out the crap, and focus on what needs to be fixed in vSphere v.Next.

If we missed something let us know in the comments.
~Glenn

I had some free time yesterday morning as I couldn’t sleep after the long weekend. I used the time to dig into into the DataOnTap PowerShell Toolkit.  I started with an easy port of one of Andrews performance monitoring scripts.  I won’t go into as it’s very straight forward, but I will say so far I am very pleased with the DataOnTAP toolkit. 

<#
.SYNOPSIS
    Get the different protocol latencies for a specified volume.
.DESCRIPTION
    Get the different protocol latencies for a specified volume.
.PARAMETER Volume
    Volume to retrieve the latency.
.PARAMETER Protocol
    Protocol to collect latency for valid values are 'all','nfs','cifs','san','fcp','iscsi'
.PARAMETER Interval
    The interval between iterations in seconds, default is 15 seconds
.PARAMETER Count
    the number of iterations to execute, default is infinant
.PARAMETER Controller
    NetApp Controller to query.
.EXAMPLE
    .Get-NaVolumeLatency.ps1 -Volume vol0 

    Get the average latency for all protocols on vol0
.EXAMPLE
    Get-NaVol | .Get-NaVolumeLatency.ps1 -Interval 5 -count 5 | ft

    Get the average latency for all protocols, all volumes, 5 samples, 5 seconds apart.
.EXAMPLE
    .Get-NaVolumeLatency.ps1 -Volume vol0 -protocol nfs

    Get the NFS latency for vol0
#>
[cmdletBinding()]
Param(
    [Parameter(Mandatory=$true,
        HelpMessage="Volume name to retrieve latency counters from.",
        ValueFromPipelineByPropertyName=$true,
        ValueFromPipeLine=$true
    )]
    [Alias("Name")]
    [string]
    $Volume
,
    [Parameter(Mandatory=$false)]
    [ValidateSet('all','nfs','cifs','san','fcp','iscsi')]
    [string]
    $Protocol='all'
,
    [Parameter(Mandatory=$false)]
    [int]
    $Interval=15
,
    [Parameter(Mandatory=$false)]
    [string]
    $count
,
    [Parameter(Mandatory=$false)]
    [NetApp.Ontapi.Filer.NaController]
    $Controller=($CurrentNaController)
)
Begin
{
    IF ($Protocol -eq 'all')
    {
       $Counters = @(
            @{
                Counter = 'read_latency'
                Base     = ''
                unit     = ''
            }
       ,
            @{
                Counter = 'write_latency'
                Base     = ''
                unit     = ''
            }
       ,
            @{
                Counter = 'other_latency'
                Base     = ''
                unit     = ''
            }
       ,
            @{
                Counter = 'avg_latency'
                Base     = ''
                unit     = ''
            }
        )
    }
    Else
    {
        $Counters = @(
            @{
                Counter  = "$($Protocol.ToLower())_read_latency"
                Base     = ''
                unit     = ''
            }
        ,
            @{
                Counter = "$($Protocol.ToLower())_write_latency"
                Base     = ''
                unit     = ''
            }
        ,
            @{
                Counter = "$($Protocol.ToLower())_other_latency"
                Base     = ''
                unit     = ''
            }
        )
    }
    foreach ($c in $Counters)
    {
        Get-NaPerfCounter -Name 'volume' -Controller $Controller |
            Where-Object {$_.name -eq $c.Counter} |
            ForEach-Object {
                $c.Base = $_.BaseCounter
                $c.unit = switch ($_.unit) {
                    "microsec"  {10000}
                    "millisec"  {1}
                }
            }
    }
}
Process
{            

    # Check if volume exists.
    if (-Not ((get-navol -Controller $Controller|select -ExpandProperty Name) -contains $Volume)) {
        Write-Warning "$volume doesn't exist!"
        break;
    }
    $iteration = 0
    $first = $null
    #loop untill we're done or Cntr ^c 
    while ((!$Count) -or ($iteration -le $count))
    {
        $second = New-Object Collections.HashTable
        Get-NaPerfData -Name volume `
            -Instances $Volume `
            -Controller $Controller `
            -Counters ($Counters|%{$_.base,$_.counter}) |
            Select-Object -ExpandProperty Counters |
            ForEach-Object {
                $second.add($_.Name,$_.value)
            }            

        if ($first -and $second)
        {
            $results = "" | Select-Object -Property ($Counters|%{$_.base,$_.counter})
            foreach ($v in $Counters)
            {
                IF ($second[$v.Base] -gt $first[$v.Base])
                {
                    #calculate the average over our interval
                    $avg = ($second[$v.Counter] - $first[$v.Counter])/($second[$v.Base] - $first[$v.Base])
                    #conver to ms
                    $results."$($v.Base)" = [math]::Round((($second[$v.Base] - $first[$v.Base])/$Interval))
                    $results."$($v.Counter)" = ("{0} ms" -f [math]::Round($avg/$v.unit))
                }
                Else
                {
                    $results."$($v.Base)" = 0
                    $results."$($v.Counter)" = "0 ms"
                }
            }
            Write-Output $results| Add-Member NoteProperty 'Volume' $Volume -PassThru
        }
        Start-Sleep -Seconds $Interval
        $first = $second.clone()
        $iteration++
    }
}

~Glenn

If you have ever changed the vCenter server certificates, you’ve experienced having all your hosts disconnected from vCenter.  I couldn’t imagine reconnecting them one at a time… You could do this all natively in PowerCLI, but that would require you to fully remove and then add the hosts.  That is very inconvenient, and almost as much trouble as doing it by hand… In this case it is both faster and easier to just use the native vSphere API.

# Get the hostsystem object for every host currently disconnected.
$VMhosts = Get-View -ViewType 'Hostsystem' `
                    -Property 'name' `
                    -Filter @{"Runtime.ConnectionState"="disconnected"}            

Foreach ($VMhost in $VMhosts)
{
    # Create a reconnect spec
    $HostConnectSpec = New-Object VMware.Vim.HostConnectSpec
    $HostConnectSpec.hostName = $VMhost.name
    $HostConnectSpec.userName = 'root'
    $HostConnectSpec.password = 'PassWord'            

    # Reconnect the host
    $taskMoRef = $VMhost.ReconnectHost_Task($HostConnectSpec)            

    # optional, but i like to return a task object, that way I can 
    # easily integrate this into a pipeline later if need be.
    Get-VIObjectByVIView -MORef $taskMoRef
}

~Glenn

While migrating a small environment to vSphere today I ran into my nemesis Host Profiles again. When are they going to Fix these things? The fact that they are incapable of even rudimentary iSCSI configuration is embarrassing. I’m sure vmware will fix it, but until then I wrote a simple one-liner that will configure iSCSI on a new host.

$VMhost = Get-VMhost 'ESX01'
$ChapUserName = 'vmware'
$ChapPassword = 'password'
$SendTargets = '192.168.1.1'            

# Enable the software ISCSI adapter if not already enabled.
$VMHostStorage = Get-VMHostStorage -VMHost $VMhost | Set-VMHostStorage -SoftwareIScsiEnabled $True            

#sleep while iSCSI starts up
Start-Sleep -Seconds 30            

# By default vSphere will set the Target Node name to iqn.1998-01.com.vmware:<HostName>-<random number> 
# This script will remove everything after the hostname, set Chap auth, and add a send Target.
#
# Example iqn.1998-01.com.vmware:esx01-165435 would become iqn.1998-01.com.vmware:esx01
$VMHostHba = Get-VMHostHba -VMHost $VMHost -Type IScsi |
    Where-object { $_.IScsiName -match "(?<IQN>iqn.1998-01.com.vmware\:[^-]+)"} |
    Set-VMHostHba -IScsiName $Matches.IQN |
    Set-VMHostHba -ChapName $ChapUserName -ChapPassword $ChapPassword -ChapType "Required" |
    New-IScsiHbaTarget -Address $SendTargets -Port "3260"                

#restart the host to make sure everything took
Restart-VMHost -VMHost $VMHost -Confirm:$false | out-null

~Glenn

NetApp’s SnapManager for Virtual Infrastructure (SMVI) is a great product, but it’s messy. If it encounters the any error, it seemingly forgets to delete the virtual machine snapshots from the Virtual Infrastructure before dying.

To prevent many orphans (I’ve seen as many as 20 on a single virtual machine) from happening, I created a quick Nagios check that simply alerts when it sees them.

This script is very elementary. It very simply uses a regex to check for any snapshots that match the default SMVI naming convention. For each one it finds, a counter is incremented. If any are found, the script returns an error to Nagios, which causes an alert to be sent.

#!/usr/bin/perl -w
#
# check_vi_smvi_snapshots.pl - written by Andrew Sullivan, 2010-06-16
#
# Please report bugs and request improvements at http://get-admin.com/blog/?p=1059
#
# A simple script to look for snapshots that match the name pattern that smvi uses.
# We are merely pulling a list of all snapshots, searching for the string "smvi" in 
# the name, if it's found, we return a warning condition.  This could lead to a 
# "false" positive if it runs while a snapshot series is still ongoing, but since
# the smvi snaps should be very short lived the condidition will not last unless
# the snap is left.
#
# Example:
#   ./check_vi_smvi_snapshots.pl --server your.esx.host --username you --password secret
#
 
use strict;
use warnings;
 
use FindBin;
use lib "$FindBin::Bin/../";
 
use VMware::VIRuntime;
 
# substitute the location of your nagios perl library
use lib "/usr/lib64/nagios/plugins";
use utils qw(%ERRORS);
 
Opts::parse();
Opts::validate();
 
Util::connect();
 
main();
 
Util::disconnect();
 
sub main {
 
	# the number of smvi snapshots
	my $smviSnaps = 0;
 
	# for setting the type of exit we want
	my $exitCondition = "";
 
	# we need MORs for each of the VMs on the host
	my $VMs = Vim::find_entity_views( view_type => 'VirtualMachine' );
 
	foreach my $vm (@$VMs) {
		if ($vm->snapshot) {			
			foreach my $childSnapshot (@{$vm->snapshot->snapshotInfo->rootSnapshotList}) {
				$smviSnaps += getSnaps($childSnapshot);
			}
 
		} else {
			#print $vm->name . " has no snapshots\n";
		}
	}
 
	if ($smviSnaps > 0) {
		print "WARNING - " . $smviSnaps . " SMVI snapshots exist.\n";
		$exitCondition = "WARNING";
 
	} else {
		print "OK - No SMVI snapshots exist.\n";
		$exitCondition = "OK";
 
	}
 
	Util::disconnect();
	exit $ERRORS{ $exitCondition };
}
 
sub getSnaps {
	my ($snapshotTree) = @_;
	my $snapcount = 0;
 
	# uncomment for debugging
	#print "Found snap: " . $snapshotTree->{name} . "\n";
 
	if ( $snapshotTree->{name} =~ /smvi/ ) {
		$snapcount++;
	}
 
	if ($snapshotTree->childSnapshotList) {
		foreach my $childSnapshot (@{$snapshotTree->childSnapshotList}) {
			$snapcount += getSnaps($childSnapshot);
		}
	}
 
	return $snapcount;
}

I’ve set the check to execute once an hour in my environment, as I don’t feel that granularity finer than that is needed…an hour’s worth of change is ok for an SMVI snapshot for me.