I had some free time yesterday morning as I couldn’t sleep after the long weekend. I used the time to dig into into the DataOnTap PowerShell Toolkit.  I started with an easy port of one of Andrews performance monitoring scripts.  I won’t go into as it’s very straight forward, but I will say so far I am very pleased with the DataOnTAP toolkit. 

<#
.SYNOPSIS
    Get the different protocol latencies for a specified volume.
.DESCRIPTION
    Get the different protocol latencies for a specified volume.
.PARAMETER Volume
    Volume to retrieve the latency.
.PARAMETER Protocol
    Protocol to collect latency for valid values are 'all','nfs','cifs','san','fcp','iscsi'
.PARAMETER Interval
    The interval between iterations in seconds, default is 15 seconds
.PARAMETER Count
    the number of iterations to execute, default is infinant
.PARAMETER Controller
    NetApp Controller to query.
.EXAMPLE
    .Get-NaVolumeLatency.ps1 -Volume vol0 

    Get the average latency for all protocols on vol0
.EXAMPLE
    Get-NaVol | .Get-NaVolumeLatency.ps1 -Interval 5 -count 5 | ft

    Get the average latency for all protocols, all volumes, 5 samples, 5 seconds apart.
.EXAMPLE
    .Get-NaVolumeLatency.ps1 -Volume vol0 -protocol nfs

    Get the NFS latency for vol0
#>
[cmdletBinding()]
Param(
    [Parameter(Mandatory=$true,
        HelpMessage="Volume name to retrieve latency counters from.",
        ValueFromPipelineByPropertyName=$true,
        ValueFromPipeLine=$true
    )]
    [Alias("Name")]
    [string]
    $Volume
,
    [Parameter(Mandatory=$false)]
    [ValidateSet('all','nfs','cifs','san','fcp','iscsi')]
    [string]
    $Protocol='all'
,
    [Parameter(Mandatory=$false)]
    [int]
    $Interval=15
,
    [Parameter(Mandatory=$false)]
    [string]
    $count
,
    [Parameter(Mandatory=$false)]
    [NetApp.Ontapi.Filer.NaController]
    $Controller=($CurrentNaController)
)
Begin
{
    IF ($Protocol -eq 'all')
    {
       $Counters = @(
            @{
                Counter = 'read_latency'
                Base     = ''
                unit     = ''
            }
       ,
            @{
                Counter = 'write_latency'
                Base     = ''
                unit     = ''
            }
       ,
            @{
                Counter = 'other_latency'
                Base     = ''
                unit     = ''
            }
       ,
            @{
                Counter = 'avg_latency'
                Base     = ''
                unit     = ''
            }
        )
    }
    Else
    {
        $Counters = @(
            @{
                Counter  = "$($Protocol.ToLower())_read_latency"
                Base     = ''
                unit     = ''
            }
        ,
            @{
                Counter = "$($Protocol.ToLower())_write_latency"
                Base     = ''
                unit     = ''
            }
        ,
            @{
                Counter = "$($Protocol.ToLower())_other_latency"
                Base     = ''
                unit     = ''
            }
        )
    }
    foreach ($c in $Counters)
    {
        Get-NaPerfCounter -Name 'volume' -Controller $Controller |
            Where-Object {$_.name -eq $c.Counter} |
            ForEach-Object {
                $c.Base = $_.BaseCounter
                $c.unit = switch ($_.unit) {
                    "microsec"  {10000}
                    "millisec"  {1}
                }
            }
    }
}
Process
{            

    # Check if volume exists.
    if (-Not ((get-navol -Controller $Controller|select -ExpandProperty Name) -contains $Volume)) {
        Write-Warning "$volume doesn't exist!"
        break;
    }
    $iteration = 0
    $first = $null
    #loop untill we're done or Cntr ^c 
    while ((!$Count) -or ($iteration -le $count))
    {
        $second = New-Object Collections.HashTable
        Get-NaPerfData -Name volume `
            -Instances $Volume `
            -Controller $Controller `
            -Counters ($Counters|%{$_.base,$_.counter}) |
            Select-Object -ExpandProperty Counters |
            ForEach-Object {
                $second.add($_.Name,$_.value)
            }            

        if ($first -and $second)
        {
            $results = "" | Select-Object -Property ($Counters|%{$_.base,$_.counter})
            foreach ($v in $Counters)
            {
                IF ($second[$v.Base] -gt $first[$v.Base])
                {
                    #calculate the average over our interval
                    $avg = ($second[$v.Counter] - $first[$v.Counter])/($second[$v.Base] - $first[$v.Base])
                    #conver to ms
                    $results."$($v.Base)" = [math]::Round((($second[$v.Base] - $first[$v.Base])/$Interval))
                    $results."$($v.Counter)" = ("{0} ms" -f [math]::Round($avg/$v.unit))
                }
                Else
                {
                    $results."$($v.Base)" = 0
                    $results."$($v.Counter)" = "0 ms"
                }
            }
            Write-Output $results| Add-Member NoteProperty 'Volume' $Volume -PassThru
        }
        Start-Sleep -Seconds $Interval
        $first = $second.clone()
        $iteration++
    }
}

~Glenn

NetApp’s SnapManager for Virtual Infrastructure (SMVI) is a great product, but it’s messy. If it encounters the any error, it seemingly forgets to delete the virtual machine snapshots from the Virtual Infrastructure before dying.

To prevent many orphans (I’ve seen as many as 20 on a single virtual machine) from happening, I created a quick Nagios check that simply alerts when it sees them.

This script is very elementary. It very simply uses a regex to check for any snapshots that match the default SMVI naming convention. For each one it finds, a counter is incremented. If any are found, the script returns an error to Nagios, which causes an alert to be sent.

#!/usr/bin/perl -w
#
# check_vi_smvi_snapshots.pl - written by Andrew Sullivan, 2010-06-16
#
# Please report bugs and request improvements at http://get-admin.com/blog/?p=1059
#
# A simple script to look for snapshots that match the name pattern that smvi uses.
# We are merely pulling a list of all snapshots, searching for the string "smvi" in 
# the name, if it's found, we return a warning condition.  This could lead to a 
# "false" positive if it runs while a snapshot series is still ongoing, but since
# the smvi snaps should be very short lived the condidition will not last unless
# the snap is left.
#
# Example:
#   ./check_vi_smvi_snapshots.pl --server your.esx.host --username you --password secret
#
 
use strict;
use warnings;
 
use FindBin;
use lib "$FindBin::Bin/../";
 
use VMware::VIRuntime;
 
# substitute the location of your nagios perl library
use lib "/usr/lib64/nagios/plugins";
use utils qw(%ERRORS);
 
Opts::parse();
Opts::validate();
 
Util::connect();
 
main();
 
Util::disconnect();
 
sub main {
 
	# the number of smvi snapshots
	my $smviSnaps = 0;
 
	# for setting the type of exit we want
	my $exitCondition = "";
 
	# we need MORs for each of the VMs on the host
	my $VMs = Vim::find_entity_views( view_type => 'VirtualMachine' );
 
	foreach my $vm (@$VMs) {
		if ($vm->snapshot) {			
			foreach my $childSnapshot (@{$vm->snapshot->snapshotInfo->rootSnapshotList}) {
				$smviSnaps += getSnaps($childSnapshot);
			}
 
		} else {
			#print $vm->name . " has no snapshots\n";
		}
	}
 
	if ($smviSnaps > 0) {
		print "WARNING - " . $smviSnaps . " SMVI snapshots exist.\n";
		$exitCondition = "WARNING";
 
	} else {
		print "OK - No SMVI snapshots exist.\n";
		$exitCondition = "OK";
 
	}
 
	Util::disconnect();
	exit $ERRORS{ $exitCondition };
}
 
sub getSnaps {
	my ($snapshotTree) = @_;
	my $snapcount = 0;
 
	# uncomment for debugging
	#print "Found snap: " . $snapshotTree->{name} . "\n";
 
	if ( $snapshotTree->{name} =~ /smvi/ ) {
		$snapcount++;
	}
 
	if ($snapshotTree->childSnapshotList) {
		foreach my $childSnapshot (@{$snapshotTree->childSnapshotList}) {
			$snapcount += getSnaps($childSnapshot);
		}
	}
 
	return $snapcount;
}

I’ve set the check to execute once an hour in my environment, as I don’t feel that granularity finer than that is needed…an hour’s worth of change is ok for an SMVI snapshot for me.

This is more for my own records.  I know for sure I will be using these as a reference over the next few month.  All my 2010 Scripting Games solutions.

• VBScript       – Beginner Event 1
• PowerShell – Beginner Event 1
• PowerShell – Beginner Event 2
• PowerShell – Beginner Event 3
• PowerShell – Beginner Event 4
• PowerShell – Beginner Event 5
• PowerShell – Beginner Event 6
• PowerShell – Beginner Event 7
• PowerShell – Beginner Event 8
• PowerShell – Beginner Event 9
• PowerShell – Beginner Event 10
• VBScript       – Advanced Event 1
• PowerShell – Advanced Event 1
• PowerShell – Advanced Event 2
• PowerShell – Advanced Event 3
• PowerShell – Advanced Event 4
• PowerShell – Advanced Event 5
• PowerShell – Advanced Event 6
• PowerShell – Advanced Event 7
• PowerShell – Advanced Event 8
• PowerShell – Advanced Event 9
• PowerShell – Advanced Event 10

~Glenn

I just submitted my last entry into this years games, and I wanted to capture a few things before I go and get busy. 

The Challenges

In years past the challenges where the equivalent of the scripting combine.  They would test ones ability to solve a logic problem in code. This created a developer friendly zone.  Whereby professional coders (aka developers) would inevitably produce a script that used some –xor feature or .net class I knew nothing about.  I would look back on my script feel incredibly inadequate and try harder.  This always lasted at most 5 days… until this year I never made it past 5 days.

This year the dread pirate Ed Wilson flipped the script. All of a sudden the event resembled a ticket at work.  As I read them I instantly knew how to do x or y, and would jump straight to the extra credit section.  This my friends is where it got interesting.  Every event this year could be solved by a PowerShell newbie, but to hit all the design criteria that would take some skill.

It was this combination that kept me in the games. Looking back it felt more like leveling in a MMO than work.  It wasn’t enough to solve the problem I wanted my stars.

PoshCode

There were a slew of issues with Poshcode V2 during the first couple days of the games.  They were fixed promptly, and the site has preformed admirably ever since.  This is the second year I’ve used PoshCode V2, and would just like to say…

JOEL… SHIP IT ALREADY…. ITS GOOD ENOUGH!!!!!1111

Seriously though, it is a massive upgrade, I’m excited to see what the poshcode.org upgrade may bring!

The Ratings

Judging criteria was put out here.  The short version is as follows.  If the script looks like it works you get two stars.  Every star beyond that is based on meeting the extra design criteria.  Now that is an incredibly open guideline…. and I am completely okay with it.

The same judges judge all the scripts so one liberal judge will give us all 5 stars.  The next judge might not like our approach and only give it three, but that stingy judge will be stingy to all the entries.  (I have some great scripts that got some whacked rankings.  Tell me how this only get’s three stars?)

That being said I would like to propose one change for next year.  We need feedback, why only four stars… did I miss something? Do you not like the formatting?  Did I go off the deep end for no reason and overcomplicated it?  All of these would be valid reasons to deduct points, but the deduction looses it’s real purpose if we can’t pass on the lesson it contains.

I’ll give you an example.  one of my close competitors this year kept creating the help by hand.  After day two or three I left a comment that said

hey great script, but you’re use of a here-string already locks you to V2.  Might as well go whole hog… check out help about_Comment_Based_Help

His very next script included comment based help.. heck he even gave me a shout out for the tip!  Awesome, this is what these games are about!

Except, that script didn’t rate too well… as of event 5 he was back to his old VBScript ways.  Somehow he associated that change with the bad rating… maybe it was part of it, he’s had all 5’s since.  Either way for on brief moment he was on track…

PowerShell Best practices

Which leads me to my final peeve about these games, and something we should fix for next year.  A script that is CLEARLY VBScript written in PowerShell should not get higher than a 3.  This isn’t about prizes or winning either.  These games are a learning event.  If we aren’t teaching best practices along the way what good are we doing anyone? You know simple stuff like don’t use the Scripting.FileSystemObject com object anymore…  I know this is a slippery slope, and I was truly blown away by the overall quality of the submissions. Nevertheless I think it’s a conversation worth having.

Motivation for the Games

As of this post I’m in first place… Looks like I might win this dang thing(what ever that means)!  Still way to early to call. Either way I didn’t participate this year for any monetary prize… I did it for the vanity of it all. I wanted to know where I stacked up.  I never thought I’d win anything… first place came out of left field.  All in all I got exactly what I wanted out of the games.  I do wish the judges had participated, they need only recuse themselves from the event they judge.  I would have liked to compete against a couple MVP’s.

New Tricks

These games forced me to step my game up.  I used EVERY trick I know, and I learned a few knew ones along the way.  Three specific tricks I will not soon forget.

Splatting:

I finally know what spatting is and why it’s a big deal.  Splatting helped me streamline half of my scripts removing 20-30 logic tree’s, by simply creating the parameter set’s upfront. I know we’ve all heard the definition before, but let me show you in code why it matters!

Before Spatting

Param(
    $Credential,
    $Computer = $ENV:COMPUTERNAME
)
End
{
    If ($credential)
    {
        Get-WmiObject win32_ComputerSystem -ComputerName $Computer -Credential $credential
    }
    Else
    {
        Get-WmiObject win32_ComputerSystem -ComputerName $Computer
    }
}

After Splatting

Param(
    $Credential,
    $Computer
)
Begin
{
    $extraparams=@{}
    If ($Computer)
    {
        $extraparams["ComputerName"] = $Computer
    }
    If ($credential)
    {
        $extraparams["Credential"] = $Credential
    }
}
End
{
    Get-WmiObject win32_ComputerSystem @extraparams
}

As you can see splatting allows you to offload all the logic that isn’t related to the task at hand to the the script/function initialization.  This cleans up your working area, and results in cleaner code!

Windows Forms:

Last week I created my very first windows form GUI ever, and I did it by hand.  Not because I’m hardcore, but because once I got started it was REALLY easy.  Like .net itself windows forms have full reflection, meaning it’s all self descriptive from within PowerShell.

Obviously using a tool like PrimalForms will save a ton of time, but don’t feel like you need a third party tool.  This stuff is easier than you’d think!

WPF/WPK:

This was the gut shot that hurt my brain!  WPF was a lot to bit off in one chunk.  After two weeks with it I feel like I still barel y know what I’m doing, but once I do… WPK will make it easy.  WPK’s strength, and weakness come from it’s meta-programming roots.

If you don’t already know, James wrote a script that wrote WPK.  It’s all machine generated code.

The good part, it is 100% complete it can do anything possible in WPF.

The bad part, there is NO abstraction layer between you and WPF.

There are some really cool helper functions that make it easy to use PowerShell code directly.  But if you want to make a listbox you’re going have to look at a c# example. I originally planned on doing event 10 with boots, and I wish I had time… I want to compare the two head to head, but I ran short and wanted to get my final submission in.

Final Thoughts

100% worth the time, These few weeks have sharpened my skill significantly.  To the point where I’ve actually caught up at work.  At this pace I’ll be ahead of schedule next week.  I think it was just the hard break the games carried… forced me out of my rut.

Take a look at my work let me know what you think. I have very thick skin and prefer the truth over candy coated fairy tales.

If you spent this year on the side line… save the excuses.  These games, like learning PowerShell will advance you professionally; ergo they’re good for work; ergo quit you’re bitchin and get to it!  Unless you were at EMC world ;P

In closing I want to personally thank the sponsors and judges who made the games possible this year:

Thank you, Tech·Ed 2010 North America
Thank you, PoshCode.org
Thank you, Microsoft TechNet
Thank you, SAPIEN Technologies
Thank you, Quest Software
Thank you, /n software
Thank you, Software FX
Thank you, 101FreeTechBooks.com
Thank you, ShellTools
Thank you, Idera
Thank you Judges

I had a blast a learned a ton, Good job all, See ya next year!

~Glenn

A couple a weeks ago I was hanging out in the PowerScripting Ustream waiting for the show to start. There were several of us caring on, when the interview subject Brandon Shell placed a call to scripting. He asked for a script to ping a subnet. There are many select-alive/ping-host scripts why write another one? The answer is simple pinging a target is easy, what Brandon was asking for was a fast way to generate the list of hosts. (Additionally, if you work with Active Directory then you can already see the usefulness in being able to ingest the subnets in sites and services as a parameter.) Shortly after Brandon made the request I opened my big mouth, and commented that the math was easy. The hard part is “threading”, as you know we can’t thread in PowerShell… but there are ways to perform concurrent operations. The PSJobs in V2 were purpose built for such a task. Unfortunately the *-job cmdlets use winrm as a backend, and require WS-MGMT to be installed/configured. As I was carrying on with this line of thought Brandon broke in, and informed me that the Get-WMIObject cmdlet in CTP3/V2 uses its own ‘backend’! He went on to inform me that all I needed was to tack on the –AsJob parameter… After a couple quick tests, I had a draft, and shortly thereafter a script!

Continue Reading »

I had the need to change the configuration of my ESX hosts so that the virtual switches had a single active and single standby adapter assigned to them. The reason for the need is rather irritating (the IBM I/O modules that we have in these particular blade centers are not really designed to handle a high amount of traffic), and it was causing some issues during vMotions.

This script allows me set the vmnics assigned to a vswitch to the desired active/standby configuration, and additionally allows me to set the port group’s vmnic active/standby policy. In my setup, I use two vSwitches, one for primary COS, vMotion and IP storage, and a second vSwitch for the virtual machines and secondary COS, each vSwitch has two NICs assigned (remember, they’re blade centers…limited network connectivity). In order to avoid vMotion taking all the bandwidth for storage I wanted to separate their traffic onto different NICs, but still provide redundancy.

The way that I accomplish this is by making the default for the vSwitch have, for example, vmnic0 active and vmnic2 standby. I then adjust the vMotion port group so that it has the opposite (vmnic2 active and vmnic0 standby). Redundancy is still present in the event of a NIC failure, but under normal circumstances, the traffic is separate.

Continue Reading »

The last few posts I’ve been mentioning how much of the configuration for my ESX hosts is automated. This post I’m going to talk a little more about how that automation is done, and provide an example kickstart script. I have been holding off on this post for a while now, as I have been planning on rebuilding my PXE server, at which point I will document each step and be able to provide a much more detailed post. However, things keep getting in the way and I haven’t had time to rebuild the server yet, so this is a slightly less detailed post, but should still be enough to get you on your way And I have no doubt that you, dear reader, are not afraid of asking questions in the comments…

This automation is currently handled (I say “currently” because I’m working to move the majority of it to a remote host and use the SDK) by kickstart when the system is loaded. Well, to be totally honest, kickstart only plays a partial role in the process…during the %post section of kickstart I copy a series of scripts from an NFS mount point into the startup process (/etc/init.d/rc3.d), which are executed at first boot and, like good one-time-only scripts, remove themselves.

This setup allows me to pxe boot a host, give it the boot command which has the host ID appended, and that’s it. I can then walk away and wait for the host to add itself to vCenter, indicating that it’s finished. Kickstart and the post install scripts then configure the hostname, ip, virtual network configuration, security policy, ntp, base user set, install any custom RPMs, etc. This makes it extremely easy for me to keep all of our hosts at the same configuration level.

In order to keep all the hosts the same I simply have to update the relevant post install script when we decide to make a global change and it will configure the host correctly the next time it is loaded/reloaded. For updating hosts that can’t be reloaded (I try to reload the hosts periodically with the newest binaries from VMware…every 4-6 months…so that the software is not a huge conglomeration of patches…I know, it’s unnecessary, but it gives me peace of mind) we use a combination of Glenn’s POSH prowess and the perl toolkit scripts I’ve created to remediate hosts en masse to our baseline configuration.

Continue Reading »

This morning I had a couple app servers just giving me fits. I turned to powershell to quickly to a diff on the servers. I started to backup the regisry and do a diff there, but decided that I should start somewhere simple. My solution was to “snap” the state of all the services on one that was working. I then restored those setting on one of the trouble nodes, rebooted problem solved.

gwmi win32_service | % { write-output "sc config $($_.name) start= $(($_.startmode).replace('Manual','demand')) " } | out-file restore_service.bat

I’ve been revamping our user account creation process lately (more on that when I finish it).  I started with the quest cmdlets, but performance/limitations lead me back to adsi.  Along the way I kept having to go from canonical naming (domain.com\ou\subou\sizemore, glenn) to the more common distinguished name.  After the third time I did this by hand i decided to script it.  I wrote the following functions to handle the conversion.

Note:  Should these be named ConvertTo-* or ConvertFrom-* ?