Really NetApp?!? You didn’t use your own SDK?

So, this post irked me. Not because of the poster or his post (honest Andy, if you ever read this, I have nothing against you or your post! I’m happy to see another VMware/NetApp blogger!), but because of the script he referenced and the problem encountered. He has a good solution, but the problem shouldn’t exist.

You see, I hate RSH. I don’t know why (well, it is quite insecure, and it can require some configuration), but I hate it. SSH is only marginally better in this case…sure it’s secure, but you have to auth each time, and if you don’t (ssh keys), well, it’s only a little better than RSH (comms are encrypted, but compromise of a single account can lead to bad things on many hosts). The script that is referenced, one that NetApp recommends that admins use to verify that their aggregates have enough free space to hold the metadata for the volumes in OnTAP 7.3 (the metadata gets moved from the volumes to the aggregate in 7.3), uses RSH to execute commands that are then parsed in a somewhat rudimentary way to get information.

Sure, it’s effective, but it’s far from graceful…especially when you have a perfectly good and effective SDK at your disposal.

I was kind of bored, so I decided to rewrite the script using the SDK. This is the end result. It reports the same data, but uses the SDK to gather all of the necessary information to make a determination for the user. The new script is significantly shorter (10KB vs 25KB, 380 lines vs 980), and it requires only one login.

Thanks to NetApp for providing their SDK, and I hope that no one over there minds me refactoring…

Continue Reading »

NetApp
Perl

Comments (2)

Permalink

Perl OnTAP SDK: Realtime Multiprotocol Volume Latency

My previous post only prints NFS latency for the NetApp as a whole, it doesn’t give any information about a specific volume. Some of my ESX hosts use iSCSI for their datastores, and because the NetApp has many iSCSI clients, looking at iSCSI stats for the filer as a whole didn’t help me very much.

The solution was this script. It is a significantly modified version of the previous script that is capable of showing the realtime latency for all protocols: NFS, CIFS, SAN (which I believe is all block level ops summarized), FCP and iSCSI. It also displays the three different types of operations for each protocol: read, write, and other.

The script, if invoked with nothing more than the connection information, will display the read, write, and “other” latency and operations for the total of all protocols. There is a fourth column as well, which shows the average latency and total operations across all operation types (r/w/o).

This script has proven quite beneficial for me. By monitoring CIFS latency during peak hours on the volume that contains shares for profiles, I have proven that the reason logins can take a significant amount of time is due to the use of high capacity, but very slow, SATA disks and not the network or desktops themselves. I’ve also been able to prove that one of our iSCSI volumes was “slow” due to bandwidth, and not spindle count (interestingly, the problem with this volume is the I/O request size…the app makes large requests which chokes bandwidth before available IOP/s runs out).

The OnTAP SDK is quite powerful, Glenn and I are quickly discovering that anything possible in FilerView and/or DFM/OpsMgr is doable through the SDK.

Continue Reading »

NetApp
Perl

Comments (0)

Permalink

Perl OnTAP SDK: Realtime NFS Latency

Since most of my Virtual Infrastructure runs on NFS datastores I like to keep a very close eye on what’s going on inside the NetApp. I generally use Cacti for long term monitoring of the status of the datastores and the NetApp as a whole.

However, when I want to see what’s going on in less than five minute increments, Cacti is pretty much useless. I wrote this script a while ago so that if I feel that latency is becoming a problem, I can check it right away and see it in frequent intervals.

Most often I use this script when Nagios starts to chirp at me. I use a slightly modified version of this script with Nagios and have it alert when latency gets out of hand. I then use this script to get a good look at what’s going on.

The OnTAP SDK is almost as entertaining to work with as VMware’s…

Continue Reading »

NetApp
Perl

Comments (1)

Permalink

RE: Post Your Powershell Profile!

I’m a little behind on my rss feeds, but this morning I came across this post… truthfully, I rewrite my profile every couple of months, but I’ll share the current iteration.

Continue Reading »

Powershell

Comments (0)

Permalink

PowerCLI: Get every VM added to vCenter in the last 30 Days!

At seemingly random intervals I get requests from management to report every Virtual Machine that was added/removed in the last x days. This has traditionally been a manual process, and one that I loath.

While deploying a VM this morning I set up my normal tell me when your done one-liner…

Get-Task|?{$_.Name -eq "CloneVM_Task"}|Wait-Task; `a`a`a

As I was typing I wondered how long a task’s history was kept, and if I could query it via the SDK? Low and behold not only can I but it’s too easy!

#requires -pssnapin VMware.Vimautomation.Core
Param(
    [int]
    $LastDays
)
Process
{
    $EventFilterSpecByTime = New-Object VMware.Vim.EventFilterSpecByTime
    If ($LastDays)
    {
        $EventFilterSpecByTime.BeginTime = (get-date).AddDays(-$($LastDays))
    }
    $EventFilterSpec = New-Object VMware.Vim.EventFilterSpec
    $EventFilterSpec.Time = $EventFilterSpecByTime
    $EventFilterSpec.DisableFullMessage = $False
    $EventFilterSpec.Type = "VmCreatedEvent","VmDeployedEvent","VmClonedEvent","VmDiscoveredEvent","VmRegisteredEvent"
    $EventManager = Get-View EventManager
    $NewVmTasks = $EventManager.QueryEvents($EventFilterSpec)
 
    Foreach ($Task in $NewVmTasks)
    {
        # If VM was deployed from a template then record which template.
        If ($Task.Template -and ($Task.SrcTemplate.Vm))
        {
 
            $srcTemplate = Get-View $Task.SrcTemplate.Vm -Property name | 
                Select -ExpandProperty Name
        }
        Else
        {
            $srcTemplate = $null
        }
        write-output ""|Select-Object @{
                Name="Name"
                Expression={$Task.Vm.name}
            }, @{
                Name="Created"
                Expression={$Task.CreatedTime}
            }, @{
                Name="UserName"
                Expression={$Task.UserName}
            }, @{        
                Name="Type"
                Expression={$Task.gettype().name}
            }, @{
                Name="Template"
                Expression={$srcTemplate}
            }
    } 
}

Enjoy,
~Glenn

PowerCLI
Powershell
VMware

Comments (0)

Permalink

PowerCLI Update VMware Tools without a reboot.

There appears to be a bug in the VMwareToolsUpgrader.exe that will cause the system to reboot no matter what. This behavior is especially bothersome because the VI API simply calls that executable to perform any non-interactive upgrade.

This morning I was updating a smaller farm to update 4. 15 minuets into the tools upgrade process I had enough… Enter PowerShell, I whipped up a quick script that will update All Windows VM’s to the latest version of VMware Tools.

$starttime = get-date
$GuestCred = Get-Credential
$OldTools = Get-View -ViewType "VirtualMachine" `
    -Property Guest,name `
    -filter @{
        "Guest.GuestFamily"="windowsGuest";
        "Guest.ToolsStatus"="ToolsOld";
        "Guest.GuestState"="running"
    }
 
Foreach ($VM in $OldTools) {
    # Mount the tools CD
    $VM.MountToolsInstaller()
 
    # Get the drive letter for the CD Drive tools is mounted in
    $DrvLetter = Get-WmiObject Win32_CDROMDrive -ComputerName $VM.name -Credential $GuestCred | 
        Where-Object {$_.VolumeName -match "VMware Tools"} | 
        Select-Object -ExpandProperty Drive
 
    # our update cmd
    $cmd = "$($DrvLetter)setup.exe /S /v`"/qn REBOOT=ReallySuppress REINSTALLMODE=vomus REINSTALL=ALL`""
 
    # Create a new process on the VM, and execute setup
    $go = Invoke-WMIMethod -path win32_process `
        -Name Create `
        -Credential $GuestCred `
        -ComputerName $VM.Name `
        -ArgumentList $cmd
 
    # If we sucessfully spawned an upgrade wait for it to finish
    IF ($go.ReturnValue -eq 0) 
    {
        $i=0
        While ($VM.Guest.ToolsStatus -ne 'toolsOk')
        {
            Start-Sleep -Seconds 1
            $VM.UpdateViewData("Guest")
            if ($I -gt 120)
            {
                Write-Warning "$($VM.name) appears to have hung, please investigate!"
                break
            }
            $i++
        }
        $vm.UnmountToolsInstaller()
        Write-verbose "$($VM.Name) Successfully updated"
    } 
    else 
    {
        Write-Warning "error $(go.ReturnValue) triggering tools install on $($VM.name). "
        $vm.UnmountToolsInstaller()
    }
}
$processed = $oldTools.count - (Get-View -ViewType "VirtualMachine" `
    -Property Guest,name `
    -filter @{
        "Guest.GuestFamily"="windowsGuest";
        "Guest.ToolsStatus"="ToolsOld";
        "Guest.GuestState"="running"
    }).count
$ts = New-TimeSpan -Start $starttime
write-host ("{0} out of {1} vm's were updated in {2}.{3} Min" -f $processed, $oldTools.count, $ts.Minutes,$ts.Seconds)

Using this script I updated 50 Vm’s in 40min. There is alot of room for improvement here. I opted for the slow and steady pace, but larger environments could easily tune this to 100 at a time.

~Glenn

P.S. I started with Invoke-VMScript but only half of the vm’s had PowerShell installed, and I needed something that would hit them all.

PowerCLI
Powershell

Comments (1)

Permalink

PowerCLI SpeedBoost: Advanced functionality hidden within Get-View

Everyone knows how to use the Get-View cmdlet to retrieve the full managed object from a VI implementation (Impl) object, but did you know you can skip the Impl entirely? For instance have you ever needed the following?

$VM = Get-VM | Get-View

If so you may want to try:

$VM = Get-View -ViewType "VirtualMachine"

I’ve found the latter to be around 1200% faster!  Additionally, by selecting the properties of interest at run time  an additional performance boost can be obtained.  For example suppose you just wanted the Name of every ESX host in vCenter.

$VMHostNames= Get-View -ViewType "HostSystem" -Property Name |
    Select-Object -ExpandProperty name

is 300% faster then the traditional

$VMHostNames= Get-VMHost | Select-Object -ExpandProperty name

When you really need to optimize you VI Scripts just add some filters. For instance a 400% performance boost can be obtained by replacing,

$VM = Get-VM -Name "XP"

with

$VM = Get-View -ViewType "VirtualMachine" -Filter @{"Name"="XP"}

but what if you need the Impl object?  It’s still 150% faster to convert back from the managed object that to retrieve then to get the impl in the first place.

$VM = Get-View -ViewType "VirtualMachine" -Filter @{"Name"="XP"} |  Get-VIObjectByVIView

Finally the grand pooh-bah of hidden potential… nested parameters within filters!  Say you want to locate all windows vm’s that have out of date tools.

traditionally we would approach this with something like,

Get-VM |
    Where-Object {
        $_.PowerState -eq "PoweredON" `
        -and `
        $_.Guest.OSFullName -match "Windows"
    } |
    Get-View |
    where { $_.guest.toolsstatus -match "toolsOld" }

Wait for it….

Get-View -ViewType "VirtualMachine" `
    -filter @{
        "Guest.GuestFamily"="windowsGuest";
        "Guest.ToolsStatus"="ToolsOld";
        "Guest.GuestState"="running"
    }

Over 800% faster and WAY less code! Although using get-view to it’s full potential can seriously increase your scripts performance I would leave it in your scripts.   The cmdlets provided in the PowerCLI are built around an interactive experience, the SDK is NOT!  While it may be faster to get all vm’s with get-view. The information returned from Get-VM is more useful when working from the cmdline.

Happy Scripting,
~Glenn

Bonus:  The function I was writing when I discovered this enhanced functionality… that’s right it was *hidden* in the help.

Function Get-OldTools {
    Get-View -ViewType "VirtualMachine" `
        -property Guest, Name `
        -filter @{
            "Guest.GuestFamily"="windowsGuest";
            "Guest.ToolsStatus"="ToolsOld";
            "Guest.GuestState"="running"
        } |
        Select-Object -Property Name, @{
            Name="ToolsVersion"
            Expression={$_.Guest.ToolsVersion}
        }, MoRef | 
        Sort-Object ToolsVersion
}

PowerCLI
Powershell

Comments (5)

Permalink

PowerShell: Enough with the text files already

A note to all PowerShell enthusiast, PowerShell has made it, it’s on the map… A significant portion of that success is due to the power of the pipeline. It’s ability to act as the glue for any task.  Unfortunately, the  skill level required to implement this glue depends on the echo-system.  As a PowerShell enthusiast you are a member of this echo-system, and any code you write either helps or hurts this effort.  In short when posting code to the world show them what well written code looks like!  Writing a function that is hard set to read a list of servers from a text file is NOT good code!   Personally I think this should throw an error.

$Servers = (Read-Host -Prompt "Path to servers.txt")
Get-Content -Path $Servers | Foreach-Object {do-something}

Okay so I don’t approve of reading from a text file, so what’s my solution?  Remove that part of the logic entirely!  There are literally hundreds (perhaps thousands) of techniques that could be used to retrieve data for any given operation.  leave the getting to the end user.  If you must get the data then show a more practical approach.  Seriously,  it’s Windows PowerShell after all, that list of computers… chances are they’re all running windows… and there is a high probability that they belong to a domain! In your examples highlight extracting the data from AD.  After all would you really use static data in one of your scripts?

$SearchRoot = 'Get-Admin.Local/Servers'
$Servers = Get-QADComputer -SearchRoot $SearchRoot

If you just can’t use the quest cmdlets use ADSI anything but a text file. Another one of my favorites is importing a list of IP Addresses.  I know it’s incredibly easy to create a list of IP Addresses in Excel, but is that really the lesson?  Hey PowerShell is the most important thing since the mouse, but if you need 254 concurrently numbered IP’s get Excel…  just add the code

1..254 | Foreach {"192.168.1.$_"}

Even better highlight why it’s so important to always emit object.  Show the benefit of such an approach.

1..254| Select @{name='Destination';Expression={"192.168.1.$_"}}| Test-Connection

I know, I know it’s all been done before.  I know we already had the why you should emit object discussion, and the foreach construct vs foreach-object cmdlet debate, etc… believe it or not the community has grown ten fold since those discussion’s where closed.  While I don’t want to constantly rehash this stuff… I’m asking that we don’t perpetuate the problem by using BAD techniques when we know better.

If you don’t have time just omit it.   Hide that part from the point of the post.  There is nothing wrong with

param([string[]]$servers)
{
    Foreach($S in $Servers)
    {
        do-something $S
    }
}

After all the do-something is the important part!  Just leave populating $servers to someone else… Why does this matter?  Simple, you never know what someone will learn from your code.  I read PowerShell post all the time, and I am constantly learning something new.  Just not always what the author intended.  The point of post x was to teach me how to use widget x from PowerShell.  Only I found this $? thing (what is that, and what does it do?)…   The next guy finds the same post via google because there trying to figure out the foreach construct, and bamn that get-content server.txt just became part of his repertoire.  Chances are he never even got to the part for widget x.  Moral of the story it’s better to omit part of the solution then to publish a poor answer.

Another penalty imposed by this hard coded methodology is it totally breaks the has-a model PowerShell was built around.  requiring a text file with a list of server names is the text book definition of the is-a model.  lamen terms by hard coding such a critical portion of your code… by importing data from a file you limit the flexibility of  your code.

It’s not all bad though, a couple years ago there where 10 guys.  Ten people who taught all of us how to use PowerShell.  Today there are 10 new PowerShell bloggers every day, and it appears that every team in Redmond (except SCCM) has PowerShell on it’s road map. I’m not trying to be the code police, but can we at least loose the text files?

~Glenn

Powershell

Comments (1)

Permalink

10th Powershell virtual usergroup announced

UPDATE: Due to a schedualing conflict the meeting has been moved to Thursday May 7th, 2009 at 8:00PM EST (lucky number 7 should be a good meeting)

Marco announced yesterday that the 10th Powershell Virtual User group will be held on Thursday April 30th, 2009 at 8:00PM EST.  On the schedule will be Bart De Smet (Microsoft), Sergei Anotonov (Microsoft), and Your’s truely!

I will be presenting Managing NetApp via Powershell, and plan on covering the OnTap SDK as well as my own PoshOnTap V2 module.

All the details can be found here

See you there
~Glenn

NetApp
Powershell

Comments (0)

Permalink

Cacti templates for NetApp’s CIFS, NFS, and iSCSI

I’m one of those people who has to know everything that is going on inside my infrastructure at all times. ESX, vCenter, MySQL, SQL Server, NetApp…I keep close tabs on all of them using both Cacti and Nagios.

Some people might find it strange that I have both running, but the two applications have very different functions. Cacti is superb at trend analysis and detecting abnormalities after the fact (only occasionally during the event). I use it’s data to determine, for example, when is a good time for Exchange to take an outage based on the number of users connected and the number of RPC requests occurring. Nagios, on the other hand, is extremely well suited to real time monitoring and alerting. It checks different data points at intervals and if it finds one out of the accepted range, it tells me.

Anyway, back to the point. I created these graphs to give me detailed information about CIFS, NFS and iSCSI on my NetApp filers. I have used these against FAS270s, FAS2020s and FAS6030s running OnTAP 7.2.1.1, 7.2.5.1 and/or 7.3.1 with success against them all.

In addition to the below templates I use some additional graphs to track other metrics. Those graphs are available here:

As I get time (which is rare) I plan on adding additional graphs, when I do, I will post them here. I would like to get and graph ASIS information, WAFL stats, and space information (raw, formatted, usable, allocated, overhead for the filer as a whole). If anyone has, or knows where to find, these graphs, please let me know!

Continue Reading »

NetApp

Comments (1)

Permalink