get-admin

VMware is a facilitator. I know, you’re thinking “yeah, they facilitate my power/space/cooling savings, they facilitate infrastructure consolidation, IT agility, high availability, etc.”, but really, they facilitate me being lazy (which for sysadmins is a good thing…a lazy admin will only want to do a task once, then automate the sh*t out of it).

I’ve already documented how I hacked the ESXi 4.0 installer to have it do the installation without interaction. However, VMware has one-upped me and integrated kickstart into their installer. This makes things VASTLY easier, requires no tomfoolery with the ISO, and is significantly more capable.

This blog post will be just a short one to demonstrate how easy it is now to have the install be “touch-free”. I am working on some more complex examples in the coming days.

So without further blithering from me, on with the install! Put the CD in the drive (or mount the iso remotely), boot the server. When it reaches the boot options menu:

Press tab to append options to the boot line. Append the following after the vmkboot.gz, but before the --- after it.

ks=file://etc/vmware/weasel/ks.cfg

It is VERY IMPORTANT that you place the kickstart file location after vmkboot.gz, but before the next boot module. It should not be at the end.

mboot.c32 vmkboot.gz ks=file://etc/vmware/weasel/ks.cfg --- vmkernel.gz ---sys.vgz ---cim.vgz --- ienviron.vgz --- install.vgz

Here is an example:

When you’re done, press enter. It will begin to load the data off the CD, and when the different install modules are done it should simply begin to install ESXi just like how I had hacked it together previously…

The only thing left will be to press “Enter” when it’s done (why?!).

A word of caution…the kickstart that VMware has provided will automatically select and format the first disk that it finds, regardless of it being local or “remote” (i.e. a SAN LUN). I would assume that the vast majority of the time it’s going to find the local disks first, however……..

Hopefully in the next few days I’ll have some more time to play with the new kickstart features and post some more examples. VMware has really done some great things with this process and it is now possible to have the entire process be automated…
1) use DHCP to provide the “permanent” IP
2) use a network PXE boot for the media and to provide a KS file
3) use the --post section of the kickstart file to have the server reach out and touch a vCLI or PowerCLI configuration host and provide permanent configuration.

The reason that step one should provide the actual IP is that it provides an easy way of having your configuration host (vCLI or PowerCLI) know what IP (and potentially hostname) to assign to the host.

Good luck, and thanks to VMware for (finally) integrating kickstart with ESXi!

I won’t attempt to provide a feature rundown or tell you why vSphere 4.1 is the greatest thing since sliced bread.  It appears to be a solid release, but  I’ll leave that analysis to the experts…Instead I want to talk about the vSphere hypervisor (previously ESXi).

Why the name change? Simple what was previously mis-branded as a separate technology is really the hypervisors core.  Previously in ESX3.5, ESXi was a separate technology, but as of vSphere 4 they have had a unified core.   In-fact the product we like to think about as vSphere 4.0/4.1 is really just a vSphere hypervisor with a special management VM!  This is important, the only difference is the console which is nothing more than a VM!

So why the distinction, Why now?  VMware is playing it’s hand this round because that special VM is going bye, bye.  The Next release of vSphere will not have a service console.. PAINIC…. RUN IN CIRCLES THE ZOMBIES ARE COMMING!!!

Don’t Panic, Personally I applaud the move.  Over the past year and a half I’ve heard every argument against the console less hypervisor, but honestly I chalk it all up to people fear change.  There are a couple thousand admins who have invested a lot of time mastering vSphere, and VMware is about to change the whole game on them.  These guys/gals bring up several arguments against the console less hypervisor, I’ll attempt to offer my counter argument to these points.

Q. No 3rd Party agents.

A. It has been public knowledge that the console was going away, and as of vSphere 4.0 VMware shipped a new management appliance vMA.  One of the intended uses of this appliance was to install 3rd party agents.  So you see we do still have 3rd Party agents they just need to be rewritten.  In most cases this will result in a better product. Unfortunately, the vast majority of 3rd party software, could better be described as a really complex perl script running over ssh!

Q. Hardware monitors/plug-in

A. Part of the original ESXi 3.5 release was the introduction of a rudimentary CIM provider.  This provider has been fully expanded , and made extensible.  While it is a change from the traditional agent based monitors CIM does fill in this gap.

Q. Automating common tasks.

A. As of vSphere 4.1 Tech Support Mode supports SSH, but you should really be using either PowerCLI or the vCLI!  While it is true that are still a couple of things that can only be done via the console.  I’m confident VMware will fix those gaps before putting the console out to pasture.

Q. Security

A. So this is the big one, and my personal pet peeve.  I’ve heard security experts bash the vSphere hypervisor claiming it was insecure.  I just don’t understand this stance, admittedly I’m no security expert.  I only work with the federal government in some of the most secure data centers in the world, but what do I know…

Let’s break this down shall we… The only difference is a VM.   Admittedly this VM has special connections into the vmkernal, but it’s still just a VM.  How exactly does the inclusion of a VM make the hypervisor more secure?  In my opinion the exclusion of this VM instantly increased the security posture of most organizations.  The reason for this simple, it was hard to properly harden the console.  Alternatively it was all too easy to open a critical security hole, and expose ones infrastructure with the console.

Yes you still have to do several things to really lock down the console less hypervisor, but it’s not nearly the feat the console once was.  In fact it’s simple;

1. Modify the Proxy.xml (turning off all unneeded web services, and make everything use https).
2. Enable Lockdown mode.
3. Physical security.

That’s it folks, that’s all it takes to secure the hypervisor.  There are a couple hundred other little things necessary to design a secure infrastructure, but as you can see the hypervisor is easy!  In fact I’m so confident in this I’m willing to hold a Bobby Flay style throw down.  If you have the means to provide a  pair of internet facing vSphere hosts. I’ll secure the console less hypervisor, we’ll get TexiWill to harden the legacy console based hypervisor, and then we’ll release the IP’s to the world.  Have at it, folks I bet the console less hypervisor holds up at least as long as the legacy hypervisor!

Why so brash? Well it will take an exploit to get in to the console less hypervisor, and any exploit will also be present in the legacy hypervisor.  The console less vSphere hypervisor without access to the physical host or vCenter there is simply no other way in.   Remember this isn’t Linux or BSD or UNIX… it’s vSphere it’s practicality firmware, and the whole point was to remove all that crap that weaken the security , and stability to begin with!

I really want to put this to bed!  Let’s develop the to do list for VMware.  The 10-20 things they need to fix before they can finally kill the console.  Then let’s collectively shut up about it.  It’s going to happen, and complaining with arbitrary little gripes… or demanding NDA meetings with engineers isn’t going to stop any of it.  The Task at hand is simple, weed out the crap, and focus on what needs to be fixed in vSphere v.Next.

If we missed something let us know in the comments.
~Glenn

If you have ever changed the vCenter server certificates, you’ve experienced having all your hosts disconnected from vCenter.  I couldn’t imagine reconnecting them one at a time… You could do this all natively in PowerCLI, but that would require you to fully remove and then add the hosts.  That is very inconvenient, and almost as much trouble as doing it by hand… In this case it is both faster and easier to just use the native vSphere API.

# Get the hostsystem object for every host currently disconnected.
$VMhosts = Get-View -ViewType 'Hostsystem' `
                    -Property 'name' `
                    -Filter @{"Runtime.ConnectionState"="disconnected"}            

Foreach ($VMhost in $VMhosts)
{
    # Create a reconnect spec
    $HostConnectSpec = New-Object VMware.Vim.HostConnectSpec
    $HostConnectSpec.hostName = $VMhost.name
    $HostConnectSpec.userName = 'root'
    $HostConnectSpec.password = 'PassWord'            

    # Reconnect the host
    $taskMoRef = $VMhost.ReconnectHost_Task($HostConnectSpec)            

    # optional, but i like to return a task object, that way I can 
    # easily integrate this into a pipeline later if need be.
    Get-VIObjectByVIView -MORef $taskMoRef
}

~Glenn

While migrating a small environment to vSphere today I ran into my nemesis Host Profiles again. When are they going to Fix these things? The fact that they are incapable of even rudimentary iSCSI configuration is embarrassing. I’m sure vmware will fix it, but until then I wrote a simple one-liner that will configure iSCSI on a new host.

$VMhost = Get-VMhost 'ESX01'
$ChapUserName = 'vmware'
$ChapPassword = 'password'
$SendTargets = '192.168.1.1'            

# Enable the software ISCSI adapter if not already enabled.
$VMHostStorage = Get-VMHostStorage -VMHost $VMhost | Set-VMHostStorage -SoftwareIScsiEnabled $True            

#sleep while iSCSI starts up
Start-Sleep -Seconds 30            

# By default vSphere will set the Target Node name to iqn.1998-01.com.vmware:<HostName>-<random number> 
# This script will remove everything after the hostname, set Chap auth, and add a send Target.
#
# Example iqn.1998-01.com.vmware:esx01-165435 would become iqn.1998-01.com.vmware:esx01
$VMHostHba = Get-VMHostHba -VMHost $VMHost -Type IScsi |
    Where-object { $_.IScsiName -match "(?<IQN>iqn.1998-01.com.vmware\:[^-]+)"} |
    Set-VMHostHba -IScsiName $Matches.IQN |
    Set-VMHostHba -ChapName $ChapUserName -ChapPassword $ChapPassword -ChapType "Required" |
    New-IScsiHbaTarget -Address $SendTargets -Port "3260"                

#restart the host to make sure everything took
Restart-VMHost -VMHost $VMHost -Confirm:$false | out-null

~Glenn

I suppose easy is relative.

One of the comments to the post I made about touch free ESXi installs asked about testing without having to reboot and wait for the install process to load, and fail, to determine what went wrong. I did this testing by switching to a different console and using the Python interactive shell to load the same modules VMware uses, call their methods, and simply look at the values returned. By reading their code I was able to determine what the dialogs (the prompts presented to the admin during the install process) return, and the simply return that without the dialog occurring.

From the install screen you can switch to a different console (Alt+F1, if I recall correctly), and then access a command line (you may need to use the “unsupported” trick to get the command line).

The install process is actually quite interesting. VMware is booting, via ISOLINUX, into a full ESXi environment (instead of the standard ESXi yellow and black, they display the install dialogs), asking you which disk you want to use, then formatting that disk with VMFS and copying a virutal machine to the new VMFS volume. They then configure the boot partition to start the ESX kernel and start the ESXi management virtual machine. It’s rather interesting (well, to me) what they are able to accomplish because of the power and flexibility having such a small hypervisor affords them. Not to mention the sheer genius of using their own hypervisor to perform the install of itself…simplicity!

Continue Reading »

I’ve ran across this particular issue myself, and submitted a bug to the PowerCLI team, but shortly after Andrew posted his ESXi 4.0 autoinstall Tim asked about this very issue.  There is a documentation error in Example #5 from the Apply-VMHostProfile cmdlet help.  Which contains the following code example.

$profile = Get-VMHostProfile -Name testProfile            

$additionalConfiguration = Apply-VMHostProfile -ApplyOnly -Profile $profile -Entity 10.23.114.166
$additionalConfiguration['network.hostPortGroup["key-vim-profile-host-HostPortgroupProfile-VMkernel"].ipConfig.IpAddressPolicy.address'] = '10.0.0.128'
$additionalConfiguration['network.hostPortGroup["key-vim-profile-host-HostPortgroupProfile-VMkernel"].ipConfig.IpAddressPolicy.subnetmask'] = '255.255.255.0'            

Apply-VMHostProfile -ApplyOnly -Profile $profile -Entity 10.23.114.166 -Variable $additionalConfiguration

Sadly if you tried to execute the above you would get the following error.

PS > $additionalConfiguration['network.hostPortGroup["key-vim-profile-host-HostPortgroupProfile-VMkernel"].ipConfig.Ip
AddressPolicy.address'] =  "10.52.8.11"
Array assignment to [network.hostPortGrou ..] failed: Cannot convert   value "network.hostPortGroup["key-vim-profile-
host-HostPortgroupProfile-VMkernel"].ipConfig.IpAddressPolicy.address" to   type "System.Int32". Error: "Input string
was not in a correct format.".
At line:1 char:26
+ $additionalConfiguration[ <<<<  '"network.hostPortGroup["key-vim-profile-host-HostPortgroupProfile-VMkernel"].ipCo
nfig.IpAddressPolicy.address'] ='10.52.8.11'
   + CategoryInfo          : InvalidOperation: (10.52.8.11:String) [],   RuntimeException
   + FullyQualifiedErrorId : ArrayAssignmentFailed 

 

At first this may appear a little cryptic, but it get’s a lot clearer once we inspect the object types in use.

PS > $additionalConfiguration.GetType()

IsPublic IsSerial Name                                     BaseType
-------- -------- ----                                     --------
True     True     Object[]                                 System.Object  

PS > $additionalConfiguration[0]

Name                           Value
----                           -----
network.hostPortGroup["key-...

PS > $additionalConfiguration[0].GetType()

IsPublic IsSerial Name                                     BaseType
-------- -------- ----                                     --------
True     True     DictionaryEntry                          System.ValueType

The example from the help docs was apparently expecting a Hashtable to be returned from apply-VMhostProfile.  Instead we found an array of DictionaryEntry objects... hence the error.

There are two possible work around's we can employ until the PowerCLI team ships a fix.  The first one is complicated, but dynamic.

$VMHostProfile = Get-VMHostProfile -Name testProfile            

$additionalConfiguration = Apply-VMHostProfile -ApplyOnly -Profile $VMHostProfile -Entity 10.23.114.166
($additionalConfiguration | Where-Object {$_.Name -eq 'network.hostPortGroup["key-vim-profile-host-HostPortgroupProfile-VMkernel"].ipConfig.IpAddressPolicy.address'}).Value = '10.0.0.128'
($additionalConfiguration | Where-Object {$_.Name -eq 'network.hostPortGroup["key-vim-profile-host-HostPortgroupProfile-VMkernel"].ipConfig.IpAddressPolicy.subnetmask'}).Value = '255.255.255.0'            

Apply-VMHostProfile -ApplyOnly -Profile $VMHostProfile -Entity 10.23.114.166 -Variable $additionalConfiguration

I actually don't like this approach even though it's a modified version of the included example.  I prefer just a simple static Hashtable.

$VMHostProfile = Get-VMHostProfile -Name testProfile            

$additionalConfiguration = @{
    'network.hostPortGroup["key-vim-profile-host-HostPortgroupProfile-VMkernel"].ipConfig.IpAddressPolicy.address'    = '10.0.0.128'
    'network.hostPortGroup["key-vim-profile-host-HostPortgroupProfile-VMkernel"].ipConfig.IpAddressPolicy.subnetmask' = '255.255.255.0'
}
Apply-VMHostProfile -ApplyOnly -Profile $VMHostProfile -Entity 10.23.114.166 -Variable $additionalConfiguration

All in all, the HostProfile cmdlets are surprisingly complete, and I think the majority of the "issues" I've ran across are a result of the SDK itself.  The Host Profiles sections of the API just don't have the same fit and finish I've come to expect in a VMware API.

I'm sure carter and team will have this fixed in the next release, untill then...

~Glenn

Being, first and foremost, lazy and getting my paychecks for being a system administrator, I felt that the amount of work involved in loading ESXi 4.0 on my blades was entirely too much. I have well over 100 blades, each one needing to have vSphere loaded onto it, configured, and added to vCenter. Even using the directions scattered across the internet about reducing the amount of effort involved in loading vSphere was too much for me.

Others have documented how to PXE boot ESXi elsewhere on the internet, however I wasn’t interested in having a “stateless” install…I merely wanted to automate installing ESXi to the local hard drive. My blades have a single hard drive, a single generation one SSD or two SAS drives in a RAID 1 depending on the vendor, and I simply want the installer to always install to that drive without bothering me. Loading from the “remote media” functionality of the DRAC/iLO for the blades takes forever, so I wanted to be able to install using PXE and push the media over that medium.

So, having been a developer for several years I decided to dive further into the the install process than others had detailed. Turns out that eliminating all input from an administrator to load the operating system was pretty simple.

The end result is that I am able to power on a blade, hit F12 to have it PXE boot and walk away. Some time later, we can use PowerShell and the PowerCLI to find the hosts (they will be somewhere in the DHCP scope of the provisioning LAN), give them a permanent IP and hostname, then configure them and add them to vCenter. By using PXE and the interactionless (yes, I did make up that word) install, I cut the time to load ESXi from about 45 minutes (using the remote media function takes FOREVER!) to less than 10.

Continue Reading »

I wrote this script about a year ago to deal with errant SMVI snapshots, and was drafting this blog post when my rss feed caught me off guard. It appears Matt Robinson has beat me to the punch line.   He has produced a Perl script that cleans up any leftover snapshots, but if you favor a PowerShell approach… I give you Remove-SMVISnapshots.
Continue Reading »

If you don’t know already PowerCLI now has two modes single and multiple.  It stands for exactly what you think it does.  In single mode when you execute a command PowerCLI runs that command against the server you’re connected too.  Multiple mode allows you to specify multiple vCenter/ESX/vSphere host, and when you execute a command it runs that command against every server you’ve specified! This had to be one painful feature to get right, but the PowerCLI team nailed it.

I’ll admit when I first played with it I thought I would never need/use multiple mode.  That is until our vCenter server was inadvertently shutdown instead of rebooted.  Normally this would lead to one of two out comes.
A.) forcefully register vCenter on the first host I hit and power it up.
B.) A twenty minute search for the host that has vCenter.

Well today I didn’t feel like doing either… On a whim I tried this new-fangled multiple connection thing… IT WORKED!

# Set PowerCLI to multiple
Set-PowerCLIConfiguration -DefaultVIServerMode Multiple -Confirm:$false
# Connect to every vSphere host in the cluster that contains vCenter
Connect-VIServer -User root -Password password -Server esx1,esx2,esx3,esx4,esx5,esx6,esx7,esx8
# Start vCenter
Get-VM vCenter01 | Start-VM

I’ve since wrapped all this up in a batch file and added it to our playbook for a lights out recovery of virtual center!

~Glenn

I’ve spent the past month updating/validating our NetBackup architecture. I’ll admit I’m more of a Backup Exec fan, but NetBackup does have some cool features. My only real complaint with NetBackup is the documentation. I’ve never seen a 700 page PDF say so little. One such topic is the installation and configuration of the NetBackup Live update server. Nowhere in the 20 admin guides I reviewed did it spell out “How”. After a brief conversation with Symantec support the veil of poor documentation was lifted.  

You “can” run live update off a CIFS share, but I wouldn’t. It will only sort of work with some windows clients. I’ve spelled out the five steps to configuring an http based live update server on a windows machine. If you hit the Symantec forums you’ll find many confused posts but nothing approaching an answer. Hence this post, I could have gotten fancy and scripted it all out (and I wanted to), but I simply don’t have that kind of time these days. If the NBLU server has been kicking your arse as it did mine… enjoy.

1. First things first install IIS.
   1. Win2k3/IIS6:
      1. Add/Remove Programs -> Add/Remove Windows Components -> Application Server -> Internet Information Services (IIS) -> World Wide Web Service
      2. check “World Wide Web Service”
      3. OK -> OK -> OK -> Next -> finish
   2. Win2K8/IIS7:
      1. Server Manager -> Roles -> Add Roles
      2. Check Web Server (IIS)
      3. Next
      4. Acknowledge the addition of ISS Management features
      5. Next -> Next -> Install
2. Create/Populate the live update directory
   1. In the web root create a folder called “liveupdate”
      mkdir c:\Inetpub\wwwroot\liveupdate
   2. Extract the contents of all five zip files into the live update folder. Pay close attention to now overwrite the Segments directory as this needs to be merged.
      NBLU_6.5.4.WINx64_XXXXXXX.zip
      NBLU_6.5.4.WINx86_XXXXXXX.zip
      NBLU_6.5.4.UNIX_1of3_XXXXXXX.zip
      NBLU_6.5.4.UNIX_1of3_XXXXXXX.zip
      NBLU_6.5.4.UNIX_1of3_XXXXXXX.zip
3. Configure directory browsing
   1. Win2k3/IIS6:
      1. Open IIS Manager
      2. Expand Web Sites -> right click on Default Web Site -> liveupdate-> select properties
      3. Check Directory browsing on the Virtual Directory tab.
      4. Click Ok
   2. Win2K8/IIS7:
      1. Open IIS Manager
      2. Expand Sites -> Default Web Site -> liveupdate
      3. Double click on directory browsing.
      4. Click enable on the right side action panel.
4. Update MIME information
   1. Win2k3/IIS6:
      1. Open IIS Manager
      2. Right click on <computer name> (Local Computer) -> select properties
      3. Click MIME Types
      4. Click New
      5. Extension: “.flg”
         MIME Type: “Application/octet-stream”
      6. Click Ok-> OK
      7. Right click on <computer name> (Local Computer) -> All Tasks -> Save Configuration to DISK
      8. Right click on <computer name> (Local Computer) -> All Tasks -> Restart IIS
   2. Win2K8/IIS7:
      1. Open IIS Manager
      2. Expand Sites -> Default Web Site
      3. Double click on MIME Types
      4. Click Add on the right side action panel.
      5. File name extension: “.flg”
         MIME Type: “Application/octet-stream”
      6. Ok
      7. Right click Default Web site -> Manage Web Site -> Restart
5. Test your new NetBackup Live update server
   1. Browse to http:/<hostname>/liveupdate, you should see the contents of the liveupdate folder
   2. Browse to http:/<hostname>/liveupdate/minitri.flg you should be prompted with an open/save file dialog

If 5.1 and 5.2 work as expected then you’re done!