get-admin

As I’ve been managing more and more infrastructures using fibre channel storage, I’ve found that it’s been somewhat difficult to keep the LUN paths to each host balanced. By balanced, I mean that for each LUN to each host, there is a number of paths and I want to make sure that, for example, each LUN 10 to each of the hosts is using path A as the primary and path B as the stand by. LUN 11 uses path B as the primary, LUN 12 back to path A, and so on.

It so happens that I’m using a DMX-4 for storage, and the policy we have is to use a fixed path policy. I realize that Round Robin would make this entire script moot, well, except for making sure that the PSP is correct. I also realize that PowerPath would be the ideal solution for EMC storage, but we don’t use it…that’s a story for another day.

This script is, admittedly, long…longer than I expected it to be. The original inspiration for this script came from Justin Emerson’s very functional and succinct script, however I was not satisfied with the way LUNs were balanced. His script queries the host for LUNs then sorts them by canonical name and round robins the paths based on the number of paths present for the first LUN.

This works well, so long as all the LUNs are present on all the hosts and they all have the same number of paths. I can only presume that he assumes that those cases have already been checked for, and fixed, prior to execution. I wanted to do that all in one script.

Additionally, and it’s rather petty, I wanted the LUNs to be balanced based off their LUN identifier rather than the canonical name…they don’t always follow the same order, and in the case of my hosts with two HBAs (and consequentially, two paths per LUN), I wanted all odd LUNs to use one path for the primary and all even LUNs to use the other. Justin’s script does an excellent job of ensuring that the paths are evenly distributed, as you will end up with the same number on each, but not in the pretty fashion I desired.

Also, thank you to Glenn, who helped me “powershellize” this script…my PowerShell looks and reads like Perl, and therefore doesn’t use a lot of the optimizations that PoSH brings…such as automatic parameter handling and other niceties.

So, without further ado…

Continue Reading »

The VMware vSphere PowerCLI Reference will officially go on sale April 12th, and if your a vSphere administrator this one is a must have. We gathered the collective automation experience of four vEXPERTS and a MVP, and then filtered it through a fifth vEXPERT. The end result was a collection of polished solutions that are not only technically proficient, but more importantly will work the first time every time! Having all built and maintained large infrastructures we combined our collective knowledge and wrote a complete reference, we cover the entire life cycle from creating your environment to monitoring it long term. I mean it, we left no stone unturned where VMware had no PowerCLI solution we wrote our own. This book includes a PowerShell Module with 79 custom advanced functions! We also considered how you would use our book, and chose to take a task driven approach, this enabled you to just flip to the answer you need. You don’t have to “read” all 780 pages… instead think of this book as a fire extinguisher for your virtual infrastructure.

So that’s great, but who needs a book? With blogs and the community forums you can just find the answer right? Yes, yes you can eventually, but if you do that your shopping at a fishery. In this book we WILL teach you to fish! As a commitment to that end we created a dedicated web site to support our work. If you find a bug, or if we missed something you can post a question in our forum!

Admittedly this is my first published work, but I couldn’t be prouder of it. In my opinion The ONYX, and vSphere SDK chapters alone would be worth the cover price. I look forward to the release, and to your honest opinions of our work!

~Glenn

VMware is a facilitator. I know, you’re thinking “yeah, they facilitate my power/space/cooling savings, they facilitate infrastructure consolidation, IT agility, high availability, etc.”, but really, they facilitate me being lazy (which for sysadmins is a good thing…a lazy admin will only want to do a task once, then automate the sh*t out of it).

I’ve already documented how I hacked the ESXi 4.0 installer to have it do the installation without interaction. However, VMware has one-upped me and integrated kickstart into their installer. This makes things VASTLY easier, requires no tomfoolery with the ISO, and is significantly more capable.

This blog post will be just a short one to demonstrate how easy it is now to have the install be “touch-free”. I am working on some more complex examples in the coming days.

So without further blithering from me, on with the install! Put the CD in the drive (or mount the iso remotely), boot the server. When it reaches the boot options menu:

Press tab to append options to the boot line. Append the following after the vmkboot.gz, but before the --- after it.

ks=file://etc/vmware/weasel/ks.cfg

It is VERY IMPORTANT that you place the kickstart file location after vmkboot.gz, but before the next boot module. It should not be at the end.

mboot.c32 vmkboot.gz ks=file://etc/vmware/weasel/ks.cfg --- vmkernel.gz ---sys.vgz ---cim.vgz --- ienviron.vgz --- install.vgz

Here is an example:

When you’re done, press enter. It will begin to load the data off the CD, and when the different install modules are done it should simply begin to install ESXi just like how I had hacked it together previously…

The only thing left will be to press “Enter” when it’s done (why?!).

A word of caution…the kickstart that VMware has provided will automatically select and format the first disk that it finds, regardless of it being local or “remote” (i.e. a SAN LUN). I would assume that the vast majority of the time it’s going to find the local disks first, however……..

Hopefully in the next few days I’ll have some more time to play with the new kickstart features and post some more examples. VMware has really done some great things with this process and it is now possible to have the entire process be automated…
1) use DHCP to provide the “permanent” IP
2) use a network PXE boot for the media and to provide a KS file
3) use the --post section of the kickstart file to have the server reach out and touch a vCLI or PowerCLI configuration host and provide permanent configuration.

The reason that step one should provide the actual IP is that it provides an easy way of having your configuration host (vCLI or PowerCLI) know what IP (and potentially hostname) to assign to the host.

Good luck, and thanks to VMware for (finally) integrating kickstart with ESXi!

I won’t attempt to provide a feature rundown or tell you why vSphere 4.1 is the greatest thing since sliced bread.  It appears to be a solid release, but  I’ll leave that analysis to the experts…Instead I want to talk about the vSphere hypervisor (previously ESXi).

Why the name change? Simple what was previously mis-branded as a separate technology is really the hypervisors core.  Previously in ESX3.5, ESXi was a separate technology, but as of vSphere 4 they have had a unified core.   In-fact the product we like to think about as vSphere 4.0/4.1 is really just a vSphere hypervisor with a special management VM!  This is important, the only difference is the console which is nothing more than a VM!

So why the distinction, Why now?  VMware is playing it’s hand this round because that special VM is going bye, bye.  The Next release of vSphere will not have a service console.. PAINIC…. RUN IN CIRCLES THE ZOMBIES ARE COMMING!!!

Don’t Panic, Personally I applaud the move.  Over the past year and a half I’ve heard every argument against the console less hypervisor, but honestly I chalk it all up to people fear change.  There are a couple thousand admins who have invested a lot of time mastering vSphere, and VMware is about to change the whole game on them.  These guys/gals bring up several arguments against the console less hypervisor, I’ll attempt to offer my counter argument to these points.

Q. No 3rd Party agents.

A. It has been public knowledge that the console was going away, and as of vSphere 4.0 VMware shipped a new management appliance vMA.  One of the intended uses of this appliance was to install 3rd party agents.  So you see we do still have 3rd Party agents they just need to be rewritten.  In most cases this will result in a better product. Unfortunately, the vast majority of 3rd party software, could better be described as a really complex perl script running over ssh!

Q. Hardware monitors/plug-in

A. Part of the original ESXi 3.5 release was the introduction of a rudimentary CIM provider.  This provider has been fully expanded , and made extensible.  While it is a change from the traditional agent based monitors CIM does fill in this gap.

Q. Automating common tasks.

A. As of vSphere 4.1 Tech Support Mode supports SSH, but you should really be using either PowerCLI or the vCLI!  While it is true that are still a couple of things that can only be done via the console.  I’m confident VMware will fix those gaps before putting the console out to pasture.

Q. Security

A. So this is the big one, and my personal pet peeve.  I’ve heard security experts bash the vSphere hypervisor claiming it was insecure.  I just don’t understand this stance, admittedly I’m no security expert.  I only work with the federal government in some of the most secure data centers in the world, but what do I know…

Let’s break this down shall we… The only difference is a VM.   Admittedly this VM has special connections into the vmkernal, but it’s still just a VM.  How exactly does the inclusion of a VM make the hypervisor more secure?  In my opinion the exclusion of this VM instantly increased the security posture of most organizations.  The reason for this simple, it was hard to properly harden the console.  Alternatively it was all too easy to open a critical security hole, and expose ones infrastructure with the console.

Yes you still have to do several things to really lock down the console less hypervisor, but it’s not nearly the feat the console once was.  In fact it’s simple;

1. Modify the Proxy.xml (turning off all unneeded web services, and make everything use https).
2. Enable Lockdown mode.
3. Physical security.

That’s it folks, that’s all it takes to secure the hypervisor.  There are a couple hundred other little things necessary to design a secure infrastructure, but as you can see the hypervisor is easy!  In fact I’m so confident in this I’m willing to hold a Bobby Flay style throw down.  If you have the means to provide a  pair of internet facing vSphere hosts. I’ll secure the console less hypervisor, we’ll get TexiWill to harden the legacy console based hypervisor, and then we’ll release the IP’s to the world.  Have at it, folks I bet the console less hypervisor holds up at least as long as the legacy hypervisor!

Why so brash? Well it will take an exploit to get in to the console less hypervisor, and any exploit will also be present in the legacy hypervisor.  The console less vSphere hypervisor without access to the physical host or vCenter there is simply no other way in.   Remember this isn’t Linux or BSD or UNIX… it’s vSphere it’s practicality firmware, and the whole point was to remove all that crap that weaken the security , and stability to begin with!

I really want to put this to bed!  Let’s develop the to do list for VMware.  The 10-20 things they need to fix before they can finally kill the console.  Then let’s collectively shut up about it.  It’s going to happen, and complaining with arbitrary little gripes… or demanding NDA meetings with engineers isn’t going to stop any of it.  The Task at hand is simple, weed out the crap, and focus on what needs to be fixed in vSphere v.Next.

If we missed something let us know in the comments.
~Glenn