Tag Archives: vSphere

How to send vCenter alarms to Slack

I’m spending some of my time in the new gig with my old sysadmin ops hat on.  We needed a quick easy way to keep an eye on alerts from a vSphere environment so….what else would be more fun than to funnel them to Slack?!  Easy peasy, even on the vCenter Appliance.  Let’s see how…

First you need to configure the integration on Slack.   In the channel you wish to see the alerts in, click the “Add a service integration” link.


Now there is not any special integration with vSphere, we are going to be using a simple REST api to push the content.  Scroll down to “Incoming WebHooks”


Now you need to approve the integration verifying the chat room and click the button:


The outcome of this will be a long URL you will need for the script.

Now we need to get your script ready. Now remember this is on vCenter (windows OR appliance), not ESXi.  Much credit to this guy that created a simple script for Zabbix, as this is a hacked up version of it.   The key here is using the environment $VMWARE_ALARM_EVENTDESCRIPTION which I use because it’s short and simple.   If you want other types of data check out the documentation here.

Now you just simply need to hook this script up to the alarm in vSphere:


Sweet.  Cool.  Let there be (kind of) chatops.

But, I hear you asking…   What if you want to apply this to all your alarms??   Also…. easy peasy.   I just whipped together some powercli and bam.

That line will apply this script action to ALL alarms in the vCenter you connect to.   It will apply this by default to the Yellow to Red action level.    For now I wanted this to trip on all four cases so I looked a little deeper and found this will do it:

Now if you are like me and you screw this up along the way, you may have to clear out the actions across the board.  This line will do that for you:



Tagged , , , ,

Using the released version of Docker-Machine (v0.1) with VMware vSphere

I began uplifting some of my content today which included upgrading to the newest docker (v1.5) and docker-machine (v0.1), and came across a number of changes.

  • The command is now officially “docker-machine” instead of just “machine” which is what it was when I first played with it.
  • All the VMware driver commands are now prefixed with “vmware”   so instead of “–vsphere-vcenter” it is now “–vmwarevsphere-vcenter”   a full example is:

    And they have an easier way to set the environment variables now:
  • I couldn’t get “–vmwarevsphere-boot2docker-url” to work with a custom URL which is probably a bug.  If you leave it out entirely it will use a default location.
  • ..Which is a good thing because boot2docker now includes VMware tools, which negates the need for a custom .ISO
  • The only other change I need to make to the boot2docker image is the use of a insecure registry, so I just include in my syntax the running of a shell script which runs: docker-machine ssh $1 sudo sed -i -e ‘s/–tlsverify/–tlsverify –insecure-registry docker-hub:5000/g’ /var/lib/boot2docker/profile  You can find this full shell script on github here.   “docker-hub” is my registry hostname on port 5000
  • I noticed the name of the VM now matches what docker-machine calls it instead of a random string.

That’s about it so far.  I have not used it too extensively yet but so far so good.  I did not see a single hang of the docker commands like I saw previously with the older versions.  Thumbs up so far.

Tagged , , , ,

Where is the OVA/OVF for vCenter 6.0 Appliance?

moooar_vsphere6You might have noticed already that the new vCenter Appliance (VCSA) 6.0 is only being distributed as an .ISO.  While I do wish an OVA/OVF would have been available too, the new installer is pretty sweet.


When you mount this .ISO, you get a web page interface that cleanly and clearly walks you through deploying directly to an ESXi host.  In my opinion this is a pretty good user experience, which is something that has been lacking at the install time of these products for quite a while.  (Side note – check out the VIO installer too.  It’s slick as well.)

I do think there are a few isolated use cases for directly using an OVA or OVF however.  A vCD (or other hosted environment) is a good example, when you do not have access to the underlying hosts.  Or if you encounter some issues in your browser, perhaps due to draconian IT security measures.  All is not lost!   The OVA is actually hiding in the ISO, you just have to find it.


Disclaimer: This is not recommended or supported by VMware.  There are a lot of user input checks that go into the new installer, so use it when you deploy your VC for real.  This should only be used for testing/sandboxing/may blow up in your face. There I said it, so don’t yell at me when you break something!


The ISO itself will look like this on disk:


When you mount it, the directory structure looks like:


In the “vcsa” directory, you will find a file with no extention like this:


This is the actual .OVA   Copy it somewhere local so it’s writable and add the .OVA extension.


Now extract it.  Did not know you can just use an unzip utility to extract an OVA?  Well now you do.


Aaaaaand here are your familiar files!


Now I will warn you, you could will have a bad day if you directly try to use this .ovf with vCD.  It seemed to work fine for me in vCenter, but vCD does not understand a lot of the new options and fields in the format of this file.  The outcome is it’s really difficult to deploy anything other than an embedded node (PSC+VC).  If you want to deploy JUST the PSC or VC services you have to do quite a bit of editing.  Luckily I asked around before I spent time on this and I found the VMware HOL Wizard Doug Baer (@dobaer) had already pulled his hair out getting it to work.   So behold!  Customized .OVF files for either PSC or VC nodes!

I warn you, your results may/will vary with these so the usual disclaimers about NOT using this in production and NOT calling support and expecting help apply.

You can find the contents of these files on a Github repo of mine here, and again all credit to Doug for these:



There you have it.  Go get some vSphere 6 vCenter appliance action on.

Tagged , , , , ,

ESXi 6 Whitebox Realtek NIC issues

moooar_vsphere6I upgraded one of my whiteboxes to ESXi 6 and noticed the Realtek NIC was no longer detected.  I tried to install the same VIB I used previously and got dependency errors.   A quick google search later and I found this post that references a communities post that has a fix.  Yay!

Note – this probably shouldn’t have been posted publically during the beta program but meh, past is the past.

Here’s the VIB: http://vibsdepot.v-front.de/depot/vft/net51-drivers-1.0/net51-drivers-1.0.0-1vft.510.0.0.799733.x86_64.vib  I can confirm this working on a Gigabyte Z68X-UD3H-B3 that has a Realtek 8168 on board.   Many thanks to the author!

Tagged , , , , ,

New release: VMware Software Manager – Download Service

Release Notes


Snip20150311_9Today VMware is releasing a trove of software.  One small tool that will surely make release days like this much easier is Software Manager.   I just tried it out and it’s exactly what you will expect.   No more browsing through the somewhat painful download web pages!  Have it all come to you.

The tool is windows only, and comes in at a whopping 17MB for the tool itself.  It comes packaged as a MSI, quickly install it, specify a location (that has enough space for the many gigs of of downloads) and it fires up a web page connecting to localhost on port 8000.  Login with your MyVMware credentials.   BAM!  You will see all the downloads you are entitled to and have a very easy one click download for a whole suite of components.



UPDATE:  vSphere 6.0 is now showing up in the product.   If you installed it first thing today, you might need to stop and restart the service for it to show up.   Worst case, kill your browser with a hammer.


Tagged , ,

vCenter alarms for VSAN not included out of the box

Snip20150309_7I have been testing VSAN in my whitebox hosts of my home lab lately, as I consider migrating off the smattering of Synology and other datastores.  My first mistake was using a spinning drive that apparently was already bad as I quickly got some permanent device loss errors after spinning up some VMs.  I was concerned why I never got any actual VSAN alarms, and just randomly stumbled on the reason why via Reddit.

Turns out, vCenter does not contain any VSAN alarms that map to VOBs.   This KB shows how to add a few of them, but Mister VSAN Rawlinson Rivera’s blog post shows a longer list that appears pretty comprehensive and also includes a bit of powershell code to add the alarms for a specific cluster.  I could not get the code to copy/paste from the blog without characters getting all screwy so it took quite some time to clean up.   Here on my git repo is a version of it that works as is.

Tagged , ,

How to modify a boot2docker .ISO for Docker Machine

I have been doing quite a bit of work with VMware vSphere + Docker + Machine lately.  I helped build a Hands On Lab for the recent VMware Partner Exchange conference with it.  I can’t promise, but it looks like it might be available publicly.  If you have access – it’s named HOL-SDC-1430.  It has been a difficult process as we’re in such alpha territory.  Things sometimes work, sometimes they don’t, and then they change a rev later..

Snip20150212_9An example of this is three specific items needed in a boot2docker image that is used as the docker host VM.  We need VMware tools (or open vm tools), some networking updates, and in my case the change to docker settings to allow pulling from an insecure local registry.

VMware’s Cloud Native Apps R&D has forked the main boot2docker repo and done the tools work and networking work (probably among others too) but I had to dive in and figure out how to edit it further to allow for a new docker option.  I really can’t claim to be an authoritative source on the docker and boot2docker side of things here but the googles failed me on a single location for all this information so here you go!

1) First you have to clone from a specific branch of VMware’s Cloud Native Apps git repo.  ovt stands for open vm tools.  See the diff’s here.

2) I found a Dockerfile is what is used to customize the iso.  How it works is the container is built from the file, and a number of other dependencies in the sub directories but is written to print out the iso data when run.  Pretty clever whoever first came up with this method.    So to do my hard-coding-not-best-practice-but-solves-my-needs I edit the Dockerfile as follows by removing the dependency on the b2d version, and just pull the latest Docker.   1.5 came out this week and I was wanting to pick up those updates.


3) I also needed to use a local repository without certificates since I am building lab environments, so I added a new config variable for $DOCKER_REG to make it easier to update later.


4) Now the rest is just following the b2d documentation.  Build the container with:

5) And write out the ISO with:

One thing that hung me up for a bit was machine doesn’t do any checksum on the ISO you tell it to use.   If machine sees the image already exists it WILL NOT overwrite it on the target datastore, so remember to delete it and let it upload the new one.  Very important.

Tagged , , , , ,

Tech Preview of Docker Machine Driver for vSphere

UPDATE:  Machine is now out of beta, and I have a newer post on some of the changes here: http://www.jaas.co/2015/03/20/using-the-released-version-of-docker-machine-v0-1/

(Also, checkout the Fusion driver here.)


On Dec 4 2014 Docker announced the “machine” command/functionality as one of the announcements at DockerCon 2014.

In short it provides a way to deploy to many different locations from the same docker cli interface (yes I know that is like ATM machine, just deal with it.).  In their initial alpha release they are including drivers for Virtualbox, and Digital Ocean (though now as of Dec 18 looking at their GitHub page they list syntax already additional for AWS & Azure though I’m not sure if this is functional yet).

The next day on Dec 5 2014 VMware announced a preview of their extension to this Docker Machine command for deploying to VMware Workstation, vSphere and vCloud Air.

I have been using the vSphere part of it a bit this week and found the existing instructions a bit lacking so I wanted to provide some tips and examples to get up and running.

Things to know – but maybe come back to this list later….
First off, a few take aways I learned the hard way.  Which included bugging a very smart dude for help.    A few of these may not make sense until you dive into the functionality yet, so you may want to revisit this section later if you have trouble.

To be clear, I am not posting this list as all the blemishes I found, but as a guide to help anyone else that is struggling to see this vision that this functionality has the potential to bring.  Remember – this is a preview release of the VMware driver, and an alpha release of the Docker code.  Totally unsupported in every meaning of the phrase.


1) Understand that for this release you need to use the bundled docker binary as it has some functionality that the newest release you’ll get from package managers don’t have.   To get it to co-exist on a system that already had docker-io installed on it, make sure to either specify the full path to either one, or make sure the $PATH env variable is set so it picks up the one you want first.   I also copied my released docker binary to docker_local, so I could easily run that command if I wanted to switch to a local docker container instance.

2) This release of the machine command requires the use of environment variables to specify the active host.  When you run machine ls it will list all the existing docker machines available.  It also specifies the “active” one.  I’m not in the loop on the dev details of this but I assume this will be cleared up in the future.   Even though it says active, you still have to set the ennvironment variable.  Either pull the URL from the machine url machinename command or you can used this nested command

Do note that this is required.  One stumbling point I found was I couldn’t find a way to make this work when I don’t have a real tty session like when vCO makes a SSH call in a workflow.  TBD there……

3) As a follow up to both of the two above, if you DO want or need to switch between using docker machine and a local docker binary, you need to clear out the environment variable with

Yes kind of annoying for now, but I’m sure this will be fixed soon enough.  They are probably discussing this issue here on github.

4) I’m not sure how many others use local docker registries out there, but I do quite a bit for lab environments.  As with this other post I made about the change to forcing SSL communication, it took a moment to figure out how to force the configuration setting on each docker machine.  The really smart guy I alluded to previously built me a boot2docker iso with it embedded in it, so that’s an option, or you could could just manually apply it like this:


5) I had quite a bit of problems with a space in the datacenter name, and special characters in passwords.  There may be a workaround, but simple escaping it out didnt work so I just renamed the datacenter and used an account with all simple characters.  Remember….alpha code….

6) Yes, it downloads the ISO every time you run the machine command.  I don’t know why. Go ask docker.  Because, alpha.

7) I even hesitate putting this one here… but in my personal lab it kept failing when transferring the ISO to the datastore.  But it works fine in another lab environment I use.
Probably my own issue with some ghetto routing issue I have….  I worked around it by uploading the ISO by hand to the datastore.  Even though as I said in #6 it downloads it every time, if it already exists in the datastore it doesn’t try to push it again.Lets-Do-This

Step by Step

This syntax is accurate as of Dec 18 with CentOS 6.6 64bit.

Grab the tarball:

GRUMBLE GRUMBLE… whoever compressed this didn’t include the directory…. so it extracts to the present working directory……

Append the directory you extracted to your path environment variable.  Lately I’ve been using ~./bash_profile to set individual settings per user on each host, your results may vary.

And now to fire it off for the first time:

And with any luck you should see a VM pop up named docker-host-ABCDEFG  (that last part is random).  If you get errors, read over the ‘PRO-Tips’ at the end, and ‘Things to know’ at the top.

Now to list the current machines, run:

Set the required environment variables with:




Now! The magic is happening!  Run a normal docker command like:

And see the magic happen for reals.   This image is being deployed on this new docker “host” which is actually a barebones vSphere VM.


1) If you are doing a bunch of trial and error and you see the message that the docker host already exists, even though it may or may not have been deployed.  This is because even if the command fails it still gets added to the local machine list.  Clean it up with machine rm -f machinename   the -f forces the remove, if the actual VM doesn’t exist.

2) If you get an error message similar to “FATA[0086] open /root/.docker/public-key.json: no such file or directory”  just run the docker binary included here and it will create this file for you.

3) I crafted a pretty sweet bash script to nuke all machines at once.  Add the -f flag to force if you have to.  It works as such:



So what does this give us.   In my mind this gives us a simple interface that you may already be familar with and already using on your local machine, the ability to deploy to any number of other endpoints like public or private clouds.   That’s powerful.   Especially with any automation you have already created – slipping this into the mix, makes it even more robust.

This post was heavy on text and light on screenshots on purpose as it’s a complicated subject in this state of development.  I hope to put together a quick video to demonstrate this functionality soon.  Stay tuned.



Tagged , , , ,

Using vSphere Tags via vCO

As vSphere 5.5 currently stands, the only way to interact with vSphere tags is via PowerCLI.   This leaves vCO out of the party without some effort to build it manually.   I am working on a solution where I wanted to include tags in some automation to enable some awesomeness so I explored if it was possible to expose to this vCO without huge effort.  Success!


It wasn’t too difficult to build this.  The two most difficult parts were setting up the PowerShell host for vCO (google it… it’s difficult.. at least it was for me the first time around), and parsing of the XML returned from PowerShell to vCO to get the data I wanted.   These workflows are a bit rough but they work as a first draft.   For anything production caliber you’ll want to evaluate the performance impact of hitting the powershell host as often, and definitely change the password field from string (proof of concepts!).

What I have built so far is a workflow “JaaS Tags- Add Tag” that accepts strings for the name of the tag, and virtual machine name.  This fires off powershell commands in a vCO action:


To show how it works running manually in the vCO client:



And to show that the tag is actually applied, you can find it in the Web Client:


Now, I also have a workflow to find VMs from the tag that is supplied.  I needed flexibility out of these in the solution that i’m working on so the output from this one is two arrays – one of the name in a string, and one of VC:VirtualMachine type.




Running this guy manually, just supply the tag name:


And to show the output parameters, you can see that in the vCO client:


Yay Tags!   Now you can include these workflows in other solutions to utilize tags in ways they weren’t intended.  Stay tuned for the solution I built this for.

I’ve posted an export of this package to FlowGrab, check it out here:  https://flowgrab.com/project/view.xhtml?id=d2623373-838f-4ee2-8d6e-c6f582cb452f




Tagged , , , , ,

Small Things: vSphere 5.5 U2 – C# Client, Editing HWv10 VMs

Maybe I’m just becoming an old get off my lawn ex-operations curmudgeon in my “old” age, but I find the vSphere Client hard to part with (this message brought to you by Me, and only Me, and no one but Me).  I found it very annoying that if you upgraded any VM hardware versions to 10, you could no longer edit settings in the old client – even something common like mounting an ISO.

vSphere 5.5 U2 has brought us this, yay!  This allows editing of any features present in the old client which is good enough for basic stuff.




EDIT: Whoops.. fixed the screenshot

Tagged ,