Tag Archives: vCenter

Making sense of Spectre and Meltdown in a vSphere and Dell environment

If you work in IT you no doubt are aware of the problems Intel has been having.  If not….here’s a link for you.  Enjoy.

In the days after the public announcement vendors released the microcode updates that Intel delivered to them, as well as, a few updates in the months prior that they slipped in without any fanfare.

Specifically, VMware had a fix in December for Spectre on VMSA-2018-0002.2

6.5: ESXi-6.5.0-20171201001s-standard – Build 7273056
6.0: ESXi-6.0.0-20171101001s-standard – Build 6856897

On January 9 after the announcement VMware posted VMSA-2018-0004.2 also for Spectre.

vCenter 6.5 U1e – Build 7515524
vCenter 6.0 U3d – Build 7462485

AND

6.5: ESXi-6.5.0-20180104001-standard – Build 7526125
6.0: ESXi-6.0.0-20180104001-standard – Build 7504637

HOWEVER, Either on January 10th or January 12th (can’t tell from the history), they updated the KB article and pulled the ESXi patches from the depot when Intel advised they were seeing unplanned reboots from hosts with the applied microcode.

At the current time – my best recommendation is to patch ESXi up to the following: (this site is REALLY handy)

6.5: 2017-12-19 ESXi-6.5.0-20171204001-standard – Build 7388607
6.0: 2017-11-09 ESXi-6.0.0-20171104001-standard – Build 6921384

If you applied those January patches for ESXi, and have a build number of 6.5 7526125 or 6.0 7504637, VMware has a process for applying a CPU mask to help avoid the problem, however it has to be done per VM and requires a reboot of each.  My personal feeling is if you aren’t seeing a problem, wait for further updates.  Your results may vary of course.

 

On the Dell side, they did too make a bios update available and has since pulled it.  I only pay attention to R430’s – they released BIOS 2.7.0 (link is now dead).

As of today (Jan 22) I noticed that bios has been removed and the newest listed is 2.6.0 released November 28.  Dell’s main support page has not been updated with a recommendation as of yet.  We’ll see what develops here.   My recommendation is hold off on deploying the new bios, and be cautious of any hosts that already have it.  Dell Support says options for rolling back are coming.

 

UPDATES –

Jan 23 – Dated yesterday (22nd) Dell is now recommending rolling back BIOS if you have applied it.  This worked fine for me on a R430 from 2.7.0 to 2.6.0.

Jan 24Duncan has some info on working around the pulled patches with Update Manager, because otherwise you’ll get an error.

Feb 26Dell has released new BIOS 2.7.1 to address CVE-2017-5715

March 20 – VMware has released the microcode updates in what they are calling Update 1 G, see details here ESXi 6.5 Build 7967591 and vCenter 8024368

 

Link roundup:

VMware’s main KB

Dell’s main KB

Intel’s Response to reboots

ESXi Patch Info

Tagged , , , , , , , ,

How to send vCenter alarms to Slack

I’m spending some of my time in the new gig with my old sysadmin ops hat on.  We needed a quick easy way to keep an eye on alerts from a vSphere environment so….what else would be more fun than to funnel them to Slack?!  Easy peasy, even on the vCenter Appliance.  Let’s see how…

First you need to configure the integration on Slack.   In the channel you wish to see the alerts in, click the “Add a service integration” link.

Snip20150806_12

Now there is not any special integration with vSphere, we are going to be using a simple REST api to push the content.  Scroll down to “Incoming WebHooks”

Snip20150806_13

Now you need to approve the integration verifying the chat room and click the button:

Snip20150806_14

The outcome of this will be a long URL you will need for the script.

Now we need to get your script ready. Now remember this is on vCenter (windows OR appliance), not ESXi.  Much credit to this guy that created a simple script for Zabbix, as this is a hacked up version of it.   The key here is using the environment $VMWARE_ALARM_EVENTDESCRIPTION which I use because it’s short and simple.   If you want other types of data check out the documentation here.

Now you just simply need to hook this script up to the alarm in vSphere:

Snip20150806_15

Sweet.  Cool.  Let there be (kind of) chatops.

But, I hear you asking…   What if you want to apply this to all your alarms??   Also…. easy peasy.   I just whipped together some powercli and bam.

That line will apply this script action to ALL alarms in the vCenter you connect to.   It will apply this by default to the Yellow to Red action level.    For now I wanted this to trip on all four cases so I looked a little deeper and found this will do it:

Now if you are like me and you screw this up along the way, you may have to clear out the actions across the board.  This line will do that for you:

 

 

Tagged , , , ,

Where is the OVA/OVF for vCenter 6.0 Appliance?

moooar_vsphere6You might have noticed already that the new vCenter Appliance (VCSA) 6.0 is only being distributed as an .ISO.  While I do wish an OVA/OVF would have been available too, the new installer is pretty sweet.

Snip20150314_36

When you mount this .ISO, you get a web page interface that cleanly and clearly walks you through deploying directly to an ESXi host.  In my opinion this is a pretty good user experience, which is something that has been lacking at the install time of these products for quite a while.  (Side note – check out the VIO installer too.  It’s slick as well.)

I do think there are a few isolated use cases for directly using an OVA or OVF however.  A vCD (or other hosted environment) is a good example, when you do not have access to the underlying hosts.  Or if you encounter some issues in your browser, perhaps due to draconian IT security measures.  All is not lost!   The OVA is actually hiding in the ISO, you just have to find it.

 

Disclaimer: This is not recommended or supported by VMware.  There are a lot of user input checks that go into the new installer, so use it when you deploy your VC for real.  This should only be used for testing/sandboxing/may blow up in your face. There I said it, so don’t yell at me when you break something!

 

The ISO itself will look like this on disk:

Snip20150314_30

When you mount it, the directory structure looks like:

Snip20150314_31

In the “vcsa” directory, you will find a file with no extention like this:

Snip20150314_32

This is the actual .OVA   Copy it somewhere local so it’s writable and add the .OVA extension.

Snip20150314_33

Now extract it.  Did not know you can just use an unzip utility to extract an OVA?  Well now you do.

Snip20150314_34

Aaaaaand here are your familiar files!

Snip20150314_35

Now I will warn you, you could will have a bad day if you directly try to use this .ovf with vCD.  It seemed to work fine for me in vCenter, but vCD does not understand a lot of the new options and fields in the format of this file.  The outcome is it’s really difficult to deploy anything other than an embedded node (PSC+VC).  If you want to deploy JUST the PSC or VC services you have to do quite a bit of editing.  Luckily I asked around before I spent time on this and I found the VMware HOL Wizard Doug Baer (@dobaer) had already pulled his hair out getting it to work.   So behold!  Customized .OVF files for either PSC or VC nodes!

I warn you, your results may/will vary with these so the usual disclaimers about NOT using this in production and NOT calling support and expecting help apply.

You can find the contents of these files on a Github repo of mine here, and again all credit to Doug for these:

Snip20150314_39

 

There you have it.  Go get some vSphere 6 vCenter appliance action on.

Tagged , , , , ,