Category Archives: Random Scribblings

HOWTO: Change Storage Policies for VSAN across entire clusters /w PowerCLI

I had a need recently to switch the applied storage policy across a ton of VMs, but I didn’t want to change the default policy that comes out of the box.   A tough proposition as I found no easy way to do it.  It took quite a bit of googling and trial and error but I came up with this two liner to get it done, so here you go world – go forth and policy change if you need such a thing.

The first line applies it to the VM object, then the next applies it to all the disks.  Easy peasy.

 

Technology found our new best friend.

Last night I built a robot that brought us to our new best friend. Meet Cash.

Cash

Before I explain this strange statement, first let’s back up.

Two weeks ago we found our beloved Maddie was stricken with a tumor on her spleen that ruptured.  I won’t dive into the heartbreaking details, but you can read about it here, here and here.

maddie_camping

To summarize: heartbroken.   That damn wonderful dog lived a great life and will never be replaced.  But we have found we’re a two dog family.  Enter the idea of visiting shelters….which is always fun..!

IMG_6736

After a few misses, we found just how competitive adopting dogs is in Boulder.  Yes, competitive.

Forget cycling, running, and climbing – the most competitive sport in Boulder is trying to adopt a dog from the pound link

Dogs fly out of the Boulder Humane Society.  There was one Jen was interested in that was adopted within hours of her becoming available.  We heard of one from employees that was going HOME within 45 minutes of stepping into the adoptables area.   Seriously.

The employees say to just keep an eye on the website.  So that’s what we did for a bit.  We noticed it was updated frequently throughout the day.  But there was no way to be notified of new dogs.  Enter my light bulb moment.

I saw there was no RSS feed, and (of course) no API.  So I took a glance at the HTML source and saw it would be super easy to screen scrape.  Muahhaha…… this will be easy peasy!    With just a little bit of hacking last night I had a working system that scraped their webpage every 15 minutes, stored it in a local database, and sent us an email when a dog became available!  Ha! Leg up, take that one, Boulder animal people.  Dog adoption performance enhancing drugs.

In the morning I surmized that wasn’t nearly geeky enough.  I added functionality to email us when a dog appeared to be adopted (wasn’t listed any more).  And since email is SO year 2000s, I spun up a new twitter account and had it tweet and direct message us when a dog showed up and went home.  I dub thee: Dog(S)talker.  Get it?  Dog Stalker.  Dogs Talker.  I kill me…

Low and behold…while I was out with the kid on his bike and Jen working on an extension to the chicken coop, DING. DM from the new robot:

Snip20160403_15

Due to an unfortunate typo in the code it is missing the details of the dog but still….. the fucking thing worked… A quick click on the link showed it was a 1 year old, Australian Kelpie Mix, and about 45 pounds.  Check check and check all the boxes!  I yelled across the street: “JEN!”  to which I immediately heard the reply, “I’M GETTING READY TO GO [to the shelter]!”

15 minutes later I received this:

IMG_6740

So an absolute max of 30 minutes from the time she was posted to the website to one of us showing up to check her out.

Long story short, he’s perfect for us.  I’ll post the code to github soon.   Perhaps if this is useful to anyone else I can add others to the notifications.

Snip20160403_16

 

Tagged , , ,

How to send vCenter alarms to Slack

I’m spending some of my time in the new gig with my old sysadmin ops hat on.  We needed a quick easy way to keep an eye on alerts from a vSphere environment so….what else would be more fun than to funnel them to Slack?!  Easy peasy, even on the vCenter Appliance.  Let’s see how…

First you need to configure the integration on Slack.   In the channel you wish to see the alerts in, click the “Add a service integration” link.

Snip20150806_12

Now there is not any special integration with vSphere, we are going to be using a simple REST api to push the content.  Scroll down to “Incoming WebHooks”

Snip20150806_13

Now you need to approve the integration verifying the chat room and click the button:

Snip20150806_14

The outcome of this will be a long URL you will need for the script.

Now we need to get your script ready. Now remember this is on vCenter (windows OR appliance), not ESXi.  Much credit to this guy that created a simple script for Zabbix, as this is a hacked up version of it.   The key here is using the environment $VMWARE_ALARM_EVENTDESCRIPTION which I use because it’s short and simple.   If you want other types of data check out the documentation here.

Now you just simply need to hook this script up to the alarm in vSphere:

Snip20150806_15

Sweet.  Cool.  Let there be (kind of) chatops.

But, I hear you asking…   What if you want to apply this to all your alarms??   Also…. easy peasy.   I just whipped together some powercli and bam.

That line will apply this script action to ALL alarms in the vCenter you connect to.   It will apply this by default to the Yellow to Red action level.    For now I wanted this to trip on all four cases so I looked a little deeper and found this will do it:

Now if you are like me and you screw this up along the way, you may have to clear out the actions across the board.  This line will do that for you:

 

 

Tagged , , , ,

A change…or pivot if you will…..

Pivotal_Logo_200I have been at VMware for 7 years (this week on the dot actually!).  That is a half a lifetime in IT Dog Years.  In that time I have done many different things, and been to many different places.  I saw (and at times helped (or tried to help) ) virtualization mature from a fringe lab thing that would never run production workloads efficiently and easily, to an established vendor that most people are using in some way.  Quite a ride!

Just after the July 4th holiday I will be (metaphorically, though not geographically) be walking a few blocks up the hill in Palo Alto from the VMware campus to a sister EMC Federation company, Pivotal.  I’ll be leaving the current Pre-Sales gig and getting my hands dirty directly in technology as a main focus.  I’m excited!

www.jaams.co

micro-services1-297x250I plan to continue the blogging weird and silly projects on here, though it will stray from a VMware focus to more broad devopsy topics in general.   Hence the slight change in name (mostly as a joke that I was told at GlueCon recently) – Josh as a (Micro) Service!  Kind of catchy don’t you think?

I’ll spare you all the pontificating on merits of focusing on one thick technology stack made up of all kinds of mashed together bits being a monolithic focus, and now for the future breaking it down into singular focus areas and doing each of them well……I don’t know… This joke might not work entirely, but I get a good laugh out of it anyway.

Onward!

“Security is mostly a superstition. It does not exist in nature, nor do the children of men as a whole experience it. Avoiding danger is no safer in the long run than outright exposure. Life is either a daring adventure, or nothing.”
Helen Keller

“Live every week like it’s Shark Week.” – Tracy Jordan

“It’s more fun to be a pirate than to join the Navy.” – Steve Jobs

Tagged , , ,

vCenter alarms for VSAN not included out of the box

Snip20150309_7I have been testing VSAN in my whitebox hosts of my home lab lately, as I consider migrating off the smattering of Synology and other datastores.  My first mistake was using a spinning drive that apparently was already bad as I quickly got some permanent device loss errors after spinning up some VMs.  I was concerned why I never got any actual VSAN alarms, and just randomly stumbled on the reason why via Reddit.

Turns out, vCenter does not contain any VSAN alarms that map to VOBs.   This KB shows how to add a few of them, but Mister VSAN Rawlinson Rivera’s blog post shows a longer list that appears pretty comprehensive and also includes a bit of powershell code to add the alarms for a specific cluster.  I could not get the code to copy/paste from the blog without characters getting all screwy so it took quite some time to clean up.   Here on my git repo is a version of it that works as is.

Tagged , ,

How to modify a boot2docker .ISO for Docker Machine

I have been doing quite a bit of work with VMware vSphere + Docker + Machine lately.  I helped build a Hands On Lab for the recent VMware Partner Exchange conference with it.  I can’t promise, but it looks like it might be available publicly.  If you have access – it’s named HOL-SDC-1430.  It has been a difficult process as we’re in such alpha territory.  Things sometimes work, sometimes they don’t, and then they change a rev later..

Snip20150212_9An example of this is three specific items needed in a boot2docker image that is used as the docker host VM.  We need VMware tools (or open vm tools), some networking updates, and in my case the change to docker settings to allow pulling from an insecure local registry.

VMware’s Cloud Native Apps R&D has forked the main boot2docker repo and done the tools work and networking work (probably among others too) but I had to dive in and figure out how to edit it further to allow for a new docker option.  I really can’t claim to be an authoritative source on the docker and boot2docker side of things here but the googles failed me on a single location for all this information so here you go!

1) First you have to clone from a specific branch of VMware’s Cloud Native Apps git repo.  ovt stands for open vm tools.  See the diff’s here.

2) I found a Dockerfile is what is used to customize the iso.  How it works is the container is built from the file, and a number of other dependencies in the sub directories but is written to print out the iso data when run.  Pretty clever whoever first came up with this method.    So to do my hard-coding-not-best-practice-but-solves-my-needs I edit the Dockerfile as follows by removing the dependency on the b2d version, and just pull the latest Docker.   1.5 came out this week and I was wanting to pick up those updates.

Snip20150212_7

3) I also needed to use a local repository without certificates since I am building lab environments, so I added a new config variable for $DOCKER_REG to make it easier to update later.

Snip20150212_8

4) Now the rest is just following the b2d documentation.  Build the container with:

5) And write out the ISO with:

One thing that hung me up for a bit was machine doesn’t do any checksum on the ISO you tell it to use.   If machine sees the image already exists it WILL NOT overwrite it on the target datastore, so remember to delete it and let it upload the new one.  Very important.

Tagged , , , , ,

Local Docker Registry Update

It appears since I last wrote about creating a local and persistent Docker registry on CentOS they changed the default behavior to force secure communication.   In basic environments like I use and build in a lab, SSL is just a headache best left alone.

Doing docker push now with docker version 1.3.2 I get the error:

The best solution I found was to add this option to /etc/sysconfig/docker like the following [1] [2]

Restart Docker, and then all is well in Docker push land once again.

 

 

Tagged , , ,

vCAC Remote Console remote privilege escalation

LINK TO VMware Advisory VMSA-2014-0013

LINK to CVE-2014-8373

If you have a vCAC (or the new name vRealize Automation, vRA) system on an untrusted network you should read up on this. (Or in truth, one could argue if you have it all in a production environment….).

VMware vCloud Automation Center has a remote privilege escalation vulnerability. This issue may allow an authenticated vCAC user to obtain administrative access to vCenter Server.

To be clear, this is not a broad virtual machine remove console (VMRC) issue, but how it is implemented in vCAC/vRA.  vSphere is not affected, vCD is not affected.   vRA 6.2 is not affected as “connect using VMRC” is disabled.  The workaround for the older versions is to disable this method.

Tagged , , , ,

Suggested reading: DZone’s Guide to Enterprise Integration

Snip20141117_8Previously I posted about DZone’s Guide to Continuous Delivery (which is excellent), and now I have been reading their guide on Enterprise Integration.   I highly suggest checking it out.   I really geek out on the trends of SOA and microservices.   The idea of “dumb pipes and smart endpoints” is intriguing to me (which to be fair I guess they are crediting this post as their source).   Also I find it fascinating how the same companies keep coming up as the example case, like Netflix, Etsy, Soundcloud etc.   I would hate to be one of their competitors…

Tagged ,

Random Thoughts on DevOps

DevOps is one of those trends that is many things to many people.  I’ve been meaning to write this for a while now and Duncan’s recent post reminded me of that.

First off – I am many things but currently an authority on everything DevOps in the big scheme of things I am not.  I’m just a guy that’s done a few things and knows a few other things.  However, I wanted to share a few nuggets I’ve come across in the last year on this topic.  Some a bit dated now but that’s ok.  It’s all good stuff.

(side note…  the amazon links are just for ease of use.  they amazingly don’t allow affiliate link programs here in colorado…)

BOOKS:

Snip20141103_13The Phoenix Project 
By Gene Kim, Kevin Behr, and George Spafford
Published: Jan 2013

Recently Duncan Epping posted his review of this book here.  I won’t rehash his summary but just say in my mind it is an excellent exposure to the subject no matter what your role or experience is.  The start of it gave me chills of being back in a chaotic operations environment – a place I hope to never find myself again (unless it’s in a role to clean up the mess like the protagonist). And throughout the book I found it interesting the author was able to really depict the evolution of the topic without name dropping specific vendors or technologies.  That’s impressive in a space dominated by hipster devs and brogrammers.   (wow I feel like I’m an old man yelling at the kids to get off my lawn…)

In short – you must read it if your job is ops, dev, anywhere in between, or work in IT in any way.

The Goal: A Process of Ongoing ImprovementSnip20141103_12
By Eliyahu M. Goldratt and Jeff Cox
Published: 1984

If you geek out on the overall ideas of Phoenix Project, and not just the technology aspect of it, reading The Goal might be for you.  If you recall in Phoenix, there are a few scenes where the characters visit a manufacturing plant to illustrate a point on bottlenecks and handoff of work. If I recall correctly this book is briefly referenced in one of those scenes.   The Goal is very very very similar to Phoenix in so many storyline ways that is obvious the authors in 2013 of Phoenix used it as a blueprint for telling their own story.  That took a small bit of the magic for me from Phoenix after reading this older book, but that’s ok – it’s still excellent in it’s own right.   The Goal is to manufacturing what Phoenix is to IT.  Call it BizPlantOps maybe?   It explores the breaking down of preconceived notions of policies and procedures that are followed in manufactoring because they are just accepted practice, and how to improve on them to compete in the market.   You can hear the methaphors for DevOps already right?

In short – if you are strictly into the technology aspect of DevOps this one will bore you.   But if you geek out on the theory of DevOps, the drivers, the principles, and the roots of it, this could be a good read for you.
Other Reading:

Snip20141103_14DZone’s 2014 Guide to Continuous Delivery
(Free PDF download with registration)

I came across this 35 page PDF recently when someone posted to a company collaboration page “if you read nothing else about DevOps, this should be it”.   I whole heartedly agree.   They do an amazing job breaking down the buzz words Continuous Integration, Continuous Delivery and Continuous Deployment into the core of what they actually are.  Sure there are paid ads and vendor highlights sprinkled throughout but it in no way feels like they are spouting sales jargon, but true research findings.  It feels more like *this is what all the details are*  and *this is the current vendor offerings are to do it* and doesn’t make an endorsement either way or fall into the zealot mindframe of the one single tool is the best always and forever no matter what.

In short – it’s 35 pages and free.  Read it.


Presentations:

I’ll just drop a few links here on videos I’ve enjoyed.  These are all from many moons ago now as I found them while doing research for a now older project.

Snip20141103_17How Do We Better Sell DevOps? (PuppetConf 2013)
From Gene Kim (author of Phoneix Project)

Keynote: Stop Hiring Devops Experts (And Start Growing Them) (PuppetConf 2013)
From Jez Humble

PCI-DSS and continuous deployment at Etsy

 

There we go, that clears out my current list of OH YOU SHOULD READ/WATCH THIS list.    Drop your current favorites in the comments!

 

Tagged , , ,