Tag Archives: devops

Experiment: Pooling in vRA & Code Stream


I recently attended DevOpsDays Rockies which is a community oriented DevOps conference (check them out in your area, it was great!).  I saw a talk by @aspen (from Twitter/Gnip) entitled “Bare Metal Deployments with Chef”.   He described something he/they built that, if I recall correctly, uses a PXE/Chef/MagicpixieDust to pull from a pool of standby bare metal hardware to fully automate bringing it into a production cluster for Cassandra (or what have you).

This got me thinking on something I was struggling with lately.  Whenever I develop blueprints in Application Director / Application Services, or just vRA/Code Stream, the bulk of the time I just hit the go button and wait.  Look at the error message, tweak and repeat.  The bottleneck by far is in waiting for the VM to provision.  Partly this is due to the architecture of the products, but also it has to do with the slow nested development environments I have to use.  We can do better…..!

Products using pooling

I then started thinking about what VDM / Horizon View have always done for this concept.  If I recall correctly, as it’s been years and years since I’ve worked with it, to speed up deployments of a desktop to a user, a pool concept exists so that there will always be one available on demand to be used.   I don’t have much visibility into it but I am also told the VMware Hands On Labs does the same – keeps a certain number of labs ready to be used so the user does not have to wait for it to spin up.  Interesting.

The idea

So I thought – how could I bring this upfront deployment time to the products I’m working with today to dramatically speed up development time?   And this is what I built – a pooling concept for vRA & Code Stream managed by vRO workflows.

Details – How Redis Works

When planning this out I realized I needed a way to store a small bit of persistent data.   I wanted to use something new (to me) so I looked at a few NoSQL solutions since I’ve wanted to learn one.  I decided on Redis as a key value store, and found Webdis which provides a light REST api into Redis.

I couldn’t find any existing vCO plugins for Redis I/O which is fine, the calls are super simple:

Example of assigning a value of a string variable:

Snip20150517_5The redis command is: “set stringName stringValue”
So the webdis URL to “put” at is “http://fqdn/SET/stringName stringValue”

Then to read the variable back:

Snip20150517_6The redis command is: “get stringName stringValue”
So the webdis URL to “get” at is “http://fqdn/GET/stringName”

Easy peasy. There is similar functional for lists, with commands to pop a value off either end of the list.  This is all I needed, a few simple variables (for things like the pool size) and a list (for things like the list of VMs storing IP addresses & names).

So in vCO I just created a bunch of REST operations that used various number of parameters in the URL line:

I found the most efficient way to run these operations was to parametrize the operation name, and pass it to a single workflow to do the I/O

Details – Workflow(s)

The bulk of the work for this pooling concept is done in the following workflow that runs every 15 minutes.

Snip20150517_8In general it works like this:

  • Check if the workloads are locked – since it can take time to deploy the VMs, only one deployment will be going at a time.
    • If locked, end.
    • If not locked, continue.
  • Lock the deploys.
  • Get the pool max target (I generally set this to 10 or 20 for testing).
  • Get the current pool size (the length of the list in Redis.  much faster than asking vSphere/vRA).
  • If the current size is not at the target, deploy until it is reached.
  • Unlock the deploys.
  • Profit.

I did not have to do it this way, but the nested workflow that does the actual VM deployments is requesting vRA catalog items.

In Action

After I got it fully working and the pool populated, you can check the list values with this type of Redis query:


Redis: lrange vmlist 0 -1 (-1 means all)
Webdis: http://fqdn/LRANGE/vmlist/0/-1

The matching machines in vSphere:


In Action – Code Stream

Normally in a simple Code Stream pipeline you would deploy a VM by requesting the specific blueprint via vRA like this:


In this solution, instead I use a custom action to grab the VM from the pool and return the IP back to the pipeline as a variable.  Then I treat the VM like it’s an existing machine and continue on and at the end delete the machine.


This reduces the list in redis by one, so the next time the scheduled workflow runs that checks the list size it will deploy a new one.

(Kind of) Continuous Deployment

I have a job in Jenkins that builds the sample application I am using from source in Git, pushes the compiled code to Artifactory and does a post build action that calls Code Stream to deploy.


I wanted to see if there were any bugs in my code, so I wanted this whole thing to run end to end over and over and over…   I configured the Jenkins job to build every 30 minutes.  I went on vacation the week after I built this solution so I wanted to see if over time anything broke down.  Amazingly enough it kept on trucking while I was gone, and even got up to the mid 700’s in Jenkins builds.   Neat!


Jenkins builds



Code Stream executions

Code Stream executions


To my surprise, this actually works pretty darn well.  I figured my implementation would be so-so but the idea would get across.  It turns out, what I’ve built here is darn handy and I’ll probably be using it the next time I am in a development cycle.

Post any questions here and I’ll try to answer them.   I’m not planning to post my workflows publicly just yet, fyi.

Tagged , , , , , , , , , , , ,

What I’ve been up to lately: CTO Ambassador, vExpert 2015, VCP6-Cloud & Conferences

Busy time in the salt mine day job!  RADIO_CTO_v6 Hope this does not read as a bunch of humble bragging, just keeping tabs on all the happenings here on the blog.

I’ve recently joined a field facing group at VMware called the CTO Ambassadors on a rotation for the next two years.  This is really exciting for me as it is similar in intent to a program that I helped build in my past life in support.  This CTOA program in their words:


“focuses on creating and developing technology focused communities that span R&D and the field. This enables the effective flow of information and knowledge, backwards and forwards, between R&D and our field, and ultimately with our customers and partners, which in turn drives more profitable relationships, greater revenue, and new and improved products. “

I am excited to be able to take part in this experience.  The next year or two will be exciting to have a chance at helping shape many things internal to the company but also have a clear priority of taking part in other communities for the company.

vexpert_logoAlso, I got word that I was renewed as a VMware vExpert for 2015.  That’s fun too!  This is a designation for being active in the social media and blogging communities.  And is kind of a fun title too.

I passeCloud_VCPd the new VCP6-Cloud test this week.  I will admit it covers such a broad range of topics that it is a bit difficult in parts as I do not touch many of these too often any more.  And there are a number of questions that would purely be answered by just glancing at a GUI.   I used to be able to breeze in to the core VCP exam, take the test in 20 minutes and pass without ever thinking twice when I was in a purely a vSphere focused role.  Not sure if it is the test or it is me now.  Probably both column A and column B.

devops-days-rockiesFinally, I signed up for a few conferences coming up in Colorado that look to be exciting.  GlueCon in May (Broomfield, Colorado) and DevOps Days Denver.  Looking forward to hearing some great talks on some hot topics right now.


Tagged , , , , , ,

Local Docker Registry Update

It appears since I last wrote about creating a local and persistent Docker registry on CentOS they changed the default behavior to force secure communication.   In basic environments like I use and build in a lab, SSL is just a headache best left alone.

Doing docker push now with docker version 1.3.2 I get the error:

The best solution I found was to add this option to /etc/sysconfig/docker like the following [1] [2]

Restart Docker, and then all is well in Docker push land once again.



Tagged , , ,

Random Thoughts on DevOps

DevOps is one of those trends that is many things to many people.  I’ve been meaning to write this for a while now and Duncan’s recent post reminded me of that.

First off – I am many things but currently an authority on everything DevOps in the big scheme of things I am not.  I’m just a guy that’s done a few things and knows a few other things.  However, I wanted to share a few nuggets I’ve come across in the last year on this topic.  Some a bit dated now but that’s ok.  It’s all good stuff.

(side note…  the amazon links are just for ease of use.  they amazingly don’t allow affiliate link programs here in colorado…)


Snip20141103_13The Phoenix Project 
By Gene Kim, Kevin Behr, and George Spafford
Published: Jan 2013

Recently Duncan Epping posted his review of this book here.  I won’t rehash his summary but just say in my mind it is an excellent exposure to the subject no matter what your role or experience is.  The start of it gave me chills of being back in a chaotic operations environment – a place I hope to never find myself again (unless it’s in a role to clean up the mess like the protagonist). And throughout the book I found it interesting the author was able to really depict the evolution of the topic without name dropping specific vendors or technologies.  That’s impressive in a space dominated by hipster devs and brogrammers.   (wow I feel like I’m an old man yelling at the kids to get off my lawn…)

In short – you must read it if your job is ops, dev, anywhere in between, or work in IT in any way.

The Goal: A Process of Ongoing ImprovementSnip20141103_12
By Eliyahu M. Goldratt and Jeff Cox
Published: 1984

If you geek out on the overall ideas of Phoenix Project, and not just the technology aspect of it, reading The Goal might be for you.  If you recall in Phoenix, there are a few scenes where the characters visit a manufacturing plant to illustrate a point on bottlenecks and handoff of work. If I recall correctly this book is briefly referenced in one of those scenes.   The Goal is very very very similar to Phoenix in so many storyline ways that is obvious the authors in 2013 of Phoenix used it as a blueprint for telling their own story.  That took a small bit of the magic for me from Phoenix after reading this older book, but that’s ok – it’s still excellent in it’s own right.   The Goal is to manufacturing what Phoenix is to IT.  Call it BizPlantOps maybe?   It explores the breaking down of preconceived notions of policies and procedures that are followed in manufactoring because they are just accepted practice, and how to improve on them to compete in the market.   You can hear the methaphors for DevOps already right?

In short – if you are strictly into the technology aspect of DevOps this one will bore you.   But if you geek out on the theory of DevOps, the drivers, the principles, and the roots of it, this could be a good read for you.
Other Reading:

Snip20141103_14DZone’s 2014 Guide to Continuous Delivery
(Free PDF download with registration)

I came across this 35 page PDF recently when someone posted to a company collaboration page “if you read nothing else about DevOps, this should be it”.   I whole heartedly agree.   They do an amazing job breaking down the buzz words Continuous Integration, Continuous Delivery and Continuous Deployment into the core of what they actually are.  Sure there are paid ads and vendor highlights sprinkled throughout but it in no way feels like they are spouting sales jargon, but true research findings.  It feels more like *this is what all the details are*  and *this is the current vendor offerings are to do it* and doesn’t make an endorsement either way or fall into the zealot mindframe of the one single tool is the best always and forever no matter what.

In short – it’s 35 pages and free.  Read it.


I’ll just drop a few links here on videos I’ve enjoyed.  These are all from many moons ago now as I found them while doing research for a now older project.

Snip20141103_17How Do We Better Sell DevOps? (PuppetConf 2013)
From Gene Kim (author of Phoneix Project)

Keynote: Stop Hiring Devops Experts (And Start Growing Them) (PuppetConf 2013)
From Jez Humble

PCI-DSS and continuous deployment at Etsy


There we go, that clears out my current list of OH YOU SHOULD READ/WATCH THIS list.    Drop your current favorites in the comments!


Tagged , , ,

How to use a local persistent Docker registry on CentOS 6.5

UPDATE: Dec 16 2014, I found a new option is needed now using Docker version 1.3.2   See more here

There a bunch of blogs out there showing a tutorial on how to use a local docker registry but none of them (that I have found) have it boiled down to the absolute simplest syntax and terms.   So here you go!

First off terminology Docker Hub is where images are typically pulled from when you just type the normal “docker pull blah” commands.  A registry is what it’s referred to as instead of a hub or repo.   To save time and bandwidth here is how you can stand up a persistent local registry to store your images.  Persistent meaning the image data is kept after the image is discarded.

Syntax here is working on CentOS 6.5

1) Install the needed bits.  This is no different than normal.

2) Start docker

3) Fire up the example registry.  This downloads and runs the registry image, exposes port 5000, and links local dir /opt/registry to /tmp/registry within the container.   This is key.  Otherwise, after the container stops the images go poof.

4) We could do this locally on this first machine, but we’ll show the syntax from somewhere else to illustrate.   On some other machine, first do the same install steps above to install the EPEL rpm and install Docker.   Then pull the images you want:


5) List the images, and we see this image separates out a few versions of the OS.  CentOS7 is the latest (see how the IMAGE ID matches 87e5b…), CentOS6 is image 68edf..  and CentOS5 is 504a65…


6) Add some tags to gives the images a new identity.  Replace “docker-reg” with your docker registry hostname.

7) List images again to verify


8) Now push the tagged images to our local registry

9) Lastly, on some third machine with docker already installed (this post makes this handy by deploying these nodes as a catalog item), pull the time.  Notice it’s WAY fast now.  ~14 seconds in this screenshot.  Notice we only have the latest centos tagged, just pull the others and you’re good.

Compare this to pulling from docker hub at about ~2:20.   For a large image like the SpringTrader app I built and this would cut down about an hour download time dramatically.

I wanted to compare this to the SpringTrader app, so I pulled it earlier and began pushing it to my local registry.  One thing I noticed was it does buffer the image to disk when you push so be aware you will need the disk space (and time) available for this.   The time savings will happen later on subsequent deployments.  And i crashed my VM when running out of space the first time….

Then on some other node

Boom.  In about 10% of the time it normally takes to deploy this image she’s up and running!  It took about 13 minutes to download from Docker Hub, and about 2 minutes from my local registry.  That’s a win if you have a need to do this over and over.



Where are the images stored?

On a docker node, what I have been using to refer to the base machine and not the containers themselves, I found the docker files here.


On the local registry, i found the files here.  Remember we told it to use /opt/registry on the base machine and map that to /tmp/registry within the container?


Tagged , , ,

Automated testing with Selenium & PHPUnit

I am working on a much larger project right now that is using Selenium to do unit testing, and found this installation procedure to be a PITA on CentOS so I thought I would share my experience with the interwebs to relieve some pain for others.  I am very new to using these tools so if anyone has suggestions for improvement I am all ears, though this post will be strictly on the install portion.   Expect a much longer and more interesting post in the future.

This is not intended to be a primier on what Selenium is at all.  If you are interested check out this video, docs are located here for PHPUnit, here for Selenium,  and here for Composer (to cover all my reference linkage).


This worked as of August 18, using Centos 6.4, JRE 7u67, Selenium 2.42.2, PHP 5.3.3, PHPUnit 4.1.6


First, install java if you need it.  I am using the super awesome alternatives command to manage the symlinks of multiple versions.

Get from Sun: jre-7u67-linux-x64.rpm
alternatives –install /usr/bin/java java /usr/java/jre1.7.0_67/bin/java 1
alternatives –config java
pick the new one
At the time of this post, this was a version that worked for me
# java -version
java version “1.7.0_67”
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)

Now we want to install Selenium

ln -s /usr/local/bin/selenium-server-standalone-2.42.2.jar /usr/local/bin/selenium-server-standalone.jar 

add to .bash_profile, then relogin or source .bash_profile
alias selenium=”java -jar /usr/local/bin/selenium-server-standalone-2.42.2.jar”

Install packages for X and Firefox, since I’m running this in a headless VM
yum install -y java firefox Xorg xorg-x11-server-Xvfb

There are multiple options to install PHPUnit.   Pear is the easiest as it was what I was familar with but appears to be going end of life at the end of 2014.  Composer is brand new to me – you kids and your fancy package management systems!!!!1 – but looks pretty useful.

(1) via PEAR (end of life at end of 2014)
yum install php php-pear php-xml php-devel -y
pear channel-update pear.php.net
pear config-set auto_discover 1
pear install phpunit
pear install phpunit/PHPUnit_Selenium

(2) via Composer
yum install php php-xml php-devel php-pdo git -y
curl -sS https://getcomposer.org/installer | php
mv composer.phar /usr/local/bin/composer
vi composer.json
Add this content:
    “require-dev”: {
        “phpunit/phpunit”: “4.1.*”,
        “phpunit/php-invoker”: “*”,
        “phpunit/dbunit”: “*”,
        “phpunit/phpunit-selenium”: “>=1.2”
Then run this:
composer install 

Docs say to add this to your path:
But I installed my stuff in a subdir as root, so I also added this to the path also

# phpunit –version
PHPUnit 4.1.6 by Sebastian Bergmann

Now to setup an X session.  Many ways to do this, here is a quick and dirty way:

Xvfb :99 -screen 0 800x600x16 &
export DISPLAY=:99
Now open a new session to the host.

Now for the actual unit test code.   I’ll share a simple example that checks this blog for something as simple as the TITLE html tag.

class JAASTest extends PHPUnit_Extensions_Selenium2TestCase
    protected function setUp()
    public function testTitle()
        $this->assertEquals(‘Josh As A Service | www.JaaS.co‘, $this->title());
Save it as  JAASTest.php
Invoke it by:
phpunit JAASTest.php

PHPUnit 4.1.6 by Sebastian Bergmann.
Time: 13.65 seconds, Memory: 5.00Mb
OK (1 test, 1 assertion) 
YAY!! Automated testing by little robot minions. Just what I have always wanted.
There is a ton more you can do with this type of testing which I will be exploring soon now that I have this up and working.  Check back later on in the year for the results of the project!
Tagged , , ,