I recently attended DevOpsDays Rockies which is a community oriented DevOps conference (check them out in your area, it was great!). I saw a talk by @aspen (from Twitter/Gnip) entitled “Bare Metal Deployments with Chef”. He described something he/they built that, if I recall correctly, uses a PXE/Chef/MagicpixieDust to pull from a pool of standby bare metal hardware to fully automate bringing it into a production cluster for Cassandra (or what have you).
This got me thinking on something I was struggling with lately. Whenever I develop blueprints in Application Director / Application Services, or just vRA/Code Stream, the bulk of the time I just hit the go button and wait. Look at the error message, tweak and repeat. The bottleneck by far is in waiting for the VM to provision. Partly this is due to the architecture of the products, but also it has to do with the slow nested development environments I have to use. We can do better…..!
Products using pooling
I then started thinking about what VDM / Horizon View have always done for this concept. If I recall correctly, as it’s been years and years since I’ve worked with it, to speed up deployments of a desktop to a user, a pool concept exists so that there will always be one available on demand to be used. I don’t have much visibility into it but I am also told the VMware Hands On Labs does the same – keeps a certain number of labs ready to be used so the user does not have to wait for it to spin up. Interesting.
So I thought – how could I bring this upfront deployment time to the products I’m working with today to dramatically speed up development time? And this is what I built – a pooling concept for vRA & Code Stream managed by vRO workflows.
Details – How Redis Works
When planning this out I realized I needed a way to store a small bit of persistent data. I wanted to use something new (to me) so I looked at a few NoSQL solutions since I’ve wanted to learn one. I decided on Redis as a key value store, and found Webdis which provides a light REST api into Redis.
I couldn’t find any existing vCO plugins for Redis I/O which is fine, the calls are super simple:
Example of assigning a value of a string variable:
The redis command is: “set stringName stringValue”
So the webdis URL to “put” at is “http://fqdn/SET/stringName stringValue”
Then to read the variable back:
The redis command is: “get stringName stringValue”
So the webdis URL to “get” at is “http://fqdn/GET/stringName”
Easy peasy. There is similar functional for lists, with commands to pop a value off either end of the list. This is all I needed, a few simple variables (for things like the pool size) and a list (for things like the list of VMs storing IP addresses & names).
So in vCO I just created a bunch of REST operations that used various number of parameters in the URL line:
I found the most efficient way to run these operations was to parametrize the operation name, and pass it to a single workflow to do the I/O
Details – Workflow(s)
The bulk of the work for this pooling concept is done in the following workflow that runs every 15 minutes.
In general it works like this:
- Check if the workloads are locked – since it can take time to deploy the VMs, only one deployment will be going at a time.
- If locked, end.
- If not locked, continue.
- Lock the deploys.
- Get the pool max target (I generally set this to 10 or 20 for testing).
- Get the current pool size (the length of the list in Redis. much faster than asking vSphere/vRA).
- If the current size is not at the target, deploy until it is reached.
- Unlock the deploys.
I did not have to do it this way, but the nested workflow that does the actual VM deployments is requesting vRA catalog items.
After I got it fully working and the pool populated, you can check the list values with this type of Redis query:
Redis: lrange vmlist 0 -1 (-1 means all)
The matching machines in vSphere:
In Action – Code Stream
Normally in a simple Code Stream pipeline you would deploy a VM by requesting the specific blueprint via vRA like this:
In this solution, instead I use a custom action to grab the VM from the pool and return the IP back to the pipeline as a variable. Then I treat the VM like it’s an existing machine and continue on and at the end delete the machine.
This reduces the list in redis by one, so the next time the scheduled workflow runs that checks the list size it will deploy a new one.
(Kind of) Continuous Deployment
I have a job in Jenkins that builds the sample application I am using from source in Git, pushes the compiled code to Artifactory and does a post build action that calls Code Stream to deploy.
I wanted to see if there were any bugs in my code, so I wanted this whole thing to run end to end over and over and over… I configured the Jenkins job to build every 30 minutes. I went on vacation the week after I built this solution so I wanted to see if over time anything broke down. Amazingly enough it kept on trucking while I was gone, and even got up to the mid 700’s in Jenkins builds. Neat!
To my surprise, this actually works pretty darn well. I figured my implementation would be so-so but the idea would get across. It turns out, what I’ve built here is darn handy and I’ll probably be using it the next time I am in a development cycle.
Post any questions here and I’ll try to answer them. I’m not planning to post my workflows publicly just yet, fyi.