Tag Archives: vcac

Experiment: Pooling in vRA & Code Stream


I recently attended DevOpsDays Rockies which is a community oriented DevOps conference (check them out in your area, it was great!).  I saw a talk by @aspen (from Twitter/Gnip) entitled “Bare Metal Deployments with Chef”.   He described something he/they built that, if I recall correctly, uses a PXE/Chef/MagicpixieDust to pull from a pool of standby bare metal hardware to fully automate bringing it into a production cluster for Cassandra (or what have you).

This got me thinking on something I was struggling with lately.  Whenever I develop blueprints in Application Director / Application Services, or just vRA/Code Stream, the bulk of the time I just hit the go button and wait.  Look at the error message, tweak and repeat.  The bottleneck by far is in waiting for the VM to provision.  Partly this is due to the architecture of the products, but also it has to do with the slow nested development environments I have to use.  We can do better…..!

Products using pooling

I then started thinking about what VDM / Horizon View have always done for this concept.  If I recall correctly, as it’s been years and years since I’ve worked with it, to speed up deployments of a desktop to a user, a pool concept exists so that there will always be one available on demand to be used.   I don’t have much visibility into it but I am also told the VMware Hands On Labs does the same – keeps a certain number of labs ready to be used so the user does not have to wait for it to spin up.  Interesting.

The idea

So I thought – how could I bring this upfront deployment time to the products I’m working with today to dramatically speed up development time?   And this is what I built – a pooling concept for vRA & Code Stream managed by vRO workflows.

Details – How Redis Works

When planning this out I realized I needed a way to store a small bit of persistent data.   I wanted to use something new (to me) so I looked at a few NoSQL solutions since I’ve wanted to learn one.  I decided on Redis as a key value store, and found Webdis which provides a light REST api into Redis.

I couldn’t find any existing vCO plugins for Redis I/O which is fine, the calls are super simple:

Example of assigning a value of a string variable:

Snip20150517_5The redis command is: “set stringName stringValue”
So the webdis URL to “put” at is “http://fqdn/SET/stringName stringValue”

Then to read the variable back:

Snip20150517_6The redis command is: “get stringName stringValue”
So the webdis URL to “get” at is “http://fqdn/GET/stringName”

Easy peasy. There is similar functional for lists, with commands to pop a value off either end of the list.  This is all I needed, a few simple variables (for things like the pool size) and a list (for things like the list of VMs storing IP addresses & names).

So in vCO I just created a bunch of REST operations that used various number of parameters in the URL line:

I found the most efficient way to run these operations was to parametrize the operation name, and pass it to a single workflow to do the I/O

Details – Workflow(s)

The bulk of the work for this pooling concept is done in the following workflow that runs every 15 minutes.

Snip20150517_8In general it works like this:

  • Check if the workloads are locked – since it can take time to deploy the VMs, only one deployment will be going at a time.
    • If locked, end.
    • If not locked, continue.
  • Lock the deploys.
  • Get the pool max target (I generally set this to 10 or 20 for testing).
  • Get the current pool size (the length of the list in Redis.  much faster than asking vSphere/vRA).
  • If the current size is not at the target, deploy until it is reached.
  • Unlock the deploys.
  • Profit.

I did not have to do it this way, but the nested workflow that does the actual VM deployments is requesting vRA catalog items.

In Action

After I got it fully working and the pool populated, you can check the list values with this type of Redis query:


Redis: lrange vmlist 0 -1 (-1 means all)
Webdis: http://fqdn/LRANGE/vmlist/0/-1

The matching machines in vSphere:


In Action – Code Stream

Normally in a simple Code Stream pipeline you would deploy a VM by requesting the specific blueprint via vRA like this:


In this solution, instead I use a custom action to grab the VM from the pool and return the IP back to the pipeline as a variable.  Then I treat the VM like it’s an existing machine and continue on and at the end delete the machine.


This reduces the list in redis by one, so the next time the scheduled workflow runs that checks the list size it will deploy a new one.

(Kind of) Continuous Deployment

I have a job in Jenkins that builds the sample application I am using from source in Git, pushes the compiled code to Artifactory and does a post build action that calls Code Stream to deploy.


I wanted to see if there were any bugs in my code, so I wanted this whole thing to run end to end over and over and over…   I configured the Jenkins job to build every 30 minutes.  I went on vacation the week after I built this solution so I wanted to see if over time anything broke down.  Amazingly enough it kept on trucking while I was gone, and even got up to the mid 700’s in Jenkins builds.   Neat!


Jenkins builds



Code Stream executions

Code Stream executions


To my surprise, this actually works pretty darn well.  I figured my implementation would be so-so but the idea would get across.  It turns out, what I’ve built here is darn handy and I’ll probably be using it the next time I am in a development cycle.

Post any questions here and I’ll try to answer them.   I’m not planning to post my workflows publicly just yet, fyi.

Tagged , , , , , , , , , , , ,

vCAC Remote Console remote privilege escalation

LINK TO VMware Advisory VMSA-2014-0013

LINK to CVE-2014-8373

If you have a vCAC (or the new name vRealize Automation, vRA) system on an untrusted network you should read up on this. (Or in truth, one could argue if you have it all in a production environment….).

VMware vCloud Automation Center has a remote privilege escalation vulnerability. This issue may allow an authenticated vCAC user to obtain administrative access to vCenter Server.

To be clear, this is not a broad virtual machine remove console (VMRC) issue, but how it is implemented in vCAC/vRA.  vSphere is not affected, vCD is not affected.   vRA 6.2 is not affected as “connect using VMRC” is disabled.  The workaround for the older versions is to disable this method.

Tagged , , , ,

New vCAC & Application Services 6.1 template prep script (linux)

UPDATE: Dec 9 2014 – vCAC is renamed to vRealize Automation (vRA).   vRA 6.2 is dropping today and the pre-req script is posted here.

UPDATE: Dec 16 2014 – Doh!!  I was multitasking too much when i posted that last update.   The pre-req script wasn’t the point of this original post, but is still useful none the less.  To recap – the pre-req script is to ease setting up a vRA IAAS machine.   The template prep script is to ease setting up a linux template to be used _WITH_ vRA.
A great tool that flew under my radar in the most recent 6.1 release for AppD….er…Application Services and vCAC proper… is a script that does all the steps to prepare a linux template for you for both agents.   If you are at all familiar with this process, you’ll find it to be a huuuuuuuge time saver.

Original Post:
If you look at the documentation it’s quite cumbersome and full of potenial human error points.  This script will check all dependencies, install them where it can, and either prompt for the appropriate server names or accept inline input.

Getting Started
First pull the script off the AppD server and make it executable.

How to use it – Interactive

If you wanted to just dive in run it for interactive mode:

How to use it – Unattended

I update templates quite often in the lab environments I work in, so I like keeping a quick reference in a note that I can quickly cut & paste from.  Now that this script accepts inline inputs I could gain an extra sysadmin merit badge and just drop it into a shell script in a common place across all templates and just manually run that.  Easy Peasy.

So here’s the help page:


Here’s what I would run which tells it the three server names, not to install java,  not to check ssl certs, a timeout of 300 secs and not to prompt to confirm:

The last line is a handy step that prevents centOS templates to increment the nic# when cloning.  There could be a better way but it works.


… it does it’s thing…..and finishes with:



Now you’re ready to shut it down, take a snapshot, start data collection, and update your blueprint!

Tagged , ,

Directory as A Service: Part 2 – vCAC Integration

Directory as A Service: Part 1 – Intro & Setup
Directory as A Service: Part 2 – vCAC Integration


In the first post I introduced JumpCloud, a hosted off-premise directory service.   In this post I will show one way to integrate it into vCloud Automation Center (vCAC).

Getting Started
I got started with re-using a simple CentOS single machine blueprint I had already configured and which does a few configurations on boot already:


For simplicty, I like using the scripted method in a build profile to do simple stuff.   I don’t take any credit for this configuration as I totally just copied what others have done before me (sorry don’t have the link to the exact post handy).   This build profile mounts a network share, and runs a DNS script from that share to automatically add the new machine to my DNS.


Now the additions we’ll make to integrate into JumpCloud are as follows.  I split it into three scripts because I built this iteratively using some of their example code, but could have just as easy done it all in one. I’ll refer to the script numbers as they appear below for clarity:

I used all of JumpCloud’s example code as examples, but am posting my modifications to github here.



Script 2 – Installs the agent (we saw that syntax in the first post)


Script 3 – Assigns tag(s) to the system being deployed


Script 4 – Sets the configuration of enabling password auth, enable root, and enable multifactor

Go ahead and do the normal vCAC configurations and end up with a catalog item for a JumpCloud enabled CentOS machine.   I have not done anything else special to prompt the user for values here.  I could see it being useful as integrating this into ALL catalog requests and give the user the choice to enter (or choose from a list of) tags, or the authentication choices (password, public key, root…).


Let’s go ahead and request three nodes:



Shortly after the three machines show up in vSphere:


And when the deploying machine stage is complete, they show up in vCAC:


And it shows up in the JumpCloud console:


And the same method of authentication now works as we discussed previously.  Using google authenticator to log into one of my machines is pretty darn awesome I have to admit.



What now?
Now what about day 2 operations.  Well I could add some menu actions to add/remove the system from tags or modify the other settings, that could be useful.  But at a bare minium I wanted tear down to be clean.  Since I was able to get the machine creation be totally automated, I also wanted cleanup to be.  Since vCAC is a self service portal, you wouldn’t want the JumpCloud console to be full of machines that no longer exist.

You don’t want your console filling up with old machines like this


Because I took the easy way out and used the Build Profile scripting method of customizing machines from vCAC, I had to go a different route for tear down.   The easiest way to inject automation at this stage today with vCAC 6.1 is firing off a vCO workflow by using a machine state workflow.   So first I built a simple vCO workflow that runs the SSH command on the machine:


I had to look it up to get the syntax right but the input parameter you want to use is “vCACVm” of type “vCAC:VirtualMachine”.   From that variable you want to pull the VM name (vCACVm.displayName), and use that to know where to run the SSH command. Simple and effective.




Why shell scripts on each node and not full vCO automation?
Normally I wouldn’t bat an eye and do all of this from vCO itself. Create the JumpCloud endpoint as a REST HOST, create various actions as REST Operations, etc.  The script just reaches out to a REST API after all.   The first reason was, well, they already had demo scripts available that work completely.   But really it is because authentication is done from each individual node, and each node (appears to) have a unique system key and this system key is used in the authentication to the REST API.   This may need to be revisited for true enterprise functionality for central provisioning, IPAM or even synchronizing directory services.   But I digress….

Implementing custom machine stages
Back to the vCO bit.  We need to run the below library workflow “Assign a state change workflow to a blueprint and its virtual machines” where we specify the Blueprint we are using and the vCO workflow we want to run, and at which machine state we want it run.


And bam, a custom property gets automatically added to the Blueprint itself.   Since we want this to happen when the machine is destroyed we chose the Machine Disposing stage.   The long ID starting with 8be39.. is the vCO workflow ID.   You may have encountered this if you ever needed to invoke workflows from the vCO REST API like here.  This above library workflow is a lot more useful for a more complicated integration with lots of values being passed but hey it saved a little time for us here.



Try it out
Now unless I missed documenting a step here (as I’m writing this after I built the integration), all we have to do is destroy the machine like we would normally and it quickly disappears from the JumpCloud console.


 And there we go.  Full integration for login access control to my lab environment machines provisioned from vCAC.  If I’m honest I may even continue using this for a few machines as handy as it is.


Tagged , , , ,

Docker as a Service via vCAC: Part 1

dockerThis project started with the question – using VMware management software today, I wonder what would it be like to manage and provide self service access to Docker containers right alongside the traditional virtual machines and other services.  So that is what I explored and that is what I built here so far.   This is just going to be part one of…many probably…  as I develop the ideas further.
What does this solution aim to do?
This solution elevates a docker based container to somewhat of a “first class” citizen, in that it sits alongside virtual machines in the self service vCAC catalog.

Really? You are able to do this today?
Well…  Mostly.   More work needs to be done to make it more functional.  But as of this part 1, provisioning is working great, and monitoring too (thanks to work from a co-worker that will be written about later).  Anything further like day 2 operations, or just tear down (of the containers) is manual currently.  But still possible.

So walk me through it?
There’s a single machine blueprint that deploys a CentOS machine and installs docker.  I needed a way to identity these machines as a special entity so I went with vSphere tags for now.   So using the vCAC Extensibility functionality I also have it fire off a vCO workflow that calls PowerShell (talked about here) to add the vSphere tag.  Not elegant but it works.  This will be improved later.      So now that a machine exists, there is additional catalog items  for various services like the demo application SpringTrader, or simply MySQL, Postgres, Ubuntu, etc, that run a vCO workflow to deploy the image onto one of the existing docker nodes.   Currently it picks a node randomly, but with some additional effort I plan to implement a (very) poor mans DRS and utilize either a hyperic plugin that my team is working on, or maybe just query cpu/memory directly to choose the node.

OK tldr;  Boil it down!?
First docker nodes are deployed from vCAC blueprints.  Then vCO workflows can identity those nodes via vSphere tags and deploy the requested docker image.


Single machine blueprint in vCAC



“DockerVM” tag in vSphere denotes the docker capability machines



Service Blueprint for various docker images



You can specify items like the image name, the port to expose, the bash command, etc..



Slight smoke and mirrors in the screenshot – this is using a local docker registry so don’t be confused by that.


…and to prove she’s working

What’s next?
I (and others) am (are) working on ways to tear down the containers, tear down the nodes in mass, automate discovery and monitoring of the containers, and so on..   Currently there’s not even a way to know where the image was deployed to – to come!

Can we have the workflows, scripts, etc…?
Not yet…!  But if it is fun for anyone, I do have the demo app springtrader available on Docker Hub if you want it.  It weighs in at a hefty 4gb.  Find it at jasper9/st_allinone.    There is no special sauce included in here, it’s simply the SpringTrader app documented here  built into an image all ready to go  (very poorly built, I’m sure….).

Sweet! How do I run SpringTrader?
This should probably be a full post on it’s own.  But in short this command will get it running.  Change the first 8080 to any open port if it’s already in use.

docker run -t -i -p 8080:8080 jasper9/st_allinone /bin/bash -c ‘/startup.sh && /bin/bash’

Then see the web interface at:  http://ip.address:8080/spring-nanotrader-web/#login



Tagged , , , , ,

Small Things: vCAC 6.1 – "Data Collection" catalog entry

For anyone that does template configuration changes in vCAC can attest to how big of a pain it is to reconfigure the agents, shut down the machine, snapshot, browse the menu structure to where you force data collection, click collect data for all the items, browse to the blueprint config and wait for it to complete.    Well, hopefully this tip can speed that up just a little, or at least make it less of a headache for you.

vCAC 6.1 comes with a ton of vCO workflows out of the box.  One that caught my eye is “Force data collection”.



Adding this workflow as a catalog item is a breeze under Advanced Services – Service Blueprints.  When complete it will show up like any other service or template:



And does its job quite well:



One warning, you will want to set a constant value for the one question it will prompt for in this workflow.   Edit the blueprint as such:



And choose your IaaS (windows) server:



I quickly installed a fresh new instance of vCAC & IaaS today, and not sure if it was an error during install or not but at first mine didn’t show any hosts here in the above screen shot.   I had to go into vCO with the client and run this workflow to add it.  Your results may vary.



EDIT Sept 22 2014:   I wasn’t clear about where to find this workflow.  It’s found within these folders:

vCloud Automation Center
Infrastructure Administration
> Force Data Collection

Tagged , , ,

JAAS Project Lake Placid – Dinner as a Service – Overview

This post is part of a project I’m calling ‘Dinner As A Service.’
Electricity Costs

This project has been a while in the making as it’s the most elaborate I’ve attempted to date. To fully document this one it is going to span multiple posts so be patient and check back over time if you are interested in these weird projects of mine.

To summarize, this project starts with VMware vCAC 6.0 where one orders ‘dinner’ and ends with the food being cooked in a Sous Vide water bath. Over engineered, over kill, and quite eccentric but hey it’s fun. And I learned a lot along the way in ever area.

VMware vCloud Automation Center
VMware vCenter Orchestrator

Raspberry Pi
Solid State Relay
Pi Cobbler
2 x 7 Segment LCD
Temperature sensor
LED, Case, other bits..
Slow Cooker
(we’ll expand on the parts and builds later…)

First lets start with the basics – What Is Sous Vide? According to Wikipedia:

In other words – food awesomeness using self service, automation & electronics.   Here’s a site I got most of my information from.  They make a pricey, but off the shelf cooker.

What is a good application automation? Electronics and vCO! In truth the software part really just kicks it off (though I did add in some functionality that makes picking your temperature easier). The special sauce is in the electronics and the script on the Raspberry Pi, but we’ll get to that.

End to end (or menu to plate?), here is how it all works:

0. In the beginning, there was meat.   The wife and I joke that we know a guy and buy meat out of the back of a truck.  It’s actually not far from the truth however.   Check these guys out if you are in Colorado.  They deliver to locations all over the front range.

1. The project starts with the menu. We’re using vCloud Automation Center 6 (vCAC) here as the front end. The main reason was so that I could learn more about the internals of it, but also because this sort of thing is right up it’s ally. Well, not cooking food – but a frontend for the automation of services.


2. Let’s say I want to cook up a nice steak tonight. I choose beef and I am given the choice of how I want the meat cooked. This is big advantage to using this front end – I don’t know off the top of my head what each temperature range is so associating it this way with a label is great. I’m going to pick Medium here (140d F) since my other/better half doesn’t like it any redder.


3. Yay!

4. Some magic happens in the background (via vCenter Orchestrator) and I get a notification to my phone that the temperature has been set.


5. Because I’m a super nerd, I also get it to my watch. In truth, it’s actually really handy.


6. On the Raspberry Pi, there is a python daemon running called Raspibrew (some guy built this for homebrewing beer! Awesome!). I used this because it has PID intelligence built in (so I didn’t have try to learn to code one…) and it has a REST api (which before I realized was included had already written one in node.js…ahwell….)


7. The rPI is housed tentatively-permanently in a project case along with the rest of the electronics. Most important is the temperature sensor, and the solid state relay. When we hit our target temperature, the relay shuts off power to the outlet the slow cooker is plugged into.


8. We wait for it to heat up. Usually when I’ve been cooking I wrap the slow cooker in this heat shielding since it appeared I have a pretty crappy cooker that looses heat fast and has a weak heating element. Getting it high enough for vegatables was almost impossible without this.


9. Also as an aside, during this whole heating and cooking process I have a separate python script I made that writes the temperature readings out to Xively (previously Cosm) for historical tracking. This has been useful for when I was studying the temperature swings to see how much I needed to tune the PID values.


10. When we get to our target temperature another alert is sent out letting me know. This has been really useful so I can go mill around doing other things and not watch it. (either preface with why it says 50, or do this screenshot again for real)



11. Drop the meat in!


12. Now this is the part I have not done any further automation on. The rPI’s PID algorithm is still in charge of tweaking power to the slow cooker when it thinks it needs it, but I have not set up any timers yet. I wanted to add another 7-segment display with a countdown, and prompt for this in vCAC but the box got crowded with other bits and I had trouble figuring out how to reliably do a countdown timer on a rPI without a real time clock. Feature request!

13. When our cooking time elapses, we continue on the cooking process. Sous Vide is an awesome way to get super tender meat, but isn’t great on the eye candy front. Browning in a hot pan or using a torch for a bit works wonders.

Pull it out of the bath.  It’s really ugly when cooked this way, but oh so tasty.


Brown the meat.  (During this cook I really should have gotten a much better crust going.  I was just excited to be using this steak sauce for the first time in a long long time….!)


14. Finish up whatever sauces and sides are in order for the day and boom…. Dinner!



I’ll follow this post up more info on the hardware, software, and other bits.   Stay tuned..

Tagged , , , , ,