Tag Archives: git

Experiment: Pooling in vRA & Code Stream

Background

I recently attended DevOpsDays Rockies which is a community oriented DevOps conference (check them out in your area, it was great!).  I saw a talk by @aspen (from Twitter/Gnip) entitled “Bare Metal Deployments with Chef”.   He described something he/they built that, if I recall correctly, uses a PXE/Chef/MagicpixieDust to pull from a pool of standby bare metal hardware to fully automate bringing it into a production cluster for Cassandra (or what have you).

This got me thinking on something I was struggling with lately.  Whenever I develop blueprints in Application Director / Application Services, or just vRA/Code Stream, the bulk of the time I just hit the go button and wait.  Look at the error message, tweak and repeat.  The bottleneck by far is in waiting for the VM to provision.  Partly this is due to the architecture of the products, but also it has to do with the slow nested development environments I have to use.  We can do better…..!

Products using pooling

I then started thinking about what VDM / Horizon View have always done for this concept.  If I recall correctly, as it’s been years and years since I’ve worked with it, to speed up deployments of a desktop to a user, a pool concept exists so that there will always be one available on demand to be used.   I don’t have much visibility into it but I am also told the VMware Hands On Labs does the same – keeps a certain number of labs ready to be used so the user does not have to wait for it to spin up.  Interesting.

The idea

So I thought – how could I bring this upfront deployment time to the products I’m working with today to dramatically speed up development time?   And this is what I built – a pooling concept for vRA & Code Stream managed by vRO workflows.

Details – How Redis Works

When planning this out I realized I needed a way to store a small bit of persistent data.   I wanted to use something new (to me) so I looked at a few NoSQL solutions since I’ve wanted to learn one.  I decided on Redis as a key value store, and found Webdis which provides a light REST api into Redis.

I couldn’t find any existing vCO plugins for Redis I/O which is fine, the calls are super simple:

Example of assigning a value of a string variable:

Snip20150517_5The redis command is: “set stringName stringValue”
So the webdis URL to “put” at is “http://fqdn/SET/stringName stringValue”

Then to read the variable back:

Snip20150517_6The redis command is: “get stringName stringValue”
So the webdis URL to “get” at is “http://fqdn/GET/stringName”

Easy peasy. There is similar functional for lists, with commands to pop a value off either end of the list.  This is all I needed, a few simple variables (for things like the pool size) and a list (for things like the list of VMs storing IP addresses & names).

So in vCO I just created a bunch of REST operations that used various number of parameters in the URL line:

Snip20150517_7
I found the most efficient way to run these operations was to parametrize the operation name, and pass it to a single workflow to do the I/O

Details – Workflow(s)

The bulk of the work for this pooling concept is done in the following workflow that runs every 15 minutes.

Snip20150517_8In general it works like this:

  • Check if the workloads are locked – since it can take time to deploy the VMs, only one deployment will be going at a time.
    • If locked, end.
    • If not locked, continue.
  • Lock the deploys.
  • Get the pool max target (I generally set this to 10 or 20 for testing).
  • Get the current pool size (the length of the list in Redis.  much faster than asking vSphere/vRA).
  • If the current size is not at the target, deploy until it is reached.
  • Unlock the deploys.
  • Profit.

I did not have to do it this way, but the nested workflow that does the actual VM deployments is requesting vRA catalog items.

In Action

After I got it fully working and the pool populated, you can check the list values with this type of Redis query:

Snip20150517_9

Redis: lrange vmlist 0 -1 (-1 means all)
Webdis: http://fqdn/LRANGE/vmlist/0/-1

The matching machines in vSphere:

Snip20150517_11

In Action – Code Stream

Normally in a simple Code Stream pipeline you would deploy a VM by requesting the specific blueprint via vRA like this:

Snip20150517_19

In this solution, instead I use a custom action to grab the VM from the pool and return the IP back to the pipeline as a variable.  Then I treat the VM like it’s an existing machine and continue on and at the end delete the machine.

Snip20150517_18

This reduces the list in redis by one, so the next time the scheduled workflow runs that checks the list size it will deploy a new one.

(Kind of) Continuous Deployment

I have a job in Jenkins that builds the sample application I am using from source in Git, pushes the compiled code to Artifactory and does a post build action that calls Code Stream to deploy.

Snip20150517_15

I wanted to see if there were any bugs in my code, so I wanted this whole thing to run end to end over and over and over…   I configured the Jenkins job to build every 30 minutes.  I went on vacation the week after I built this solution so I wanted to see if over time anything broke down.  Amazingly enough it kept on trucking while I was gone, and even got up to the mid 700’s in Jenkins builds.   Neat!

Snip20150517_12

Jenkins builds

Artifacts

Artifacts

Code Stream executions

Code Stream executions

Summary

To my surprise, this actually works pretty darn well.  I figured my implementation would be so-so but the idea would get across.  It turns out, what I’ve built here is darn handy and I’ll probably be using it the next time I am in a development cycle.

Post any questions here and I’ll try to answer them.   I’m not planning to post my workflows publicly just yet, fyi.

Tagged , , , , , , , , , , , ,

Using vFLOWer and GitHub to bring (better) version control to vCO

Update March 15, 2014:  This type of functionality is now included in the product in vCO 5.5.1  See more here

vCO has very basic versioning built into the workflows.  It works for simple mistakes from version to version, but isn’t great for cross environment and multiple developer use cases.  Enter the vFLOWer Toolkit made by ByteLife.   They are a VMware Partner that built, what I think is, a BCDR solution using a lot of Orchestrator workflows.  When building their solution they found they really needed tighter control over versioning their work for their developers, so they built a solution and made it available.  Now I know it’s a little bit of a hack, but it works great.   Let’s look at how to use it here.  This post will be screenshot heavy, but is worth it to illustrate what is going on and how you do it.

I wanted to post all my workflows for the Dinner as a Service project, so let’s walk through that here as an excercise.  First you need to export your items from vCO. For this project, I used the instance of vCO embedded with the vCAC appliance.  The URL for the client is https://fqdn_of_vcac_appliance:8281/vco/client/client.jnlp

vCO Login

vCO Login

Once we’ve logged in, switch to the administrator view. Click on the Package tab.  If you aren’t already keeping your workflows in a package, go ahead and create one.  Right click, Add Package.

Create package

Create package

Give it a name

Give it a name

Now we need to add items to the package, so right click “Edit”

Edit the package

Edit the package

Switch to the Workflows tab, click Insert Workflows and add the workflows you want to put into this package.

Add the good stuff

Add the good stuff

Screen Shot 2014-03-02 at 9.37.47 AM

You’ll notice it captures all associated items with these workflows you specify.  In my case for this project, it added an Action and Plugin for REST.

REST Operation action

REST Operation action

REST Plugin

REST Plugin

Awesome, so we have the items we’re concerned about in a single package.   Right click and export package.

Export Package

Export Package

Options for export. Note the "export the values of configuration settings" that I will call out later.

Options for export. Note the “export the values of configuration settings” that I will call out later.

Now we have the package exported to the file system.

Now we have the package exported to the file system.

Now that we have our package exported, we need to convert it from the native binary format (I think?) to something that GitHub can use to do plain text revision control on.   Follow the User Guide on the vFLOWer website here or continue reading for a more detailed step by step process.

 We need to satisfy the requirements for vFLOWer which are – Apache ANT, Java, OpenSSL and Git.

Starting with ANT,  download the package here.  There is no installer for this one so we’re just going to extract it to our source dir at C:srcapache-ant-1.9.3  (use whatever path you want, but I’m going to use this src directory throughout).

ANT directory with the vCO package.

ANT directory with the vCO package.

We need to edit our environment variables to include ANT.     If you are unfamiliar with this process these steps will help as it’s not exactly an everyday activity.   Right click on Computer, select Properties.  (or Control Panel, ‘System and Security’, ‘System’).

Computer - Properties

Computer – Properties

Select ‘Advanced System Properties’.

Advanced System Settings

Advanced System Settings

Select ‘Environment Variables’.

Environment Variables

Environment Variables

Click ‘New’.   Add ‘ANT_HOME’ with our path to ‘C:srcapache-ant-1.9.3’.

ANT_HOME

ANT_HOME

This makes ANT_HOME available to the system but it’s not in the system path yet.    Scroll down to Path, click ‘Edit’, and add ‘;%ANT_HOME%bin’ to the end.

Path for ANT

Path for ANT

Test this by opening a new command prompt and type ‘ant -version’

Test ANT

Test ANT

That’s one down.   Now we need a Java JDK.  The newest at the time of this writing is jdk-7u51-windows-x64.exe and can be found here.

You want the JDK on the left.

You want the JDK on the left.

Choose your flavor.

Choose your flavor.

There is a windows installer for this one so it’s straight forward.    When complete, similar to what we did for the ANT environment variables, add JAVA_HOME, set it to C:Program FilesJavajdk1.7.0_51

JAVA_HOME

JAVA_HOME

Test with a new command prompt ‘echo %JAVA_HOME%’    This does not need to be added to the system path.  (EDIT: when I did this on Win7 it added automatically, on 2008 it didn’t.  That’s annoying.)

Testing JAVA_HOME

Testing JAVA_HOME

One more down, a few to go.   You may need to install the 2008 C++ Redistributable’s.   I have never known what’s in here but they say you should use it.  Kind of like vitamins.   Find it here.

For OpenSSL, grab the pre-compiled windows version here and install.  You may get a warning about the 2008 C++ package, this probably goes away if we would have rebooted – yay windows.  Choose the defaults.  Add to the system Path variable ‘;C:OpenSSL-Win32bin’ .

Update Path

Update Path

Test with the command ‘openssl version’ in a new command prompt.

Testing OpenSSL

Testing OpenSSL

Now we’re in the home stretch.  We just need the windows git package from here.

Download GIT because you can't get GIT without getting GIT first. GIT it?

Download GIT because you can’t get GIT without getting GIT first. GIT it?

Choose most of the defaults except two.  On the PATH screen, simplify things and have the installer modify the variable for you by choosing ‘Run GIT from the Windows Command Prompt’

Adjusting PATH automatically

Adjusting PATH automatically

On the line ending option page choose: Checkout as-is, commit unix-style line endings.

Line Ending Conversions

Line Ending Conversions

And test git real quick with ‘git –version’

Testing GIT

Testing GIT

Alrighty.  Now for the meat of it.   Everything up until now was just prep.  From here on out the vFLOWer doc is pretty descriptive.

You need to log into GitHub (create an account if you need to), and create a few ‘Fork’ of the vFLOWer repo.  If you are new to git, what this does is creates a new copy of this repo that is now owned by you but still chained to the parent.   If this was a real dev effort, we could push changes up to the parent if/when needed.  This is how developers work on multiple ‘branches’ of a code base without stomping on each other in one single repo location.    So login, browse to vFLOWer here, click fork.

Fork the original vFLOWer repo.

Fork the original vFLOWer repo.

Now you have your own fork of vFLOWer.  This will be where you store all your vCO stuff.   Let’s make it your own by renaming and edit the README.md   Click ‘Settings’ to rename.

Screen Shot 2014-03-02 at 2.54.24 PM

We’ll want to name it something meaningful.  For me, that always includes “stuff”:

my-vco-stuff

my-vco-stuff

When done, click ‘README.md’, click ‘edit’,  add your own content.  The read me file is similar to a index.html – we now see our new content at the root of the repo.

Click Edit

Click Edit

Enter your content

Comments Rock.

Comments Rock.

Add a comment before saving (we’ll talk more about comments later):

Screen Shot 2014-03-02 at 12.51.38 PM

And refresh the main page to see it:

Screen Shot 2014-03-02 at 12.51.51 PMNow comes the real magic of Git.   On the right hand of the main screen, copy the URL to the clipboard.   Mine is https://github.com/jasper9/my-vco-stuff.git

Grab the URL

Grab the URL

Back on the original windows machine we were on, change dir to our C:src directory and pull your Git repo.   This downloads the full thing to a subdir of the same name.

git clone

git clone

Check it out in windows explorer and it’ll look identical to the web interface.

We have files!

We have files!

Back to the vCO package we originally exported.  Copy this package into the /inout sub directory (C:srcmy-vco-stuffinout):

Package file in inout

Package file in inout

This is where ANT comes in. From the root of the repo (not the inout dir) run ‘ant pre commit’     You should see a lot of text scroll and end with a success message.

Text...text...DONE!

Text…text…DONE!

Check the inout directory and you should see the package is removed:

InOut Dir now

InOut Dir now

Check the content directory and you should see items from vCO that is familiar to you.  In my case I am looking at the Workflows dir.  My workflows were in a folder in vCO and that structure is maintained here at C:srcmy-vco-stuffcontentWorkflowsx My Library

Workflow Content

Workflow Content

Open up one of the workflows and you will see what it looks like in clear text.   One item of note is you might see values that were entered into parameters in vCO.   In my case I have ones for the Pushover service that is sensitive (API keys).  I want to remove these with placeholder values.  If you wish, you can uncheck a box in the vCO export process to not include any parameter data – if you have a lot this could be handy, but could also limit you in other ways by requiring some manual work to re-setup.

XML Content

XML Content

Now back in the root directory of this repo, type ‘git status’ to see the current status of this repo we’re in.  You’ll see a message complaining that there are new directories we weren’t expecting.

'git status' output

‘git status’ output

Add these automatically with the ‘git add .’ command.   Alternatively you could add items one by one if you ever need to in the future.

Now run status again and you’ll see it’s a bit happier, reporting cleanly that we have some new files in this snapshot of your code base.
Happier git status

Happier git status

Remember when you edited the README.md file and we added a message describing what we did?  That’s a comment.  Like you learned in “BASIC programming 101″ comments are a wonderful thing.  Maybe that was just me.  (Oh yeah, I can’t talk about git commit comments without dropping this XKCD on you)

MY HANDS ARE TYPING WORDS

MY HANDS ARE TYPING WORDS

So that change to the README.md we could have done here if we wanted to by manually editing the file.  To comment the changes we just made adding the vCO content type:

git comment attempt

git comment attempt

Two problems arise here (only the first is currently shown there)

Problem 1 I sometimes encounter – cut/paste from PDF’s or other sources don’t always work.  I blogged a little about it here.   In this case, it’s the PDF’s fault and the OSX setting didn’t fix it.  Replace the hyphen looking character with a real hyphen and run it again.
Problem 2 Now your commit was probably successful but it complains you haven’t set your account up right.  Let’s do that next.
Successful, kinda

Successful, kinda

Add your name and email with:

Then reset the author in the comment with:

My intention in this post isn’t to give a full primer on git (here’s one that looks good), so for the last part I’ll leave it simply at a high level explanation. We need to push our changes that we just committed to the server. We do so with the command:

Git push

Git push

Now if we refresh our github webpage, you should see the updates

Ch-cha-cha-cha-changes

Ch-cha-cha-cha-changes

Browse through the content and verify your workflows are there:

New stuff!

New stuff!

Awesome – now that’s how you export your packages, use this tool to convert them, and use Git to post them to GitHub.  Now to show how you would pull it down again, I’ll flip over to a fresh environment (but with all the prereq’s met).  This would be like if you were someone else and stumbled upon these workflows – how do I use someone else’s work.   Because this is a public repo, this is unauthenticated.

git clone

git clone

If you look into what was downloaded, it is as you would expect.  There is a content directory with our vCO stuff in XML format and no inout directory.

Fresh bits

Fresh bits

Now we need to use ANT to convert it back into a format we can import into vCO.

Yay!

Yay!

And check into the inout directory, and behold our package!

vCO Package

vCO Package

Load up the vCO client now.  Switch to the Administrator view.  The packages tab.  click the import button

Load it up!

Load it up!

You will see an SSL error you can ignore because the vFLOWer tool created a self signed cert for the process.   You may see a conflict message like this one shown if anything exists already or is at a newer version:

Warning(s)

Warning(s)

And there you go.  Now you can run your workflow in a different environment.  Or in this case, make dinner in someone else’s datacenter?

Ready to go!

Ready to go!

Tagged , , , , , ,