Monthly Archives: May 2015

Using a time series DB, starting with InfluxDB

devops-everywhereAt the last two conferences I attended (DevOpsDays Rockies and GlueCon) I heard a lot of mentions of NoSQL and time series databases.   I hate not knowing about things and not having experience with it so I’ve been playing with both of these.    First I integrated a NoSQL db using Redis into a project of mine recently.  And just now I’ve been playing with InfluxDB as a monitoring system and here I’m going to tell you about my experience.

I didn’t want to get caught up in any installation shenanigans so I tracked down docker images to assist in getting up and running fast.  And glad I did because it worked immediately!

index

1: InfluxDB

InfluxDB docker image: https://registry.hub.docker.com/u/tutum/influxdb/

Docker command:

And then right away you can load in a webbrowser: http://your.ip.address:8083 and you will get the below screen:

Snip20150527_9Once you log in with root/root, you will be shown that you have no databases out of the box, go ahead and insert a name and hit create:

Snip20150527_10You are now given a simple UI to be able to push and pull data into the system by hand.  To test this we will add some values that are in the same format as some of my scripts that deal with temperature.  Basically you can think of the time series as the table, and the values as key/value pairs.

Snip20150527_11

Then you can craft a simple query to verify the data:

Snip20150527_12

Neat!   Now unlike some other solutions, InfluxDB doesn’t provide any visualization functionality (other than a basic line).  I spun up a Grafana container to do this.

grafana

2: Grafana:

Grafana docker image: https://registry.hub.docker.com/u/tutum/grafana/

Docker command:

There are simpler ways to start up this container but I found all of these parameters got me quickly to where I wanted.

Now you can login to port 80 on this machine and you will be presented with an empty first page:

Snip20150527_13

Empty graphs aren’t very exciting, so let’s configure them real quick…

The syntax in Grafana is slightly different than we saw directly in InfluxDB but is mostly straight forward.  We put the database name (temperature) into the “series” field.  We fill in the blanks for the select statement – use last(value) and item=’80027_temp’ to specify the key/value.  Click in somewhere else to change focus and the graph should reload showing the values we entered by hand.

Snip20150527_14

Now I wanted to play around with it further so I modified some existing scripts I have for doing various types of monitoring like pulling data from weather underground (temperature, humidity, wind), and some data for free disk space from a NAS.  Mix it up and it came out looking like this:

Snip20150527_15

To feed the data in, I took the easy way out and used a perl client documented here.  So I then just modified my existing scripts to feed the data to this perl script every 30 seconds and bam, I’m done.

 

 

Tagged , , ,

New backup option for Synology devices

synology1512I have two Synology NAS devices in my home lab that I’ve always struggled with being sure I have full backups of as they have grown pretty large over time.   I have the 5 bay DS1512+ (with an additional 5 bay expansion), and I have a tiny 2 bay DS214SE.    I didn’t plan my use of them this why but it just kind of evolved over time, such is how my lab is.  Something breaks or is slow, I tweak to squeeze out better performance on a small budget and life goes on.

Cur615976_0_frently I use the large array for normal file storage (music, photos, videos, ISOs, etc…), I used the extra expansion for transition storage over time when I moved stuff around (mainly VMs).   I originally used the tiny 2bay NAS for my tiny portable lab based on NUCs.  I now have the management components living on a iSCSI lun on there (VC, PSC, vCO, DNS, AD…) and for all compute I am now using VSAN across three white box machines (which is working fantastic!).

I’ve always struggled with backups in my lab.  Any free options out there either won’t cover two VCs, cpu core limited or VM count limited.  I’ve been using William Lam’s GhettoVCB forever, which is solid but mostly manual.

Enter….what I found this past weekend.  Synology management software images that will work in a VM or baremetal!   This is literally the same OS that runs on their devices.  I first tried it in a VM to test it out.  All seemed well except for updating as it appears to break so you have to wait for unoffical patches.   To use this for real, I went ahead and swapped out the USB drive on my HP N40L which was previously running FreeNAS for backups.

This allowed me to setup a reoccurring RSync from the DS1512:

Snip20150527_4…And also allowed me to setup a reoccurring backup of the iSCSI LUN (holding the management VMs):

Snip20150527_5

While the ~250gb iSCSI backup was pretty quick, the Rsync of 6 TB of small & large files is taking a while.   Performance seems pretty decent, at least for my home lab that can be kind of…iffy given the amount of crazy crap I run on the large flat network of consumer level 1GB switches no tuning whatsoever.

Snip20150527_6

Prior to this I was doing all my backups manually – both the rsyncs and ghettovcb backups.   I would then a few times a year move a backup set outside of my house to a family member.  I highly suggest you do the same!  I do my best to follow the 3-2-1 rule, though I’m not doing great on the multiple types of media as my photo collection has grown too large to use “cloud” storage useful or economical.

Check it out for yourselves!

Install information (credit as the source!) http://www.bussink.ch/?p=1672

More information http://www.xpenology.nl/

Downloads http://xpenology.me/downloads/

Tagged , , , ,

Experiment: Pooling in vRA & Code Stream

Background

I recently attended DevOpsDays Rockies which is a community oriented DevOps conference (check them out in your area, it was great!).  I saw a talk by @aspen (from Twitter/Gnip) entitled “Bare Metal Deployments with Chef”.   He described something he/they built that, if I recall correctly, uses a PXE/Chef/MagicpixieDust to pull from a pool of standby bare metal hardware to fully automate bringing it into a production cluster for Cassandra (or what have you).

This got me thinking on something I was struggling with lately.  Whenever I develop blueprints in Application Director / Application Services, or just vRA/Code Stream, the bulk of the time I just hit the go button and wait.  Look at the error message, tweak and repeat.  The bottleneck by far is in waiting for the VM to provision.  Partly this is due to the architecture of the products, but also it has to do with the slow nested development environments I have to use.  We can do better…..!

Products using pooling

I then started thinking about what VDM / Horizon View have always done for this concept.  If I recall correctly, as it’s been years and years since I’ve worked with it, to speed up deployments of a desktop to a user, a pool concept exists so that there will always be one available on demand to be used.   I don’t have much visibility into it but I am also told the VMware Hands On Labs does the same – keeps a certain number of labs ready to be used so the user does not have to wait for it to spin up.  Interesting.

The idea

So I thought – how could I bring this upfront deployment time to the products I’m working with today to dramatically speed up development time?   And this is what I built – a pooling concept for vRA & Code Stream managed by vRO workflows.

Details – How Redis Works

When planning this out I realized I needed a way to store a small bit of persistent data.   I wanted to use something new (to me) so I looked at a few NoSQL solutions since I’ve wanted to learn one.  I decided on Redis as a key value store, and found Webdis which provides a light REST api into Redis.

I couldn’t find any existing vCO plugins for Redis I/O which is fine, the calls are super simple:

Example of assigning a value of a string variable:

Snip20150517_5The redis command is: “set stringName stringValue”
So the webdis URL to “put” at is “http://fqdn/SET/stringName stringValue”

Then to read the variable back:

Snip20150517_6The redis command is: “get stringName stringValue”
So the webdis URL to “get” at is “http://fqdn/GET/stringName”

Easy peasy. There is similar functional for lists, with commands to pop a value off either end of the list.  This is all I needed, a few simple variables (for things like the pool size) and a list (for things like the list of VMs storing IP addresses & names).

So in vCO I just created a bunch of REST operations that used various number of parameters in the URL line:

Snip20150517_7
I found the most efficient way to run these operations was to parametrize the operation name, and pass it to a single workflow to do the I/O

Details – Workflow(s)

The bulk of the work for this pooling concept is done in the following workflow that runs every 15 minutes.

Snip20150517_8In general it works like this:

  • Check if the workloads are locked – since it can take time to deploy the VMs, only one deployment will be going at a time.
    • If locked, end.
    • If not locked, continue.
  • Lock the deploys.
  • Get the pool max target (I generally set this to 10 or 20 for testing).
  • Get the current pool size (the length of the list in Redis.  much faster than asking vSphere/vRA).
  • If the current size is not at the target, deploy until it is reached.
  • Unlock the deploys.
  • Profit.

I did not have to do it this way, but the nested workflow that does the actual VM deployments is requesting vRA catalog items.

In Action

After I got it fully working and the pool populated, you can check the list values with this type of Redis query:

Snip20150517_9

Redis: lrange vmlist 0 -1 (-1 means all)
Webdis: http://fqdn/LRANGE/vmlist/0/-1

The matching machines in vSphere:

Snip20150517_11

In Action – Code Stream

Normally in a simple Code Stream pipeline you would deploy a VM by requesting the specific blueprint via vRA like this:

Snip20150517_19

In this solution, instead I use a custom action to grab the VM from the pool and return the IP back to the pipeline as a variable.  Then I treat the VM like it’s an existing machine and continue on and at the end delete the machine.

Snip20150517_18

This reduces the list in redis by one, so the next time the scheduled workflow runs that checks the list size it will deploy a new one.

(Kind of) Continuous Deployment

I have a job in Jenkins that builds the sample application I am using from source in Git, pushes the compiled code to Artifactory and does a post build action that calls Code Stream to deploy.

Snip20150517_15

I wanted to see if there were any bugs in my code, so I wanted this whole thing to run end to end over and over and over…   I configured the Jenkins job to build every 30 minutes.  I went on vacation the week after I built this solution so I wanted to see if over time anything broke down.  Amazingly enough it kept on trucking while I was gone, and even got up to the mid 700’s in Jenkins builds.   Neat!

Snip20150517_12

Jenkins builds

Artifacts

Artifacts

Code Stream executions

Code Stream executions

Summary

To my surprise, this actually works pretty darn well.  I figured my implementation would be so-so but the idea would get across.  It turns out, what I’ve built here is darn handy and I’ll probably be using it the next time I am in a development cycle.

Post any questions here and I’ll try to answer them.   I’m not planning to post my workflows publicly just yet, fyi.

Tagged , , , , , , , , , , , ,