Category Archives: Random Scribblings

Random Thoughts on DevOps

DevOps is one of those trends that is many things to many people.  I’ve been meaning to write this for a while now and Duncan’s recent post reminded me of that.

First off – I am many things but currently an authority on everything DevOps in the big scheme of things I am not.  I’m just a guy that’s done a few things and knows a few other things.  However, I wanted to share a few nuggets I’ve come across in the last year on this topic.  Some a bit dated now but that’s ok.  It’s all good stuff.

(side note…  the amazon links are just for ease of use.  they amazingly don’t allow affiliate link programs here in colorado…)

BOOKS:

Snip20141103_13The Phoenix Project 
By Gene Kim, Kevin Behr, and George Spafford
Published: Jan 2013

Recently Duncan Epping posted his review of this book here.  I won’t rehash his summary but just say in my mind it is an excellent exposure to the subject no matter what your role or experience is.  The start of it gave me chills of being back in a chaotic operations environment – a place I hope to never find myself again (unless it’s in a role to clean up the mess like the protagonist). And throughout the book I found it interesting the author was able to really depict the evolution of the topic without name dropping specific vendors or technologies.  That’s impressive in a space dominated by hipster devs and brogrammers.   (wow I feel like I’m an old man yelling at the kids to get off my lawn…)

In short – you must read it if your job is ops, dev, anywhere in between, or work in IT in any way.

The Goal: A Process of Ongoing ImprovementSnip20141103_12
By Eliyahu M. Goldratt and Jeff Cox
Published: 1984

If you geek out on the overall ideas of Phoenix Project, and not just the technology aspect of it, reading The Goal might be for you.  If you recall in Phoenix, there are a few scenes where the characters visit a manufacturing plant to illustrate a point on bottlenecks and handoff of work. If I recall correctly this book is briefly referenced in one of those scenes.   The Goal is very very very similar to Phoenix in so many storyline ways that is obvious the authors in 2013 of Phoenix used it as a blueprint for telling their own story.  That took a small bit of the magic for me from Phoenix after reading this older book, but that’s ok – it’s still excellent in it’s own right.   The Goal is to manufacturing what Phoenix is to IT.  Call it BizPlantOps maybe?   It explores the breaking down of preconceived notions of policies and procedures that are followed in manufactoring because they are just accepted practice, and how to improve on them to compete in the market.   You can hear the methaphors for DevOps already right?

In short – if you are strictly into the technology aspect of DevOps this one will bore you.   But if you geek out on the theory of DevOps, the drivers, the principles, and the roots of it, this could be a good read for you.
Other Reading:

Snip20141103_14DZone’s 2014 Guide to Continuous Delivery
(Free PDF download with registration)

I came across this 35 page PDF recently when someone posted to a company collaboration page “if you read nothing else about DevOps, this should be it”.   I whole heartedly agree.   They do an amazing job breaking down the buzz words Continuous Integration, Continuous Delivery and Continuous Deployment into the core of what they actually are.  Sure there are paid ads and vendor highlights sprinkled throughout but it in no way feels like they are spouting sales jargon, but true research findings.  It feels more like *this is what all the details are*  and *this is the current vendor offerings are to do it* and doesn’t make an endorsement either way or fall into the zealot mindframe of the one single tool is the best always and forever no matter what.

In short – it’s 35 pages and free.  Read it.


Presentations:

I’ll just drop a few links here on videos I’ve enjoyed.  These are all from many moons ago now as I found them while doing research for a now older project.

Snip20141103_17How Do We Better Sell DevOps? (PuppetConf 2013)
From Gene Kim (author of Phoneix Project)

Keynote: Stop Hiring Devops Experts (And Start Growing Them) (PuppetConf 2013)
From Jez Humble

PCI-DSS and continuous deployment at Etsy

 

There we go, that clears out my current list of OH YOU SHOULD READ/WATCH THIS list.    Drop your current favorites in the comments!

 

Tagged , , ,

How to use a local persistent Docker registry on CentOS 6.5

UPDATE: Dec 16 2014, I found a new option is needed now using Docker version 1.3.2   See more here

There a bunch of blogs out there showing a tutorial on how to use a local docker registry but none of them (that I have found) have it boiled down to the absolute simplest syntax and terms.   So here you go!

First off terminology Docker Hub is where images are typically pulled from when you just type the normal “docker pull blah” commands.  A registry is what it’s referred to as instead of a hub or repo.   To save time and bandwidth here is how you can stand up a persistent local registry to store your images.  Persistent meaning the image data is kept after the image is discarded.

Syntax here is working on CentOS 6.5

1) Install the needed bits.  This is no different than normal.

2) Start docker

3) Fire up the example registry.  This downloads and runs the registry image, exposes port 5000, and links local dir /opt/registry to /tmp/registry within the container.   This is key.  Otherwise, after the container stops the images go poof.

4) We could do this locally on this first machine, but we’ll show the syntax from somewhere else to illustrate.   On some other machine, first do the same install steps above to install the EPEL rpm and install Docker.   Then pull the images you want:

Snip20141023_4

5) List the images, and we see this image separates out a few versions of the OS.  CentOS7 is the latest (see how the IMAGE ID matches 87e5b…), CentOS6 is image 68edf..  and CentOS5 is 504a65…

Snip20141023_7

6) Add some tags to gives the images a new identity.  Replace “docker-reg” with your docker registry hostname.

7) List images again to verify

Snip20141023_8

8) Now push the tagged images to our local registry

9) Lastly, on some third machine with docker already installed (this post makes this handy by deploying these nodes as a catalog item), pull the time.  Notice it’s WAY fast now.  ~14 seconds in this screenshot.  Notice we only have the latest centos tagged, just pull the others and you’re good.

Snip20141023_13
Compare this to pulling from docker hub at about ~2:20.   For a large image like the SpringTrader app I built and this would cut down about an hour download time dramatically.
Snip20141023_14

I wanted to compare this to the SpringTrader app, so I pulled it earlier and began pushing it to my local registry.  One thing I noticed was it does buffer the image to disk when you push so be aware you will need the disk space (and time) available for this.   The time savings will happen later on subsequent deployments.  And i crashed my VM when running out of space the first time….

Then on some other node

Boom.  In about 10% of the time it normally takes to deploy this image she’s up and running!  It took about 13 minutes to download from Docker Hub, and about 2 minutes from my local registry.  That’s a win if you have a need to do this over and over.

 

 

Where are the images stored?

On a docker node, what I have been using to refer to the base machine and not the containers themselves, I found the docker files here.

Snip20141023_15

On the local registry, i found the files here.  Remember we told it to use /opt/registry on the base machine and map that to /tmp/registry within the container?

Snip20141023_17

Tagged , , ,

Small Things: vCAC 6.1 – "Data Collection" catalog entry

For anyone that does template configuration changes in vCAC can attest to how big of a pain it is to reconfigure the agents, shut down the machine, snapshot, browse the menu structure to where you force data collection, click collect data for all the items, browse to the blueprint config and wait for it to complete.    Well, hopefully this tip can speed that up just a little, or at least make it less of a headache for you.

vCAC 6.1 comes with a ton of vCO workflows out of the box.  One that caught my eye is “Force data collection”.

Snip20140919_22

 

Adding this workflow as a catalog item is a breeze under Advanced Services – Service Blueprints.  When complete it will show up like any other service or template:

Snip20140919_25

 

And does its job quite well:

Snip20140919_30

 

One warning, you will want to set a constant value for the one question it will prompt for in this workflow.   Edit the blueprint as such:

Snip20140919_26

 

And choose your IaaS (windows) server:

Snip20140919_27

 

I quickly installed a fresh new instance of vCAC & IaaS today, and not sure if it was an error during install or not but at first mine didn’t show any hosts here in the above screen shot.   I had to go into vCO with the client and run this workflow to add it.  Your results may vary.

Snip20140919_31

 

EDIT Sept 22 2014:   I wasn’t clear about where to find this workflow.  It’s found within these folders:

Orchestrator
Library
vCloud Automation Center
Infrastructure Administration
Extensibility
> Force Data Collection

Tagged , , ,

Small Things: NUC Sack

I transitioned some of my home lab onto tiny Intel NUC machines this year and just love the form factor and the fact that they are so portable. In fact I flew with them, that tiny NAS in the picture and a router to an event to demo some projects.  If only they went to 32 gb or higher, they would be perfect.

My buddy @jrrickard and his co-worker 3d printed me a rack for these little guys that we’re loving calling the “NUC Sack”.   Love love love it!!! Thanks guys!

If you want to print one yourself they posted the files on github here.   I’m told if you want to use this with the taller NUCs that will take a spinning disk inside, add 15mm to each of the sections.

NUC-SAC

Tagged ,

Small Things: vSphere 5.5 U2 – C# Client, Editing HWv10 VMs

Maybe I’m just becoming an old get off my lawn ex-operations curmudgeon in my “old” age, but I find the vSphere Client hard to part with (this message brought to you by Me, and only Me, and no one but Me).  I found it very annoying that if you upgraded any VM hardware versions to 10, you could no longer edit settings in the old client – even something common like mounting an ISO.

vSphere 5.5 U2 has brought us this, yay!  This allows editing of any features present in the old client which is good enough for basic stuff.

Snip20140917_15

 

 

EDIT: Whoops.. fixed the screenshot

Tagged ,

VMware vExpert 2014

vexpert_logo

Well all the other kids are posting it, so I guess I should.   I was tapped for a vExpert designation this year (probably for some of these mad science ramblings on the blog….just wait to see whats coming…)  I’m not really sure what this means yet but we’ll see!

Looks like companies use this as a marketing ploy to get people talking about their products.   I guess it works because there’s huge buzz on twitter about it.   PluralSight gave us a year service to their training videos which is cool.  Thanks PluralSight!

 

Tagged ,

VMware vCO Port and Account Quick Reference

Maybe it’s just me… but I can never for the life of me remember the correct ports and accounts to use for each part of VMware vCenter Orchestrator (vCO).  If it happens to me – it’s probably happening to a ton of people out there.

I hope this helps!

This is current as of vCO 5.5

Port 8281/https  Appliance Page.
Port 8283/https VCO Configuration, Default: vmware/vmware (or user vmware and PW set in OVF/OVA deployment)
Port 8480/https Appliance Configuration, Default: root/vmware (or user root and PW set in OVF/OVA deployment)
Client URL: https://fqdn:8281/vco/client/client.jnlp
Tagged ,

Version control and vCO Revisited (vCO 5.5.1 released)

Recently I posted a walkthrough of using a 3rd party framework called vFLOWer to convert packages exported from vCO to text format such that a version control system like github can read.   Well, that was a waste of time….!    VMware just released a new version of vCO (5.5.1) with this functionality built into the product now!  Excellent.

Screen Shot 2014-03-15 at 11.57.02 AMThere is a menu option on the package object to “Expand package to folder”

Which results in the following on the file system:

Screen Shot 2014-03-15 at 11.59.08 AM

Very cool!  Well done and timely for my bloggering.

Tagged , ,

Using vFLOWer and GitHub to bring (better) version control to vCO

Update March 15, 2014:  This type of functionality is now included in the product in vCO 5.5.1  See more here

vCO has very basic versioning built into the workflows.  It works for simple mistakes from version to version, but isn’t great for cross environment and multiple developer use cases.  Enter the vFLOWer Toolkit made by ByteLife.   They are a VMware Partner that built, what I think is, a BCDR solution using a lot of Orchestrator workflows.  When building their solution they found they really needed tighter control over versioning their work for their developers, so they built a solution and made it available.  Now I know it’s a little bit of a hack, but it works great.   Let’s look at how to use it here.  This post will be screenshot heavy, but is worth it to illustrate what is going on and how you do it.

I wanted to post all my workflows for the Dinner as a Service project, so let’s walk through that here as an excercise.  First you need to export your items from vCO. For this project, I used the instance of vCO embedded with the vCAC appliance.  The URL for the client is https://fqdn_of_vcac_appliance:8281/vco/client/client.jnlp

vCO Login

vCO Login

Once we’ve logged in, switch to the administrator view. Click on the Package tab.  If you aren’t already keeping your workflows in a package, go ahead and create one.  Right click, Add Package.

Create package

Create package

Give it a name

Give it a name

Now we need to add items to the package, so right click “Edit”

Edit the package

Edit the package

Switch to the Workflows tab, click Insert Workflows and add the workflows you want to put into this package.

Add the good stuff

Add the good stuff

Screen Shot 2014-03-02 at 9.37.47 AM

You’ll notice it captures all associated items with these workflows you specify.  In my case for this project, it added an Action and Plugin for REST.

REST Operation action

REST Operation action

REST Plugin

REST Plugin

Awesome, so we have the items we’re concerned about in a single package.   Right click and export package.

Export Package

Export Package

Options for export. Note the "export the values of configuration settings" that I will call out later.

Options for export. Note the “export the values of configuration settings” that I will call out later.

Now we have the package exported to the file system.

Now we have the package exported to the file system.

Now that we have our package exported, we need to convert it from the native binary format (I think?) to something that GitHub can use to do plain text revision control on.   Follow the User Guide on the vFLOWer website here or continue reading for a more detailed step by step process.

 We need to satisfy the requirements for vFLOWer which are – Apache ANT, Java, OpenSSL and Git.

Starting with ANT,  download the package here.  There is no installer for this one so we’re just going to extract it to our source dir at C:srcapache-ant-1.9.3  (use whatever path you want, but I’m going to use this src directory throughout).

ANT directory with the vCO package.

ANT directory with the vCO package.

We need to edit our environment variables to include ANT.     If you are unfamiliar with this process these steps will help as it’s not exactly an everyday activity.   Right click on Computer, select Properties.  (or Control Panel, ‘System and Security’, ‘System’).

Computer - Properties

Computer – Properties

Select ‘Advanced System Properties’.

Advanced System Settings

Advanced System Settings

Select ‘Environment Variables’.

Environment Variables

Environment Variables

Click ‘New’.   Add ‘ANT_HOME’ with our path to ‘C:srcapache-ant-1.9.3’.

ANT_HOME

ANT_HOME

This makes ANT_HOME available to the system but it’s not in the system path yet.    Scroll down to Path, click ‘Edit’, and add ‘;%ANT_HOME%bin’ to the end.

Path for ANT

Path for ANT

Test this by opening a new command prompt and type ‘ant -version’

Test ANT

Test ANT

That’s one down.   Now we need a Java JDK.  The newest at the time of this writing is jdk-7u51-windows-x64.exe and can be found here.

You want the JDK on the left.

You want the JDK on the left.

Choose your flavor.

Choose your flavor.

There is a windows installer for this one so it’s straight forward.    When complete, similar to what we did for the ANT environment variables, add JAVA_HOME, set it to C:Program FilesJavajdk1.7.0_51

JAVA_HOME

JAVA_HOME

Test with a new command prompt ‘echo %JAVA_HOME%’    This does not need to be added to the system path.  (EDIT: when I did this on Win7 it added automatically, on 2008 it didn’t.  That’s annoying.)

Testing JAVA_HOME

Testing JAVA_HOME

One more down, a few to go.   You may need to install the 2008 C++ Redistributable’s.   I have never known what’s in here but they say you should use it.  Kind of like vitamins.   Find it here.

For OpenSSL, grab the pre-compiled windows version here and install.  You may get a warning about the 2008 C++ package, this probably goes away if we would have rebooted – yay windows.  Choose the defaults.  Add to the system Path variable ‘;C:OpenSSL-Win32bin’ .

Update Path

Update Path

Test with the command ‘openssl version’ in a new command prompt.

Testing OpenSSL

Testing OpenSSL

Now we’re in the home stretch.  We just need the windows git package from here.

Download GIT because you can't get GIT without getting GIT first. GIT it?

Download GIT because you can’t get GIT without getting GIT first. GIT it?

Choose most of the defaults except two.  On the PATH screen, simplify things and have the installer modify the variable for you by choosing ‘Run GIT from the Windows Command Prompt’

Adjusting PATH automatically

Adjusting PATH automatically

On the line ending option page choose: Checkout as-is, commit unix-style line endings.

Line Ending Conversions

Line Ending Conversions

And test git real quick with ‘git –version’

Testing GIT

Testing GIT

Alrighty.  Now for the meat of it.   Everything up until now was just prep.  From here on out the vFLOWer doc is pretty descriptive.

You need to log into GitHub (create an account if you need to), and create a few ‘Fork’ of the vFLOWer repo.  If you are new to git, what this does is creates a new copy of this repo that is now owned by you but still chained to the parent.   If this was a real dev effort, we could push changes up to the parent if/when needed.  This is how developers work on multiple ‘branches’ of a code base without stomping on each other in one single repo location.    So login, browse to vFLOWer here, click fork.

Fork the original vFLOWer repo.

Fork the original vFLOWer repo.

Now you have your own fork of vFLOWer.  This will be where you store all your vCO stuff.   Let’s make it your own by renaming and edit the README.md   Click ‘Settings’ to rename.

Screen Shot 2014-03-02 at 2.54.24 PM

We’ll want to name it something meaningful.  For me, that always includes “stuff”:

my-vco-stuff

my-vco-stuff

When done, click ‘README.md’, click ‘edit’,  add your own content.  The read me file is similar to a index.html – we now see our new content at the root of the repo.

Click Edit

Click Edit

Enter your content

Comments Rock.

Comments Rock.

Add a comment before saving (we’ll talk more about comments later):

Screen Shot 2014-03-02 at 12.51.38 PM

And refresh the main page to see it:

Screen Shot 2014-03-02 at 12.51.51 PMNow comes the real magic of Git.   On the right hand of the main screen, copy the URL to the clipboard.   Mine is https://github.com/jasper9/my-vco-stuff.git

Grab the URL

Grab the URL

Back on the original windows machine we were on, change dir to our C:src directory and pull your Git repo.   This downloads the full thing to a subdir of the same name.

git clone

git clone

Check it out in windows explorer and it’ll look identical to the web interface.

We have files!

We have files!

Back to the vCO package we originally exported.  Copy this package into the /inout sub directory (C:srcmy-vco-stuffinout):

Package file in inout

Package file in inout

This is where ANT comes in. From the root of the repo (not the inout dir) run ‘ant pre commit’     You should see a lot of text scroll and end with a success message.

Text...text...DONE!

Text…text…DONE!

Check the inout directory and you should see the package is removed:

InOut Dir now

InOut Dir now

Check the content directory and you should see items from vCO that is familiar to you.  In my case I am looking at the Workflows dir.  My workflows were in a folder in vCO and that structure is maintained here at C:srcmy-vco-stuffcontentWorkflowsx My Library

Workflow Content

Workflow Content

Open up one of the workflows and you will see what it looks like in clear text.   One item of note is you might see values that were entered into parameters in vCO.   In my case I have ones for the Pushover service that is sensitive (API keys).  I want to remove these with placeholder values.  If you wish, you can uncheck a box in the vCO export process to not include any parameter data – if you have a lot this could be handy, but could also limit you in other ways by requiring some manual work to re-setup.

XML Content

XML Content

Now back in the root directory of this repo, type ‘git status’ to see the current status of this repo we’re in.  You’ll see a message complaining that there are new directories we weren’t expecting.

'git status' output

‘git status’ output

Add these automatically with the ‘git add .’ command.   Alternatively you could add items one by one if you ever need to in the future.

Now run status again and you’ll see it’s a bit happier, reporting cleanly that we have some new files in this snapshot of your code base.
Happier git status

Happier git status

Remember when you edited the README.md file and we added a message describing what we did?  That’s a comment.  Like you learned in “BASIC programming 101″ comments are a wonderful thing.  Maybe that was just me.  (Oh yeah, I can’t talk about git commit comments without dropping this XKCD on you)

MY HANDS ARE TYPING WORDS

MY HANDS ARE TYPING WORDS

So that change to the README.md we could have done here if we wanted to by manually editing the file.  To comment the changes we just made adding the vCO content type:

git comment attempt

git comment attempt

Two problems arise here (only the first is currently shown there)

Problem 1 I sometimes encounter – cut/paste from PDF’s or other sources don’t always work.  I blogged a little about it here.   In this case, it’s the PDF’s fault and the OSX setting didn’t fix it.  Replace the hyphen looking character with a real hyphen and run it again.
Problem 2 Now your commit was probably successful but it complains you haven’t set your account up right.  Let’s do that next.
Successful, kinda

Successful, kinda

Add your name and email with:

Then reset the author in the comment with:

My intention in this post isn’t to give a full primer on git (here’s one that looks good), so for the last part I’ll leave it simply at a high level explanation. We need to push our changes that we just committed to the server. We do so with the command:

Git push

Git push

Now if we refresh our github webpage, you should see the updates

Ch-cha-cha-cha-changes

Ch-cha-cha-cha-changes

Browse through the content and verify your workflows are there:

New stuff!

New stuff!

Awesome – now that’s how you export your packages, use this tool to convert them, and use Git to post them to GitHub.  Now to show how you would pull it down again, I’ll flip over to a fresh environment (but with all the prereq’s met).  This would be like if you were someone else and stumbled upon these workflows – how do I use someone else’s work.   Because this is a public repo, this is unauthenticated.

git clone

git clone

If you look into what was downloaded, it is as you would expect.  There is a content directory with our vCO stuff in XML format and no inout directory.

Fresh bits

Fresh bits

Now we need to use ANT to convert it back into a format we can import into vCO.

Yay!

Yay!

And check into the inout directory, and behold our package!

vCO Package

vCO Package

Load up the vCO client now.  Switch to the Administrator view.  The packages tab.  click the import button

Load it up!

Load it up!

You will see an SSL error you can ignore because the vFLOWer tool created a self signed cert for the process.   You may see a conflict message like this one shown if anything exists already or is at a newer version:

Warning(s)

Warning(s)

And there you go.  Now you can run your workflow in a different environment.  Or in this case, make dinner in someone else’s datacenter?

Ready to go!

Ready to go!

Tagged , , , , , ,

Evernote, Mac, Linux, command line……annoyances!

This has been been one of my absolute largest headaches working OSX into my daily workflow.   I use Evernote constantly.  And by constantly I mean I keep it all in there.   I have cooking stuff.  Toddler stuff.  Beer. Wine. School.  ToDo lists.   And that’s just the “Personal” stack.    If there’s something I don’t want to forget, I paste it in there.

Problem is when you do a lot of CLI work on servers, cutting/pasting in and out of evernote doesn’t always work as you’d expect…which can lead to possibly bad consequences.  Luckily it can be worked around!

First is pasting INTO Evernote.     Here’s a good example from a class I took.  my shell on OSX is black with green foreground.  When you copy and paste from this it keeps the formatting in Evernote:

Image

Argh.  Over time this gets wacky if you are pulling from websites….windows prompts…etc. (side note: that for loop is super useful for restarting all Puppet services..)

However there is a simple way to avoid it.  I found recently you can do SHIFT-CMD-V instead of just CMD-V to paste without formatting.  Brilliant!  Works the same with font sizes and faces too.

Ahhh that's better.

Ahhh that’s better.

Now that’s awesome and all, but that doesn’t solve my other annoyance.   Dashes and quotes!   Working in IT I’m constantly doing stuff at the command line of servers.  Lots of times arguments and flags are passed to commands via dashes (– or -) and “double quotes” or ‘single quotes’  etc.   You would think all would be well, but oh no no.

A lot of times I’ll have a block of text with a lot of these quotes and dashes.  A simple example:

Paste that into evernote,  later pull it out and the server yells back at you OH NO YOU DONT.  DENIED!   Looking closer I pasted:

Because it tried to be helpful and converted characters on you.  Imagine a large block of text and the manual editing that would require.  Let alone troubleshooting.

Avoid it all together by turning it off in the OSX settings:  System Preferences -> Keyboard -> Text,  uncheck “Use smart quotes and dashes”.   BOOM.  No more.

Be gone with thee!

Be gone with thee!

Hope that saves someone some headache.  Partial credit source given here.

Tagged , , , , ,