Category Archives: NewStuff

Technology found our new best friend.

Last night I built a robot that brought us to our new best friend. Meet Cash.

Cash

Before I explain this strange statement, first let’s back up.

Two weeks ago we found our beloved Maddie was stricken with a tumor on her spleen that ruptured.  I won’t dive into the heartbreaking details, but you can read about it here, here and here.

maddie_camping

To summarize: heartbroken.   That damn wonderful dog lived a great life and will never be replaced.  But we have found we’re a two dog family.  Enter the idea of visiting shelters….which is always fun..!

IMG_6736

After a few misses, we found just how competitive adopting dogs is in Boulder.  Yes, competitive.

Forget cycling, running, and climbing – the most competitive sport in Boulder is trying to adopt a dog from the pound link

Dogs fly out of the Boulder Humane Society.  There was one Jen was interested in that was adopted within hours of her becoming available.  We heard of one from employees that was going HOME within 45 minutes of stepping into the adoptables area.   Seriously.

The employees say to just keep an eye on the website.  So that’s what we did for a bit.  We noticed it was updated frequently throughout the day.  But there was no way to be notified of new dogs.  Enter my light bulb moment.

I saw there was no RSS feed, and (of course) no API.  So I took a glance at the HTML source and saw it would be super easy to screen scrape.  Muahhaha…… this will be easy peasy!    With just a little bit of hacking last night I had a working system that scraped their webpage every 15 minutes, stored it in a local database, and sent us an email when a dog became available!  Ha! Leg up, take that one, Boulder animal people.  Dog adoption performance enhancing drugs.

In the morning I surmized that wasn’t nearly geeky enough.  I added functionality to email us when a dog appeared to be adopted (wasn’t listed any more).  And since email is SO year 2000s, I spun up a new twitter account and had it tweet and direct message us when a dog showed up and went home.  I dub thee: Dog(S)talker.  Get it?  Dog Stalker.  Dogs Talker.  I kill me…

Low and behold…while I was out with the kid on his bike and Jen working on an extension to the chicken coop, DING. DM from the new robot:

Snip20160403_15

Due to an unfortunate typo in the code it is missing the details of the dog but still….. the fucking thing worked… A quick click on the link showed it was a 1 year old, Australian Kelpie Mix, and about 45 pounds.  Check check and check all the boxes!  I yelled across the street: “JEN!”  to which I immediately heard the reply, “I’M GETTING READY TO GO [to the shelter]!”

15 minutes later I received this:

IMG_6740

So an absolute max of 30 minutes from the time she was posted to the website to one of us showing up to check her out.

Long story short, he’s perfect for us.  I’ll post the code to github soon.   Perhaps if this is useful to anyone else I can add others to the notifications.

Snip20160403_16

 

Tagged , , ,

VMware Annoucements @ DockerCon 2015

Two announcements are being made by VMware at DockerCon today that I am pretty stoked about. Here’s a snipped of the details and a link roundup.  I’ll revisit these soon with deeper posts.

VMware AppCatalyst

VMware AppCatalyst is an API and Command Line Interface (CLI)-driven Mac hypervisor that is purpose-built for developers, with the goal of bringing the datacenter environment to the desktop.

“Introducing AppCatalyst – the desktop hypervisor for developers” – VMware Cloud Native Blog link

VMware Communities & download link

Update June 23, 2015 – Using Pivotal Lattice with AppCatalyst by @jrrickard link

Update June 23, 2015 – Vagrant provider for AppCatalyst link

Project Bonneville

..an extension of VMware vSphere that enables seamless support for Docker containers

VMware-Project-Bonneville

“Introducing Project Bonneville” – VMware Cloud Native Blog link

Overview video from VMware’s Brit Mad Scientist Ben Corrie here.

Update June 23, 2015 – Demo video of Bonneville link

Update June 23, 2015 – Bonneville running MS-DOS to play Lemmings link

Other Links

“Extending the Data Center with VMware AppCatalyst and Project Bonneville” – VMware Tribal Knowledge blog post

“VMware previews Project Bonneville, a Docker runtime that works with vSphere” – Venture Beat post (with some weird upside down image that is giving me a headache)

“A Different VMware: An API-Driven Hypervisor and a Docker Oriented vSphere” – The New Stack post

Update June 23, 2015 – “VMware targets new DevOps tools at Docker” – Silicon Angle link

Update June 23, 2015 – “VMware Doubles Down on Docker Integration with Project Bonneville” – Server Watch link

Update June 23, 2015 – “VMware AppCatalyst and Project Bonneville: ‘Datacenter On the Desktop'” – Virtualization Review link

Update June 23, 2015 – “VMware brings AppCatalyst and Project Bonneville technology previews” – InfoTech Lead link

Update June 23, 2015 – “VMware Brings More Tools To Docker Development” – Information Week link

Update June 23, 2015 – “VMware builds a magic mirror for containers and a desktop cloud” – The Register link

Update June 24, 2015 –“VMware Blunts Container Attack With Bonneville VM” – The Platform link

Update June 24, 2015 – “VMware containers go soup-to-nuts for cloud apps” – TechTarget link

Update June 24, 2015 – “VMware Embeds Docker Container Capabilities in Hypervisor” – Datacenter Knowledge link

 

A change…or pivot if you will…..

Pivotal_Logo_200I have been at VMware for 7 years (this week on the dot actually!).  That is a half a lifetime in IT Dog Years.  In that time I have done many different things, and been to many different places.  I saw (and at times helped (or tried to help) ) virtualization mature from a fringe lab thing that would never run production workloads efficiently and easily, to an established vendor that most people are using in some way.  Quite a ride!

Just after the July 4th holiday I will be (metaphorically, though not geographically) be walking a few blocks up the hill in Palo Alto from the VMware campus to a sister EMC Federation company, Pivotal.  I’ll be leaving the current Pre-Sales gig and getting my hands dirty directly in technology as a main focus.  I’m excited!

www.jaams.co

micro-services1-297x250I plan to continue the blogging weird and silly projects on here, though it will stray from a VMware focus to more broad devopsy topics in general.   Hence the slight change in name (mostly as a joke that I was told at GlueCon recently) – Josh as a (Micro) Service!  Kind of catchy don’t you think?

I’ll spare you all the pontificating on merits of focusing on one thick technology stack made up of all kinds of mashed together bits being a monolithic focus, and now for the future breaking it down into singular focus areas and doing each of them well……I don’t know… This joke might not work entirely, but I get a good laugh out of it anyway.

Onward!

“Security is mostly a superstition. It does not exist in nature, nor do the children of men as a whole experience it. Avoiding danger is no safer in the long run than outright exposure. Life is either a daring adventure, or nothing.”
Helen Keller

“Live every week like it’s Shark Week.” – Tracy Jordan

“It’s more fun to be a pirate than to join the Navy.” – Steve Jobs

Tagged , , ,

New backup option for Synology devices

synology1512I have two Synology NAS devices in my home lab that I’ve always struggled with being sure I have full backups of as they have grown pretty large over time.   I have the 5 bay DS1512+ (with an additional 5 bay expansion), and I have a tiny 2 bay DS214SE.    I didn’t plan my use of them this why but it just kind of evolved over time, such is how my lab is.  Something breaks or is slow, I tweak to squeeze out better performance on a small budget and life goes on.

Cur615976_0_frently I use the large array for normal file storage (music, photos, videos, ISOs, etc…), I used the extra expansion for transition storage over time when I moved stuff around (mainly VMs).   I originally used the tiny 2bay NAS for my tiny portable lab based on NUCs.  I now have the management components living on a iSCSI lun on there (VC, PSC, vCO, DNS, AD…) and for all compute I am now using VSAN across three white box machines (which is working fantastic!).

I’ve always struggled with backups in my lab.  Any free options out there either won’t cover two VCs, cpu core limited or VM count limited.  I’ve been using William Lam’s GhettoVCB forever, which is solid but mostly manual.

Enter….what I found this past weekend.  Synology management software images that will work in a VM or baremetal!   This is literally the same OS that runs on their devices.  I first tried it in a VM to test it out.  All seemed well except for updating as it appears to break so you have to wait for unoffical patches.   To use this for real, I went ahead and swapped out the USB drive on my HP N40L which was previously running FreeNAS for backups.

This allowed me to setup a reoccurring RSync from the DS1512:

Snip20150527_4…And also allowed me to setup a reoccurring backup of the iSCSI LUN (holding the management VMs):

Snip20150527_5

While the ~250gb iSCSI backup was pretty quick, the Rsync of 6 TB of small & large files is taking a while.   Performance seems pretty decent, at least for my home lab that can be kind of…iffy given the amount of crazy crap I run on the large flat network of consumer level 1GB switches no tuning whatsoever.

Snip20150527_6

Prior to this I was doing all my backups manually – both the rsyncs and ghettovcb backups.   I would then a few times a year move a backup set outside of my house to a family member.  I highly suggest you do the same!  I do my best to follow the 3-2-1 rule, though I’m not doing great on the multiple types of media as my photo collection has grown too large to use “cloud” storage useful or economical.

Check it out for yourselves!

Install information (credit as the source!) http://www.bussink.ch/?p=1672

More information http://www.xpenology.nl/

Downloads http://xpenology.me/downloads/

Tagged , , , ,

Experiment: Pooling in vRA & Code Stream

Background

I recently attended DevOpsDays Rockies which is a community oriented DevOps conference (check them out in your area, it was great!).  I saw a talk by @aspen (from Twitter/Gnip) entitled “Bare Metal Deployments with Chef”.   He described something he/they built that, if I recall correctly, uses a PXE/Chef/MagicpixieDust to pull from a pool of standby bare metal hardware to fully automate bringing it into a production cluster for Cassandra (or what have you).

This got me thinking on something I was struggling with lately.  Whenever I develop blueprints in Application Director / Application Services, or just vRA/Code Stream, the bulk of the time I just hit the go button and wait.  Look at the error message, tweak and repeat.  The bottleneck by far is in waiting for the VM to provision.  Partly this is due to the architecture of the products, but also it has to do with the slow nested development environments I have to use.  We can do better…..!

Products using pooling

I then started thinking about what VDM / Horizon View have always done for this concept.  If I recall correctly, as it’s been years and years since I’ve worked with it, to speed up deployments of a desktop to a user, a pool concept exists so that there will always be one available on demand to be used.   I don’t have much visibility into it but I am also told the VMware Hands On Labs does the same – keeps a certain number of labs ready to be used so the user does not have to wait for it to spin up.  Interesting.

The idea

So I thought – how could I bring this upfront deployment time to the products I’m working with today to dramatically speed up development time?   And this is what I built – a pooling concept for vRA & Code Stream managed by vRO workflows.

Details – How Redis Works

When planning this out I realized I needed a way to store a small bit of persistent data.   I wanted to use something new (to me) so I looked at a few NoSQL solutions since I’ve wanted to learn one.  I decided on Redis as a key value store, and found Webdis which provides a light REST api into Redis.

I couldn’t find any existing vCO plugins for Redis I/O which is fine, the calls are super simple:

Example of assigning a value of a string variable:

Snip20150517_5The redis command is: “set stringName stringValue”
So the webdis URL to “put” at is “http://fqdn/SET/stringName stringValue”

Then to read the variable back:

Snip20150517_6The redis command is: “get stringName stringValue”
So the webdis URL to “get” at is “http://fqdn/GET/stringName”

Easy peasy. There is similar functional for lists, with commands to pop a value off either end of the list.  This is all I needed, a few simple variables (for things like the pool size) and a list (for things like the list of VMs storing IP addresses & names).

So in vCO I just created a bunch of REST operations that used various number of parameters in the URL line:

Snip20150517_7
I found the most efficient way to run these operations was to parametrize the operation name, and pass it to a single workflow to do the I/O

Details – Workflow(s)

The bulk of the work for this pooling concept is done in the following workflow that runs every 15 minutes.

Snip20150517_8In general it works like this:

  • Check if the workloads are locked – since it can take time to deploy the VMs, only one deployment will be going at a time.
    • If locked, end.
    • If not locked, continue.
  • Lock the deploys.
  • Get the pool max target (I generally set this to 10 or 20 for testing).
  • Get the current pool size (the length of the list in Redis.  much faster than asking vSphere/vRA).
  • If the current size is not at the target, deploy until it is reached.
  • Unlock the deploys.
  • Profit.

I did not have to do it this way, but the nested workflow that does the actual VM deployments is requesting vRA catalog items.

In Action

After I got it fully working and the pool populated, you can check the list values with this type of Redis query:

Snip20150517_9

Redis: lrange vmlist 0 -1 (-1 means all)
Webdis: http://fqdn/LRANGE/vmlist/0/-1

The matching machines in vSphere:

Snip20150517_11

In Action – Code Stream

Normally in a simple Code Stream pipeline you would deploy a VM by requesting the specific blueprint via vRA like this:

Snip20150517_19

In this solution, instead I use a custom action to grab the VM from the pool and return the IP back to the pipeline as a variable.  Then I treat the VM like it’s an existing machine and continue on and at the end delete the machine.

Snip20150517_18

This reduces the list in redis by one, so the next time the scheduled workflow runs that checks the list size it will deploy a new one.

(Kind of) Continuous Deployment

I have a job in Jenkins that builds the sample application I am using from source in Git, pushes the compiled code to Artifactory and does a post build action that calls Code Stream to deploy.

Snip20150517_15

I wanted to see if there were any bugs in my code, so I wanted this whole thing to run end to end over and over and over…   I configured the Jenkins job to build every 30 minutes.  I went on vacation the week after I built this solution so I wanted to see if over time anything broke down.  Amazingly enough it kept on trucking while I was gone, and even got up to the mid 700’s in Jenkins builds.   Neat!

Snip20150517_12

Jenkins builds

Artifacts

Artifacts

Code Stream executions

Code Stream executions

Summary

To my surprise, this actually works pretty darn well.  I figured my implementation would be so-so but the idea would get across.  It turns out, what I’ve built here is darn handy and I’ll probably be using it the next time I am in a development cycle.

Post any questions here and I’ll try to answer them.   I’m not planning to post my workflows publicly just yet, fyi.

Tagged , , , , , , , , , , , ,

Introducing VMware Project Photon (#vmwcna)

VMW-LOGO-PHOTONUnless you have been hiding under an IT rock, you no doubt have heard about the new crop of tiny linux OS releases as of late that are positioned as a “Container Host Runtime” or “Linux Container OS” (here, here, here).   They are stripped down to the bare essentials and geared towards running containers efficiently at scale.   CoreOS, Atom, Snappy and so on.  Today VMware’s Cloud Native team is introducing Project Photon as their flavor of this ecosystem.  (Link to GitHub page)

Entirely open source. (Free as in beer.)  Built in VMware tools.  Optimized for the VMware hypervisors. There are lots of benefits for VMware building their own from the kernel and not forking an existing OS that will become more clear over time, but I will leave it to the official messaging for now.

What is Project Photon?

Project Photon is a tech preview of an open source, Linux container host runtime optimized for vSphere. Photon is extensible, lightweight, and supports the most common container formats including Docker, Rocket (rkt) and Garden.

Project Photon includes a small footprint, yum-compatible, package-based lifecycle management system, and will support an rpm-ostree image-based system versioning.

When used with development tools and environments such as VMware Fusion, VMware Workstation, HashiCorp (Vagrant and Atlas) and production runtime environment (vSphere, vCloud Air), Photon allows seamless migration of container based Apps from development to production.

From “Getting Started” documentation

It may not make sense to some why VMware is releasing a linux OS.  This will become more clear over time.  But for today, just think about the power of VMware owning the hypervisor underneath, AND the VM operating system as a platform for running containers.  You get all the benefit of the vSphere world (HA, DRS, FT, NSX, vSAN, vMotion….) and all the benefits of containers!  Plus… remember VMfork that Duncan has blogged about?  hmmmmm….

 

Installation

Snip20150418_75

 

Snip20150418_74

 

Seriously….. Using the minimal install, 12second install time in Fusion on my MacBook Pro.  303 mb footprint.  That. is. awesome.  The following are the sizes and average install times I’ve noticed.  Booting is literally just a few moments.

The install comes in three flavors from the same .ISO, (or you can custom pick packages)

Full: 1.7GB.  40 to 60 seconds to install
Minimum: 303mb. 10 to 20 seconds to install
Micro: 259mb. 8 to 12 seconds to install

 

Photon OS (Micro): Photon Micro is a completely stripped down version of Photon that can serve as an application container, but doesn’t have sufficient packages for hosting containers. This version is only suited for running an application as a container. Due to the extremely limited set of packages installed, this might be considered the most secure version.

Photon Container OS (Minimum): Photon Minimum is a very lightweight version of the container host runtime that is best suited for container management and hosting. There is sufficient packaging and functionality to allow most common operations around modifying existing containers, as well as being a highly performant and full-featured runtime.

Photon Full OS (All): Photon Full includes several additional packages to enhance the authoring and packaging of containerized applications and/or system customization. For simply running containers, Photon Full will be overkill. Use Photon Full for developing and packaging the application that will be run as a container, as well as authoring the container, itself. For testing and validation purposes, Photon Full will include all components necessary to run containers.

Photon Custom OS: Photon Custom provides complete flexibility and control for how you want to create a specific container runtime environment. Use Photon Custom to create a specific environment that might add incremental & required functionality between the Micro and Minimum footprints or if there is specific framework that you would like installed.

From “Getting Started” documentation

Using Photon / SystemD

I’ll be the first to admit I have not adopted CentOS7 yet as all my labs are still using CentOS6, so I was not familiar with the new SystemD commands as of yet.  There is some good info on it here and here.

TLDR; for services,  Project Photon uses systemd:
You no longer are running chkconfig or /etc/init.d/ scripts.  Instead you use systemctl enable service and systemctl start postfix.

Also networking is different, you edit files in /etc/systemd/network instead of sysconfig.  I’ll show more info on that below.

One more helpful thing to know is there are no logs in your familiar home of /var/log/, they are managed centrally in journalctl. Digital Ocean has a great overview of the usage of it here.  I won’t rehash all of the functionality that they wrote about but I’ll give a quick example.

TLDR; for logging, Project Photon uses journalctl:
You no longer use /var/log/postfix.log.  Instead you use (to continuously tail) journalctl -f -u postfix


How to Get Started

VMware has posted a bunch of great getting started guides here that walk through deploying on Fusion, vSphere, GCE, AWS, Air, etc…  In addition to those guides, here are some tips on configuration to help get those that are not familiar up and running right away.

Here is what I’ve been doing when I deploy a new machine.  I’ve found each of these have exact syntax and capitalization that are important, otherwise the IP does not get configured.

  • Allow root SSH access in /etc/ssh/sshd_config
  • Set the correct hostname in /etc/sysconfig/network
  • Configure a static IP by:

mv /etc/systemd/network/10-dhcp-eth0.network /etc/systemd/network/static.network

Edit the contents to be:

  • Update the hosts file to be sure you have short and fqdn set on 127.0.0.1

  • Then run the commands to configure the hostname

  • I like using keys for SSH to login quicker in my lab

Good to go!

 

Tagged , , , , , ,

Where is the OVA/OVF for vCenter 6.0 Appliance?

moooar_vsphere6You might have noticed already that the new vCenter Appliance (VCSA) 6.0 is only being distributed as an .ISO.  While I do wish an OVA/OVF would have been available too, the new installer is pretty sweet.

Snip20150314_36

When you mount this .ISO, you get a web page interface that cleanly and clearly walks you through deploying directly to an ESXi host.  In my opinion this is a pretty good user experience, which is something that has been lacking at the install time of these products for quite a while.  (Side note – check out the VIO installer too.  It’s slick as well.)

I do think there are a few isolated use cases for directly using an OVA or OVF however.  A vCD (or other hosted environment) is a good example, when you do not have access to the underlying hosts.  Or if you encounter some issues in your browser, perhaps due to draconian IT security measures.  All is not lost!   The OVA is actually hiding in the ISO, you just have to find it.

 

Disclaimer: This is not recommended or supported by VMware.  There are a lot of user input checks that go into the new installer, so use it when you deploy your VC for real.  This should only be used for testing/sandboxing/may blow up in your face. There I said it, so don’t yell at me when you break something!

 

The ISO itself will look like this on disk:

Snip20150314_30

When you mount it, the directory structure looks like:

Snip20150314_31

In the “vcsa” directory, you will find a file with no extention like this:

Snip20150314_32

This is the actual .OVA   Copy it somewhere local so it’s writable and add the .OVA extension.

Snip20150314_33

Now extract it.  Did not know you can just use an unzip utility to extract an OVA?  Well now you do.

Snip20150314_34

Aaaaaand here are your familiar files!

Snip20150314_35

Now I will warn you, you could will have a bad day if you directly try to use this .ovf with vCD.  It seemed to work fine for me in vCenter, but vCD does not understand a lot of the new options and fields in the format of this file.  The outcome is it’s really difficult to deploy anything other than an embedded node (PSC+VC).  If you want to deploy JUST the PSC or VC services you have to do quite a bit of editing.  Luckily I asked around before I spent time on this and I found the VMware HOL Wizard Doug Baer (@dobaer) had already pulled his hair out getting it to work.   So behold!  Customized .OVF files for either PSC or VC nodes!

I warn you, your results may/will vary with these so the usual disclaimers about NOT using this in production and NOT calling support and expecting help apply.

You can find the contents of these files on a Github repo of mine here, and again all credit to Doug for these:

Snip20150314_39

 

There you have it.  Go get some vSphere 6 vCenter appliance action on.

Tagged , , , , ,

ESXi 6 Whitebox Realtek NIC issues

moooar_vsphere6I upgraded one of my whiteboxes to ESXi 6 and noticed the Realtek NIC was no longer detected.  I tried to install the same VIB I used previously and got dependency errors.   A quick google search later and I found this post that references a communities post that has a fix.  Yay!

Note – this probably shouldn’t have been posted publically during the beta program but meh, past is the past.

Here’s the VIB: http://vibsdepot.v-front.de/depot/vft/net51-drivers-1.0/net51-drivers-1.0.0-1vft.510.0.0.799733.x86_64.vib  I can confirm this working on a Gigabyte Z68X-UD3H-B3 that has a Realtek 8168 on board.   Many thanks to the author!

Tagged , , , , ,

New release: VMware Software Manager – Download Service

Release Notes

Documentation

Snip20150311_9Today VMware is releasing a trove of software.  One small tool that will surely make release days like this much easier is Software Manager.   I just tried it out and it’s exactly what you will expect.   No more browsing through the somewhat painful download web pages!  Have it all come to you.

The tool is windows only, and comes in at a whopping 17MB for the tool itself.  It comes packaged as a MSI, quickly install it, specify a location (that has enough space for the many gigs of of downloads) and it fires up a web page connecting to localhost on port 8000.  Login with your MyVMware credentials.   BAM!  You will see all the downloads you are entitled to and have a very easy one click download for a whole suite of components.

Snip20150311_13

 

UPDATE:  vSphere 6.0 is now showing up in the product.   If you installed it first thing today, you might need to stop and restart the service for it to show up.   Worst case, kill your browser with a hammer.

Snip20150312_20

Tagged , ,

What I’ve been up to lately: CTO Ambassador, vExpert 2015, VCP6-Cloud & Conferences

Busy time in the salt mine day job!  RADIO_CTO_v6 Hope this does not read as a bunch of humble bragging, just keeping tabs on all the happenings here on the blog.

I’ve recently joined a field facing group at VMware called the CTO Ambassadors on a rotation for the next two years.  This is really exciting for me as it is similar in intent to a program that I helped build in my past life in support.  This CTOA program in their words:

 

“focuses on creating and developing technology focused communities that span R&D and the field. This enables the effective flow of information and knowledge, backwards and forwards, between R&D and our field, and ultimately with our customers and partners, which in turn drives more profitable relationships, greater revenue, and new and improved products. “

I am excited to be able to take part in this experience.  The next year or two will be exciting to have a chance at helping shape many things internal to the company but also have a clear priority of taking part in other communities for the company.

vexpert_logoAlso, I got word that I was renewed as a VMware vExpert for 2015.  That’s fun too!  This is a designation for being active in the social media and blogging communities.  And is kind of a fun title too.

I passeCloud_VCPd the new VCP6-Cloud test this week.  I will admit it covers such a broad range of topics that it is a bit difficult in parts as I do not touch many of these too often any more.  And there are a number of questions that would purely be answered by just glancing at a GUI.   I used to be able to breeze in to the core VCP exam, take the test in 20 minutes and pass without ever thinking twice when I was in a purely a vSphere focused role.  Not sure if it is the test or it is me now.  Probably both column A and column B.

devops-days-rockiesFinally, I signed up for a few conferences coming up in Colorado that look to be exciting.  GlueCon in May (Broomfield, Colorado) and DevOps Days Denver.  Looking forward to hearing some great talks on some hot topics right now.

 

Tagged , , , , , ,