Coté Memo #055: It's cold in Toronto

Nov 18, 2014

I mentioned the Software Defined Talk recording last week, the episode is up for those who were waiting.
At DockerCon EU - turns out I'll be at the Amsterdam DocerCon the first week of Dec, just for a day and half, though. If you're there, it'd be fun to meetup. I'll have been traveling all of that week, so you can see my brain turned to mush!

Tech & Work World

Quick Hits

Netcraft: DigitalOcean Now Third-Largest Cloud - measurements like these are always weird, but the point is still there: DigitalOcean is on mega-growth, them developers like it.
Apple Pay grabs 50% of McDonald’s tap-to-pay transactions, gives retailers hope for future of mobile payments - I've been using Apple Pay when possible. Since you can use it where NFC is accepted, this means it works in taxi cabs, random stores, etc. I tried it at our local mega-grocery chain HEB the other day and it didn't take. What I find most annoying about the interface is how little feedback there is: I'd like to hold the phone close to the terminal and have it tell me things like "working" or "can't find it." I often find myself holding my phone up to a terminal and just sort of waiting for something...or nothing to happen. Yup. First world problems.
The problems with "large, complex projects" in the enterprise - some nice visualizations in this all too brief post.
Why Entrepreneurs Should Be Respected More Than Loved - "leadership" and "management" are exhausting, no matter how many tips and such you get.
EclipseCon 2015 - Register Now - I used to go to EclipseCon a lot when I was RedMonk, it's always a fun conference, full of coders and lots of Europeans.
Lydia at Gartner: "AWS could be as much as 20% of all VMs, 10%+ of all OS instances, and well over 5x all other cloud IaaS providers put together."
Podcasts are neither exploding nor dying - they've been steadily growing for a decade - file under "don't call it a come back." To be fair, awareness and desire are what's needed. That would lead to good advertising rates in podcasts, injecting more money in there (I'd hope). I'm pretty sure the brand value of Squarespace amongst nerds is purely based on their multi-year podcasting carpet bombing. Igloo too. I'd have never come across those brands as much as I do if it weren't for podcasts.

"Script" for Docker orchestration

Oftentimes, I write a "script" for presentations. I never read it for the presentation (maybe I should!), but it helps me organize my thoughts and presentation. Here's the one I used for the Docker orchestration webinar I did with CloudSoft last week:

Macro-context: the demand for building your own PaaS - DevOps, cloud, etc.

[chart on DevOps delivery pipeline, old dto solutions DevOps pipeline]

I like to reduce things down to brutal simplicity. I have a lot going on in my work and personal life so I have to leave nuance for entertainment. To me, cloud is mostly about supporting custom written software, whether that's for something like a SaaS (from social to ERP), consumer facing business applications (like online banking), or applications used by companies to help run their business. And what that means is creating a delivery pipeline that encompass all the phases of an applications life and automates each step as much as possible to reduce bottlenecks and increase throughput. You have to look at this pipeline as a mission critical process in your business: from development to production, it's you're factory, the thing that helps you make money...not just a cost center. You're pushing out incrementally improving software that runs your business.

Custom written software is the most valuable work-load for cloud, I'd theorize, where value is rated by how technology can help a company get competitive differentiation.

So, something like Docker is especially interesting because it promises to speed up that pipeline. I don't think anyone know exactly what it will shape up to be, but it looks like the "private PaaS" answer we've all be looking for.

[chart with SaaS, PaaS, ISaaS, IaaS market-sizing - put OpenStack market-sizing in there next to it]

Now, PaaS is an odd category of public cloud. It seems like the perfect realization of the efficiencies of cloud, and yet it keeps limping along as a market as this 451 market-sizing shows.

Much of that revenue comes from Salesforce which has its own Force.com platform (a "PaaS for SaaS" as we call it) and Heroku.

[chart on demand for private cloud]

For some reason, developers and companies have a lust for build your own cloud: they either get their own gear or rent raw IaaS and build up their own stacks to support as automated as a DevOps delivery pipeline as possible. Maybe it's cost (I haven't ever heard a developer say public PaaSes like Heroku are cheap), maybe it's the need to exactly customize functionality, maybe it's good old paranoia and FUD (which could be justified, who knows).

Whatever the case, people want control over their stacks, and that's where things get interesting. The more control you have, the more you have to worry about the more hassle there is.

We seem to have a long way to go to replicate the magical, effortless push to deploy demo we remember from early public PaaS days. Much of what's needed is what's currently going under the title of "orchestration" which, roughly, means "making sure my complex distributed system is installed and configured properly...and then allowing me to modify its runtime characteristics and upgrade it." You know, getting the application up and running, tuning its performance ongoing as needed, and upgrading it.

[insert chart showing rise of new automation tools/brands]

This area has long been the domain of custom shell scripts, manual configuration, and, thankfully, in recent years configuration management companies like Chef and Puppet (who are taking over the automation reigns from OpsWare and Bladelogic).

Docker has burst on to the scene of late as an interesting salve for cloud infrastructure woes. To me, it starts with the right goals: make using cloud easier for developers. That may seem subtle, but it's different than most infrastructure software goals which is make like easier for sysadmins and auditors.

To keep pushing on the dream of being able to build your own PaaS, the ad hoc community around all of this has been obsessing about orchestration of Docker-based clouds, let's call them, of late. So let's look at that.

Emerging market for orchestration

Mindmap from Krishnan Subramanian

When it comes Docker orchestration, there are almost too many projects to count, and even a few products. I love this mindmap from Krishnan that shows just full this market is - and tedious for analyst to keep up with. This is a good sign, however: there's so much interest and passion in figuring out Docker and how to orchestrate it that surely, something will work.

Emerging requirements for orchestration

When I look across what all of these projects are trying to do - and slap in some old IT Service Management think - I come up a list of requirements for orchestration. Some of them may seem obvious, but it's always good to be explicit. If you spot ones that are wrong, or missing, you should pass them along and perhaps we can winnow down a list. We don't need a manifesto or any nonsense like that, but in studying this space, you do find a distinct lack of architectural-level specifications and requirements - which is fine, people are busy coding.

Cluster/fleet management

Operate in terms of multiple nodes, not single nodes - configuring a single Docker node is mind-blowingly easy, doing it over 50 or 100 nodes gets to be tedious, esp. if you want to continually be turning over builds. Pets vs. cattle and all that.

Configuration management & Automation

Application modeling that describes the layout and configuration of various components - this is an old ITSM notion, "service modeling," which got bogged down in drag and drop fantasy (just like UML). You need to model what all the different components are and how they fit together
Basic CRUD - creating nodes, updating nodes, restarting them as needed. You want more than just modeling what a node looks like, you want you orchestrator to actually do something.
Separating configuration from basic state - easily modify configuration without having to change too much about each node/image, like changing port numbers easily without rebuilding the entire node
Ensure proper configuration passing across nodes - passing server names and ports to servers, handing out credentials, wiring in service directories, etc.

Heterogeneous platform support

Support for different infrastructure, bare-metal, to plain old virtualization, to multipule clouds - some might call this "hybrid cloud" or "multi-cloud" - useful just for moving along the pipeline

Baby and bathwatering ITSM

Asset database to track all your cows - another ITSM trick. this starts getting into "enterprise" needs, but it handy even if aren't tweedy. You need to know what you have out in the wild and quickly locate it when things go wrong.
Capacity management and adjustment of resources - not only monitoring if you're over (or under!) capacity, but actually going back to your CRUD operations to make adjustments on your nodes. This is also where keeping configuration separate from node state is handy: you could increase memory, keeping the same configuration, for example, without having to rebuild or swap out nodes.

ABC

Ease of use, and esp. low cost - otherwise, why not just use a full on PaaS? This is always easy to forget, but it's sort of the point of all of this. Ask yourself, is this easy and quick to use? If it's not, something is wrong.

This last point is key. You need to remember that once you've done all the above, that's when the difficult work begins. You still need to come up with an idea for an actual application and its features that will help your business. You need to stop orchestrating and start coding, not to mention working on the product management that will tell you what to code in the first place. Don't get all caught up in all this Heathkit stuff: save your cycles for the most valuable thing: ABC.

[Always be coding chart]

And with that, I want to pass it over to CloudSoft to tell you how they're helping you get closing to coding.

Fun & IRL

Meta-data

Elsewhere: @cote, cote.io, memo@cote.io
Archives: http://coteindustries.com/tagged/cotememo
Subscribe: http://coteindustries.com/memo