The Silo Unifier’s Paradox
Solving the IT silo problem is challenging due to the lack of a single responsible authority, requiring several approaches and acceptance of the system's inherent limitations.
If no one owns the problem, no one can fix it - The Silo Unifier’s Paradox
Solving the IT silo problem is challenging as there’s no one person responsible for the entire process. Each solo is owned by different groups like software developers, operations, security, project managers, and more.
These groups often have competing goals, desires, incentives, and lift-styles/worldviews. Agile and DevOps practices attempted to break these silos, shifting activities like testing, security and ops “left” to developers, or centralizing shared services (“platforms” now-a-days). The people shifted “left” probably don’t like these tasks being taken from them, and developers don’t want to be told what to do.
In general, this silo approach is bad because the conflicting goals and the need to coordinate (have meetings, explain what’s going on, get approvals, etc.) add tremendous friction and toil to the system that tends to bring down the overall “goodness” of the whole system.
The problem fixing the silo problem lies in the same inherent nature of siloed organizations that makes it bad: there is no single person in charge who can decide, mandate, and reward good behavior. To tackle this problem, one must accept and work within the limitations of the system they’re trying to fix. You have to be the enemy to defeat the enemy.
Here’s some approaches to fixing the problem. None are perfect, many are crappy.
The Powerless Tzar
Appointing a silo unification “tzar” may seem like a solution, but they often lack the authority to reward good behavior or change organizational structures. As a result, they have limited tools for driving change, relying heavily on silo owners to do so.
You’ll also have a staffing problem. Very few executives want to go from owning a P&L to “VP of Special Projects.”
Bottom-Up Change and De Facto Standards
Introducing bottom-up change using de facto standards can work, but it relies on long cycles beyond your control. For instance, Kubernetes is now a de facto standard with widespread agreement across silos, but public cloud adoption was not as straightforward in the past.
Slow and Steady Change
You can start with one project doing things in the new way, more or less creating a new organization. Then you can add a new group, a new set of tools. And two more, three more, etc. This often takes years and ends up with a new organization - in not-vendor enterprises I’ve heard of this organization being called “product” with a “VP of Product” which makes sense.
The issue here is that it takes multiple years of hands-on attention, learning, adapting, and eventually, I’m assuming, board-level authorization to setup a new group. Also, it’s slow. This is the tactic I find myself writing the most about in my books, especially Monolithic Transformation. It’s what I’ve seen work the several times. But, again, it takes commitment and perseverance.
Your “best” bet
Fixing silos is extremely difficult, nearing impossible. In general, it’s probably better to start with incrementally improving what you’re already doing. And, really, that’s probably good enough.
Just start by making sure you’re actually a learning and adapting organization: do you have feedback loops in place that tell you what’s working or not, are you making changes each quarter to try new things and cement new, better ways of working? If you look back over the past year and you don’t have, let’s say, five instances where you god feedback about a process or way of working and put a change in place that stuck, you’re probably not doing any of this. Do that first!
At the same time, you need to introduce a (pardon me, here) “workstream” in your annual planning to look at how you can latch onto the open source/de facto standard method and the slow and steady method. More than likely, if you’re running your business on software, you should be applying new ways of working to at least three projects a year just to experiment with and learn new ways of working. If they achieve whatever goals you have and seem to be, let’s say 300% better than what you’re currently doing, in the next fiscal cycle, you should do more.
I don’t know: I just made all those numbers up. You should start doing all that and learn what your numbers are.
I mean, if you can more rapidly change things, unify the silos, for God’s sake, do that. But if not, first make sure you’re actually a learning organization and then start slowly experimenting with the slow and steady plan.
Vendor sidebar: Navigating the Silo Landscape
I’ve mostly worked at vendors that are trying to sell the fix to this silo problem, so I think a lot about the vendor problem. By “vendor” I mean companies that sell software and also public cloud services.
Vendors have to communicate with various stakeholders, often with conflicting motivations, needs, and skill levels. For example, infrastructure personnel prioritize reliability and cost-effectiveness, developers focus on new software creation, and security experts seek to prevent breaches. These conflicting interests create challenges for vendors who must engage in lengthy conversations, negotiations, and planning to accommodate these differences.
Top-down sales can be challenging but achievable for larger vendors with the resources and patience to execute them. This approach involves numerous meetings, negotiations, and multi-year business plans for ROI.
Having a significant competitive advantage based on differentiation can also work, though it’s more difficult and rare. Examples include AWS in its early days (alternatives simply didn’t exist), IntelliJ, and freemium/PLG models from companies like Atlassian and systems management vendors such as Datadog.
Vendors can also use open source and establishing de factor standards to drive bottoms-up change. For developer and infrastructure markets, this approach is often favored, or simply emergent. Vendors must invest considerable effort in finding, defining, and chasing standards over the years while remaining active community members.
You have to take a leap of faith that it’ll work as the payoff will likely take years and you’ll probably never have a satisfying link between the money you invest in the open source stuff and your profits. PLG-ish and SaaS models - open core - can solve that link, probably. But, those models also take a lot of faith and work.
Navigating the Silo Unifier’s Paradox
If you’re faced with the Silo Unifier’s Paradox you must accept the limitations of the system they’re trying to improve and find creative ways to work within these constraints.
On the enterprise side, you probably need to lower your expectations and make sure your silos are at least following best practices. You should put in place annual plans to, basically, do sanctioned skunk works organizations. On the vendor side: good luck storming the castle!" You’ll need to be clever, persistent, and be even more of a learning organization than your buyers.
Wastebook
“Creating a [market] category seems like a good idea because it seems like a shortcut to fame. What you’re actually doing is digging a small pond you can sit in all by yourself next to a sign you painted yourself with whatever category name you picked on it. “A leading minnow in this tiny pool” doesn’t sound so great, does it?” Justin Warren’s March 20th, 2023 newsletter.
I’ve been looking at hosting plans for ghost, which seems to provide a lot of the things I’d like to do with my online thing. Have a newsletter, have a blog, have static pages, and have some kind of “walled garden.” This last is hard to express, but I want something like the space a podcast provides: people have to really want to be in that space. It’s like charging a dollar for something: it’s not about making money, it’s gatekeeping. Of course, long term, who wouldn’t want to make money off blogging too, but that’s a long way of, or just a layoff off. Anyhow! You’re going to end up paying rates close to ghost.org, which are high - especially because you pay per member whether that member pays or not. What the alternate hosters hide from their pricing is that you have to sign up with mailgun - or something - to send emails. There seems to be some trick where you can avoidng paying $35 a month to mailgun…but…I can’t figure that out. In the meantime, substack plus using micro.blog for cote.io is good enough.
Speaking of: if you’re a big fan of the wastebook, I’ve created a side-newsletter that’s much more raw and unfiltered, a true scrap book/common place book. I’m doing something weird/stupid: it costs money to sign up for it. ¯_(ツ)_/¯ But, not much at all. You get a seven day free trial, so You can check it out. The annual price is as cheap as I could make it. I plan on putting, you know, “random crap” in there, including the summarization of articles I ask ChatGPT to make - I often don’t include those links here, drafts of essays it helps me write, things I’ve found that I haven’t read yet, and whatever else might seem “not polished enough to share.” My idea is that if you really like heaps of info and dumps of stuff (like I do!), you’ll like it. And it’ll save the info-normies the hassle and horror of the info-dump. I make no promises of anything. We’ll see! Check it out for seven days free here.
Relevant to your interests
On graduate student mental health (from my email) - Related to the link from a many days ago about working too much being a self-medication for depression: “Many in the field can be characterized as overachievers. This behavior can easily turn pathological if it is driven by a fear of failure or a sense that self-worth is contingent on competence. Moreover in a competitive academic environment. Exit may be psychologically very difficult if your self-worth is on the line.”
Best printer 2023: just buy this Brother laser printer everyone has, it’s fine - This is a triumph of the review format.
Kubernetes as a platform vs. Kubernetes as an API - This piece is intriguing. It doesn’t got a step further and ask the question: why use Kubernetes then? // Someone needs to write a reply to that original post. While it’s certainly advocating for AWS components (I mean - of course!) there’s an important theory being proposed: you should take Kubernetes at it’s word and just treat it like an API and data standard. Just a “facade” as the poster says. All the stuff under the covers does the hard work and it just has to satisfy the behavior kubernetes wants (is that hill-billy word for “promises”?). The database is a good example: why would you manage your own database when you could just use Kubernetes as a pass through to a managed database? Or load balancer. Or monitoring/logging thing, or event queue. I don’t really know what I’m talking about here.
Logoff
I should be working on other things. Probably. Our old pal ChatGPT helped me write the big part above, based on some rough notes from me. It took several rounds and, you know, you can tell it’s not entirely me. I’d never use a word like “navigating.” But, to my point of being too precious, it’s fine. I have a few more like that stored up that need some work.
I’m speaking at two VMware enterprise architect meetings tomorrow, these are EAs at organizations, not an internal VMware meeting. I’m speaking in Brussels in the morning, driving back to Amsterdam, and speaking there in the afternoon. Pretty nuts!
After that, I’m free until KubeCon EU, so I’m hoping to work on a bunch of content that I can queue up. In my excitement to click publish once I’m done with something, my flow of content can be too bumpy. I’m trying to pace things out more this year, and also schedule more guests on podcast. I’ve been experimenting (as above!) with getting ChatGPT to be a co-writer and editor. I think that’ll help.