You should automate your builds and tests - 71% of people do not “use continuous integration to automatically build and test my code changes.”
35+ years later, most people are still not automating their builds and testing. Is that possible? A review of the CD Foundation 2024 Survey.
The CD Foundation Survey, 2024
Today’s survey: “State of CI/CD Report 2024: The Evolution of Software Delivery Performance,” CD Foundation and SlashData, April, 2024.
Are people getting better at frequently releasing software and fixing problems in production? The most recent CD Foundation survey says…no:
On average, 29% of respondents say they release software once a week or even more frequently; 40% take a more than month. The numbers here have been pretty stable over the past 4 years. This suggests that (a) improvement has stalled, and/or, (b) this is the normal baseline.
59% of respondents can fix production in a week or less, 40% in a day or less. This has gotten worse over the past 4 years. In 2020, 65% of respondents could fix production in a week or less.
Looking at the past 4 years of these surveys I also get the sense that we haven’t progressed very far in recent years with two of the goals of DevOps: more frequent app releases and fixing problems in production faster. This matches older surveys that show a very slow increase in release frequency, so slow that it might as well have stopped changing.
Does every organization need to release software frequently (weekly to daily)? Probably not, but in general what this survey tells us is not great. Does this means DevOps is a failure? Not at all: if you think these release cycles are long, you should have seen them in the 2000s and the 2010s!
My quick, hunch-made (read: “unscientific”) conclusion is that these numbers are “the new normal.” That means if asked how often enterprises release their apps, I’d say something like “about 50% to 60%1 of release their apps once a month or even daily (maybe 13% to 15% do daily or less?), and the rest take longer than a month, I’d guess most of those are quarterly.”
How frequently do most organizations really need to deploy, especially if you don’t count at-will bug and security fixes and patches? Does my pharmacy need to add new features every day, every week? My bank? The maintenance scheduling app my power company uses to send out trucks? Probably not. Maybe deploying once a month or more is actually perfect for most enterprises. What you’d like to see are survey questions that ask “and are you happy with this?” or “what are you plans, if any, to speed these up?” You’d want to divide these up by individual contributor and management/executives. I’d suspect the first would be like “meh,” and the second would like “fuck yeah!”2
To summarize: “You’re improving only getting slightly worse. Let’s put it like that.”
Deployment to production frequency
The main question for app release frequency is: “On average, how often do you or your team deploy code to production?”
In 2020Q3, 35% were at a month or more. In 2024Q1, it went up to 40% (this is bad, the wrong direction).
The other time periods are mostly stable, but multiple deploys a day went from 12% to 9% over that time range.
You glass half-full this and read it as: 60% of people release their software once a month or even more frequently. That reads pretty good, actually!
If you want to get real detailed, you could pay close attention to the “you or your team” part of the question. This is different than “your entire organizations’ average across all apps.” See further discussion of this point below.
How long does it take to fix problems in production?
Fixing problems in production is measured her by time to restore service (MTTR)
“The proportion of developers who can restore service in less than an hour has remained at around 11% since Q3 2022, though down from 17% in Q3 2020.
However, the proportion of the worst-performing developers – more than one week to restore service – has been steadily increasing and is now the condition for 41% of developers.”
Same point about “you and your team” versus “your entire org” as above.
Build automation and testing are not the norm
I use deployment frequency as the quickest/best metrics for measuring how we’re doing with getting better at software. Like I said above, you may not need to deploy at will. But if you’re doing all The Best Practices and DevOps Things, you should be able to deploy at least once a day if not more.
Can you deploy daily with manual deployments? Sure, we’ve all scp’ed files to production. Is that good? No. It is very not good. You want to automate your deployments, you know: Continuous Deployment. In an enterprise, like a big bank or pharmaceutical company, this is especially important because it’s the only way you’re going to reliably speed up the audit, compliance, and security bottlenecks.
Before CD, though, you need to automate your build and testing, you know: Continuous Integration.3
This has been known and recommended for a long time, 35+ years if I did the math right:
Grady Booch first proposed the term CI in his 1991 method, although he did not advocate integrating several times a day. Extreme programming (XP) adopted the concept of CI [circa 1989] and did advocate integrating more than once per day – perhaps as many as tens of times per day. Wikipedia.
I’d add that Thought Works championed this, as did many other people, and it became a one of the foundational principles of DevOps.
So. How are CI and CD usage doing? NOT GOOD.
Let’s look at people who agreed with the statement “I test my applications for security vulnerabilities” across the last four years of the survey:4
Above you can see people who answered yes/agree to the statement In 2024Q1, 71% of people said they do not “use continuous integration to automatically build and test my code changes.”
I asked Lawrence Hecht (survey analyzer extraordinaire!) what he thought of all this, and he pointed out that the wording of the question is for individuals (“I use continuous integration” instead of “My team uses continuous integration”). It could be that individuals on a team don’t use CI, but that their team does.
For example, let’s say you had a team of 8 developers. On that team, 6 could just write code and (hopefully!) get it into version control. Instead of one of that code commit or one of those six manually triggering an automated build, one of the other two people on their team does it. So, those six do not use CI themselves. That’d be a pretty nuanced way to answer the question, but programmers are often very nuanced in their answers.5
For more on CI and CD usage, let’s look at a different survey that goes back further, the State of Agile survey.6 They stopped asking about tools usage in 2022, but we have surveys that go back to 2007:
These numbers are (much) better, but there’s a problem with the results from both surveys: improvement is flat. Things haven’t gotten better or worse in a ~10% band.
So much ROI in your future
So, something like 47% to 71% of people don’t do CI. I’m not sure what to do with that. I hesitate to even think it’s real.
¯\_(ツ)_/¯
But, let’s say this analysis is right.
What it means is that:
Need something to boost your annual bonus? If you don’t have CI in place, it should be easy for enterprises to get better at software by simply automating builds and testing. If you don’t have CI in place (never mind CD!), set aside everything else you’re doing and put CI in place. Anything new you’re doing in apps (cloud native, DevOps, Kubernetes, etc.) is going to fail (or at least fail to achieve ROI) if you don’t have CI.
If you can get people to pay for CI/CD, you’ve got a market goin’! Selling these tools is difficult because you’re selling to a group that thinks they can do it on their own and have small budgets - you’ve got to go in with a freemium/cheap option and at some point, you add in RBAC, and you can sell big, company wide deals.
Other Details
This report is done by Slashdata. They have huge data sets and do many of the reports for the CNCF. If you dig around on their site you can find PDFs of their general survey, Developer Nation.
Demographics aren’t covered very well. I can’t tell what percentage is tech companies versus “normal,” company size, or the split between management and staff in respondents. But:
“We survey 30,000+ developers annually.” And: “The report is based on a
large-scale, online developer survey designed, produced, and carried out by
SlashData over a period of ten weeks between November 2023 and February 2024… from 136.”
Everyone is doing the DevOps: “As of Q1 2024, 83% of developers are involved in DevOps-related activities.” // There’s some variation by industry and org. size, but it doesn’t really matter. People think of themselves as doing DevOps, and many of the practices are likely practices. DevOps Thought Leading wins!
BUT! If you look at how they determine this, they ask what DevOps practices people follow. There are 9 practices, some of which are just general programming practices, e.g., using CI, automating testing. This is different than asking “do you do the DevOps”? I don’t really like this method, so I wouldn't draw any conclusion about how many people are “doing the DevOps.” Maybe how many people are using DevOps practices if narrowed down from those nine.
It’s probably better to use fewer (if not just one) CI/CD tools than more: “using multiple managed CI/CD platforms has minimal impact on improving deployment frequency, but is much more likely to lead to an increase in low performance.” More: “One possible reason for this may be that lead time for code changes is impacted not just by platform usage, but also by their CI and development processes. Multiple managed CI/CD platforms may introduce fragmentation of the CI process, leading to greater negative impacts. Similarly, development practices like code review, collaboration, and testing may be impacted by having to adapt to multiple platforms throughout the workflow, and this challenge has a larger impact on lead-time performance.” I’m all for centralizing and standardizing tools/platforms/stack, so, sure, I like that one.
I think the PDF is saying that if you use hosted (public cloud) CI/CD tools, things are better.
There’s an attempt (which I’m really interested in) to see if running your tool stack on public cloud versus on your own is better. The answer is that doing both (“multi-cloud”!) is the best. I don’t think this tells us enough.
//
If you liked this survey review, check out the two recent ones on platform engineering (the Perforce Puppet one and the one from Port, also, a bunch of other ones from over the years).
🪵 Logoff
I’ll put some links and waste book stuff into the next episode. In the meantime:
Yesterday, I had the wrong link for the “Things I Like” post I mentioned: here is the correct link.
On this week’s Software Defined Talk, I interview Jana Werner about some digital transformation cards she and her AWS friends have been using when they work with executives. It’ll post this Friday (April 26th at 7am Amsterdam time. It’s good stuff! You can find it in our YouTube channel or by subscribing to the podcast.
Here’s my case for buying your cloud native platform instead of building it. I’ll probably regurgitate this in a future newsletter as well.
The survey doesn’t actually breakout “monthly” as a time period - it has “more than a month” and “one week to one month” - so I’d put it as a range like this.
In general, I don’t think individual staff are much motivated to improve organizational productivity. They generally get paid the same either way. Of course, they don’t want their work to be boring, tedious, or seem worthless, sure. But this is different than the business-driven goals of management to increase productivity.
I guess I should “prove” this somehow, but I think it’s just accepted “theory” like gravity is accepted “theory” or that the Earth is “round.”
There’s two for 2023 - Q3 snuck into the 2024 report, while Q1 was used in the 2023 report.
This is an example of very important survey nuance: you have to be careful about questions that ask “does your organization…” versus “does your app/team…” and, in this survey, “do you…” For example, for the first you could ask “does your organization use Generative AI?” And if there’s just one person in a 300,000 person company using it, the answer could be “yes.” That “yes,” of course, is much different than if 40,000 people are using AI, or if the top 3 products (by revenue, marketshare, units sold, etc.) in that company rely on AI.
There are other surveys about CI/CD usage and release frequency. Check out my collection of them from last year, in particular the ones from Forrester (which I’d rate as high in relevance/accuracy for enterprises, that is, NOT tech companies).