While helping a range of different organizations with their DevOps transformations, I’ve ran into a challenge explaining what does the adoption of a true DevOps culture means to the whole organization. It was challenging to articulate why, when the transformation applied only to a portion of the organization, will not only yield slim results, but, in many cases, may bring more harm to the organization than good. Finally I’ve came up with a model which can explain the dynamics of such transformations as well as to expose the failures of incorrect transformations. This model heavily relies on the Value Chain Mapping technique evangelized by Simon Wardley.
The value chain, I’m using, is a depiction of an abstract organization/company containing all of the major necessary stages needed to make a software product and to deliver it to the end customers. You’re welcome to adjust the value chain to reflect your own organization, but, the model will still preserve all the major chain elements and will have the same behavior.
In this article we’re talking about the transformation model for the companies that ARE looking for agility and adoption of the DevOps culture. This transformation is not for everyone. Stop here if the organizational agility is not applicable to your company. Continue reading if you want to understand the key moving forces and to devise methods achieving your organizational transformation success.
Before we proceed forward I’d like to address the “elephant in the room” – the direction of the Evolution axis from “traditional” to “next generation” and from “rigidity” to “agility”. When I talk about “agility”, I’m not talking about the “Agile brand” or “Agile practice”, but about the speed of the reaction to the unexpected changes and unpredictable needs.
For different organizations the direction and the destination of their evolution and improvements would be different due to the specifics of their business models (ex: manufacturing vs software). In this article we’ll be talking about the software companies that have aspirations to optimize for agility, i.e. increase the speed of their reactions to the market and customer demands. So the map’s horizontal axis depicts the range of “agility” from the far left, representing a very rigid form of a waterfall-type organizations, to the far right where you can imagine a type of an organization that responds instantly to any type of change the world can throw at it.
I want to be very clear, when we’re talking about “agility”, we’re not talking about the “chaos”, which, for many organisations became synonymous with the failed Agile implementations.
The horizontal axis shows the journey from the “rigid low autonomy” to the “high alignment with high autonomy” culture. On the slide below, taken from Spotify “culture deck”, I’ve shown the direction of evolution with the red arrow, where the top left represents a rigid top-down waterfall-type organisation and the top right depicts a desired state of the “true agility”.
On the Value Chain map the horizontal position of a stage (circle) depicts “current culture” of this element as it perceived by an outside observer. Be aware that the “culture” is a manifestation of a combination of different factors, such as “results”, “actions”, “beliefs”, “experiences”, “norms”, “expectations”, “incentives” and others. These factors are the “anchors” of the culture. These anchors can be shifted towards the right side by applying both, the global “enabling” leadership force and the local leadership forces, as we’ve talked in the previous R44E blog article “Leadership is a hidden ingredient in successful DevOps stories“. By using only local forces you’re going to see only localized improvements. When an organization replaces leadership with the top-down control, the nodes tend shift to the left side.
Each level of this model represents a different “function” of an organization and is responsible for different aspects of the “flow“. The flow can cover things such as information, knowledge, requirements, risks, finances and materials (Position, Flow and Movement by Simon Wardley). Agile organizations are recognizing and catering to the flow. They are relentlessly looking for the bottlenecks and removing them one by one (Theory of Constraints by Eliyahu M. Goldratt). They are aware that it is required to analyze and apply different approaches, techniques and tools to the different elements of the flow – one way of doing things across the whole organization is not applicable in most cases. These organizations are religious about shortening communication paths and creating faster and tighter feedback loops to shorten the flow i.e. reducing the end-to-end time of delivering value to the end customers.
One of the main techniques of shortening communication paths and increasing agility of the whole organization is distribution of the contextual knowledge to each and every level of the flow and continuously providing up-to-date aggregated information up the chain. Empowering localized decision-making at the right place and at the right time removes the paralysis of the centralized decision-making inertia. This can be achieved only in the organizations that are building trust across all of the levels of the organization. Such organizations trust employees to make informed decisions about the way they work and what they work on.
The “Split Brain” model
By applying localized changes (in silos) an organization can shift a single flow element to either direction. Example: applying agile practice only in the development team shifts the “execution” stage to the right. This increases the length of the communication channels between the Portfolio Management and the Execution and between the Development and the Operations stages. In essence these localized improvements foster differences in “modalities” and create an “impedance mismatch” between the levels – these levels are starting to talk in different languages which causes a lot of friction. Example: Portfolio Management asks for a long-term plan but the Execution talks about short iterations, or Developers want to deploy continuously, but Operations are geared only for quarterly deploys with manual approvals and run-books, etc…
Note the difference between the “blue” and the “orange” channels’ lengths (Figure #3 below) between the Portfolio Management function and the Execution stages. The longer the length of the channel the stronger the force that pulls the outliers back “into the fold”. It works like a “spring” according to Hoke’s law – the force F needed to extend or compress a spring by some distance X is proportional to that distance. This model explains why applying DevOps locally within a specific team or a function, creates unwelcome tensions within the organizations and doesn’t increase the overall value flow.
When an organization wants to become innovative, competitive and responsive to the market demands it needs to shift all it’s layers to the right side.
Q: What if an organization has only one product that is in a maintenance mode or doesn’t need a fast innovation?
A: In this case we observe organizations align elements of their flows towards the left side of the model – more traditional slow-paced controlled processes.
Beware! This answer doesn’t work when an organization has more than one product or service. It is very important to note that in this case we’re talking about a product/service portfolio and not about a single product or a service. It is fundamental to understand that the different products, as well as services, that a company provides to the end customers have their own value chains. It is critical to recognize that usually products and services have different evolution/maturity states. Different stages of their life-cycle require different approaches and practices, different skills of people and different types of organizational structures. More common than not “One size fits all” approach is not applicable and, in many cases, is harmful. When an organizations trying to optimize functions and treat everything equally, it creates implicit dependencies between value chains, starting to break the flows which want to shift to the right side.
Note: this situation is a classical example of what is happening after acquisition of a start-up (more agile) by a larger (more traditional) organizations, when their flows are being forcefully merged.
To explain this, let’s see a hypothetical example of an organization that has 2 different products or 2 different delivery approaches. We’ve usually seen this situation happening in the organizations that have an old “legacy” product, which has been decided to be rewritten or, similarly, an alternative service or a product which was envisioned to capture a new business or a new market. The “legacy” (blue flow) product, in our example, is in sustaining and maintenance mode and is deployed in a company’s traditional data center. The “new” product is envisioned as SaaS offering and has to be deployed on a public cloud.
Such organization immediately runs into the described above problem of a “split brain” with different modalities. The Figure #3 (above) visualizes a couple of problems we’ve observed over the years in a range of different organizations trying to capture new markets or attempting to adopt new technologies this way. Let’s discuss them one by one.
In our example, the newly formed R&D team adopts the DevOps culture and tries to “own” the full life-cycle of the new SaaS product. They usually pick a new modern technology stack and micro-services architecture. They are ready to own their deployment processes and eager to answer support calls at night. They are geared towards incremental deliveries, applying continuous delivery practices and use public cloud hosting. A business PO is excited to review these new changes as they become ready. All of the above shows that almost immediately this new team leapfrogged the rest of the organization in “agility”. Most of such cases turn out to be more problematic than an organization can expect.
When an organization contracts such “split personality”, problems, similar to these following ones, arise:
The development teams are starting to have a conflicts with the “portfolio management” function of the organisation. Usually questions such as “when you’ll be done?” can’t be properly answered.
The DoD (“Definition of Done”) changes but a traditional QA organisation starts to demand the test plans against the final scope of the product: “tell me what is not implemented so we’ll not test it”.
Ops organisations are demanding run-books for the final implementation, capacity and budget planning, deployment schedules, etc… These organizations are aggressively pushing back on different ways of doing deployments and operations. Traditional arguments arise like a “lack of training” and “we need X months/years to plan and certify new ways of working”.
The development organisation itself starts to experience “split brain” and starts to feel complexity between the need to maintain the old, usually monolithic, code base and the new one. Such organizations start to chat about “technical debt” and about massive rewrites of the “legacy” products.
The “knee jerk” reaction in similar situations is to pull the outliers back “into the fold”. But the better way to resolve them is to stop fighting the inevitable and to recognize that it is a normal way of evolution and it is needed to be fostered and supported and not to be fought with. By doing local cost and complexity optimizations (usually in a form of “do it the old way” until we “productize the new way”) such organizations are missing their opportunity to rapidly evolve and to discover new ways of doing things. The best way to move forward is to apply PST approach (Pioneers, Settlers and Town Planners by Simon Wardley) and to recognize, in fact, that it is totally normal to have multiple ways of doing things. It is totally normal to hire different types of talent for this opposing modalities – traditional “by-the-book” types and the new types with the new required skills to support rapid discovery and experimentation in tight rapid iterations.
The DevOps “Chasm”
If you’ve ever participated in a discussion about adopting DevOps culture, a name one of the leading agile organizations (ex: Amazon, Google, Netflix, etc…) is usually mentioned as an example of “better/best practices” of tight and efficient flows, uber-optimized for agility, allowing these companies to eat new markets every day for breakfast. The push back usually is a variation of: “that won’t work here”, “they have a different products/markets/users/use cases” or “they can afford to break their products but we will loose all/lots of our customers if we ever fail once”. This is happening due to a large gap in the divide between the current flows of your organizations and the flows of the leaders.
This gap has formed due to a whole range of different reasons. Most likely these leaders have never had a “legacy baggage” or they have been formed and founded on “day one” on the right side of the journey to agility. This gap, in many cases, is very large and most often causes local leaders to recognize the need to close it by applying tremendous heroism to push and pull “kicking and screaming” organizations towards improvements. They are faced with inertia and push back from their organizations up and down the flow chains. Sooner or later these leaders burn out (Karōjisatsu
by John Willis) and give up because their actions dissipate like a “drop in a bucket”.
I call this divide the “DevOps chasm” (Figure #4). This chasm is very wide and it is very hard to cross over, but, it is very measurable in terms of captured or lost revenue. Earlier a company gets a product to the market, more customers it acquires and more revenue it captures. It has been estimated that a company looses 2%-6% revenue for each month it is late to the market – lost opportunity cost.
Exploration and Exploitation
It makes sense to mention that the journey to “market capture” consist of 2 distinct parts as Clayton M. Christensen in The Innovator’s Dilemma writes – Exploration and Exploitation.
Exploring new opportunities and exploiting existing ones are fundamentally different strategies requiring different organizational structures, competencies, processes, and mindsets. This means that it doesn’t make sense to argue about what modality is better, the “exploration” or “exploitation”, but how fast a company can switch between these modalities. It is important to understand how to keep the proper balance between these modalities (Lean PMO: Explore vs. Exploit great post by Barry O’Reilly) and how these domains require different management and leadership practices as it described in the Lean Enterprise book ( Lean Enterprise: How High Performance Organization Innovation At Scale by Jez Humble, Joanne Molesky and Barry O’Reilly).
I believe, we have to stop arguing about what type of an organization is better and to start having a discussion of what type of an organization you want to have – a static and a rigid one or the one that can quickly change modalities and to react instantaneously to the changes around. It is all about choosing the proper trade-offs. It is very rare when an organization can afford to have 2 distinct sub-organizations and 2 different modes of operation, so naturally any organization wants to “collapse” these 2 modalities into one. To pull it off properly an organization has to choose the more agile modality which will enable it to learn and adopt faster. So agility is not only about how fast you can iterate on the product, but, also, how fast you can react and switch focus.
Info-graphics of the Journey
There is much more to this model and we’re going to talk about different parts of it in the future posts, but, meanwhile, I’d like to leave you with the info-graphics that captures a lot of what we’ve discussed so far.
I’m very interested in hearing your thoughts and comments.
References used to build Info-graphics:
- Value chain was influenced by Agile adoption patterns post by Richard Durnall (2009).
- Talk from GOTO conference Why Agile doesn’t scale and what can you do about it by Dan North (2014).
- Modified comparison table was presented at Leading Edge Forum by Simon Wardley (2012) and referenced in The Next Generation post.