Google Cloud’s DevOps Research and Assessment team (DORA) has spent the last eight years surveying more than 33,000 professionals from around the world. It's the largest and longest-running academically rigorous research investigations of its kind. In my opinion it's one of the most influential pieces of literature in software development today.
For the first time software developers have been given a body of research to help move software development away from a craft, with anecdotal and subjective values, and grow it into a true engineering discipline of evidence based best practice.
In this article I'm going to breakdown a software development process in alignment with DORA's findings.
TLDR: focus on DORA's 4 key metrics.
Index:
- History
- Why the Software Development Process Matters
- The Culture We Work in
- How to Choose What to Build
- How to Build it
- How to Deploy it
- Four Key Metrics
- So what about Agile?
- Conclusion
Let's Start with some History
In 2011 Alanna Brown at Puppet had the idea of creating a survey and report to track the state of the growing DevOps movement and assess its impact. Alanna, Nigel Kersten and James Turnbull began working on The State Of DevOps report that very year.
In 2012 they were joined by Gene Kim the ground breaking author of the novel on IT and DevOps The Phoenix Project and Jez Humble the author of Continuous Delivery. They went on to release the first developer survey which was completed by 4,000 technical professionals, a response that was unheard of at the time.
In 2013 the first State of DevOps report was published and received massive interest. It was in that same year that the authors met Dr. Nicole Forsgren who joined the team and established the academic rigor for which the State of DevOps report is now renowned.
Nicole, Jez and Gene went on to form DORA (DevOps Research and Assessment) and wrote the now famous book Accelerate: The Science of Lean Software and DevOps based on the data they gathered.
In 2018 DORA was acquired by Google and their research team became part of the Google Cloud. Dr Forsgren continued to the lead the DORA research program until 2020 when she left Google to join GitHub as Vice President of Research and Strategy.
Why the Software Development Process Matters
DORA's research shows us how companies that get their software development process right are twice as likely to hit both their commercial and non commercial goals in terms of:
- Profitability
- Productivity
- Market share
- Number of customers
- Quantity of products or services
- Operating efficiency
- Customer satisfaction
- Quality of products or services provided
- Achieving organization or mission goals
These Organisational Performance measurements have been validated and found to be highly correlated to measures of return on investment (ROI), and they are robust to economic cycles.
To put it simply, companies do measurably better when they get software development right.
DORA identified 3 key software development capabilities that drive these organisational performance measurements:
- Lean Product Development
- Software Delivery Performance
- Culture and Work Environment
Let's talk about how this all comes together.
The Culture We Work in
Culture is arguably the single most important part of the software development process. DORA's research shows us that companies do better, in terms of both software delivery and organisational performance, if they can foster a culture that is high in trust and that emphasizes good information flow.
This isn't a new idea and it's based strongly on Dr. Ron Westrum's research into the human factors in system safety.
This model gives us a way to measure and classify organisations based on their culture and DORA were able to prove that this classification was predictive of organisational performance.
DORA went further to identify 5 key capabilities and outcomes that identify good culture:
- Climate for Learning. An organization with a climate for learning views learning as an investment that is needed for growth, not as a necessary evil, undertaken only when required.
- Westrum Organizational Culture. This model of organizational culture was developed by sociologist Dr Ron Westrum. It classifies organizations as pathological, bureaucratic, or generative based on levels of cooperation, how problems are surfaced, the extent to which the organization is siloed, and how people react to failure and novelty.
- Culture of Psychological Safety. In teams with a culture of psychological safety, team members trust each other, are able to resolve conflict, take calculated and moderate risks, speak up, and are more creative.
- Job Satisfaction. People feel supported by their employers, have the tools and resources to do their work, and feel their judgement is valued.
- Identity. Employees identify with the organization they work for. They say that the organization is a good place to work. They feel that the organization cares about them. And they’re willing to put in extra effort to help the organization succeed.
What is so interesting about DORA's research is that they were able to show that companies can drive better culture by making technical process changes. By implementing things like continuous delivery, lean product development and lean management practices companies are able to improve their culture in measurable ways.
Ultimately good culture relies on effective leadership. By enabling the adoption of technical and product management capabilities and practices our leaders are able to drive software delivery performance which in turn drives the outcomes leaders care about.
If companies are looking to improve how they build software, this is where they should start.
How to Choose What to Build
Selecting which features or products to build next is unfortunately more of an art form than a science. The perfect next thing can be found somewhere in a delicate balance of your long term strategy, your mid term organisational goals/bets and your iron triangle constraints of cost, scope and time.
Unfortunately there's no easy way to replace a good Product manager.
What DORA's research does tell us is that companies that follow Lean Product Development practices have a statistically significant better chance of not only scoring higher in terms of Organisational Performance (defined above) but also in terms of software delivery performance and culture.
DORA was able to narrow down Lean Product Development to 4 key capabilities:
- Working in small batches. The extent to which teams break products and features into small batches that can be completed in less than a week and released frequently, including the use of MVPs (minimum viable products).
- Visibility of work in the value stream. Whether teams understand the workflow from business to customers, and whether they have visibility into this flow, including the status of products and features.
- Customer feedback. Whether organizations actively and regularly seek customer feedback and incorporate it into their product design.
- Team experimentation. Whether development teams have the authority to create and change specifications as part of the development process without requiring approval.
When we choose what to build next we should try and break things up into small steps that we can show our customers and validate our choices before we get too invested.
These small features that are built and deployed in order to get customer feedback are often referred to as vertical slices.
This approach isn't always easy to follow for some products and it can require us to get pretty creative. If you're in a bind, building prototypes is a great way of gathering feedback on products or features that are difficult to break into working pieces.
The take home message here is that we shouldn't make too many assumptions about what our customers want and that we should seek out and be open to their feedback as often as possible. How that happens is up to you.
How to Build it
The process of designing, developing and validating a product or feature, otherwise known as the "fuzzy front end of innovation", is difficult to optimise as it varies so much from problem to problem.
So, while DORA's work doesn't tell us how to do every part of the software development process it does give us some things that we should focus on.
Architecture
DORA's research shows us the importance of independent and cross functional teams.
They shouldn't rely on fine-grained outside communication or coordination. They should be empowered to make sweeping design changes to their systems without outside permission and they should be able to deploy and test independently of any other services that depends on them.
These values highlight the importance of a loosely coupled architecture, one that's been built with continuous delivery in mind.
It doesn't matter if you use microservices or monoliths, or if you're working with mainframes or serverless functions. The important thing is that your system is easy to change, easy to maintain and easy to monitor.
Security
In 2021 more than 22 billion records were exposed due to data breaches, this combined with the worlds growing awareness of privacy concerns makes security one of our top focuses when building software. We need to keep our customer data safe and we need to keep our business up and running.
Traditionally however security is a consideration that's left till the end of the software development process. Once your feature has been built and tested it would then go for a security audit, if one is done at all. Leaving security till this late stage means it will always be treated as an afterthought. At best it will result in expensive rework to correct any issues that are found. At worst it will cause a large number of missed security vulnerabilities.
DORA's research shows us the value in shifting left on security and making InfoSec a part of our daily development process. By doing this we can avoid costly rework, improve the security of our applications and ensure that security doesn't slow down the development process.
DORA gives us 3 key areas to help us improve:
- Make security part of the review process: every good software development process includes peer review of some kind, either as part of the process in pair programming or in a formal code review step. Security should be a first class citizen in this process.
- Include security in automated testing: automated tests that cover a large percentage of the security requirements should be developed and run as part of the automated tests suites.
- Use pre-approved libraries and tool chains: by making use of audited and pre-approved packages the likelihood of introducing new security issues in common areas decreases.
Security is it's own skill and the importance of a highly activated and sufficiently staffed InfoSec team can't be understated. This team should be involved in review, training, creating automated tests and the development of common tools and libraries.
Continuous Testing
Hopefully this doesn't come as a surprise to anyone but manual testing is full of problems. It's time consuming, expensive and it creates a bottleneck in the process. What's worse is that it's unreliable, people are just bad at performing repetitive tasks like doing the same set of tests again and again.
DORA shows us the value in curating and building up a comprehensive test suite throughout the development process. This is especially powerful when run automatically as part of your continuous delivery pipelines.
The goal here is to find as many bugs as possible early on in the development process when fixing them is as cheap as possible.
Note that DORA doesn't specify a specific test methodology like TDD or BDD. Rather what's important here is that meaningful and reliable automated tests are created as part of the development process and they are automatically run as part of the delivery pipeline.
Trunk Based Development
This is a controversial one as many developers have grown comfortable with feature branches and pull requests as modelled by patterns like Git Flow. However I think we can all accept that using feature branches has its problems.
It's very easy to fall into the trap of having long lived feature branches which can lead to both difficult mergers and difficult code reviews.
DORA's research shows us that teams that are able to effectively address these two problems perform better.
Trunk based development that incorporates pair/mob programming is the current state of the art for solving these problems. This style of development uses feature flags to merge and deploy to main/trunk extremely frequently and it has code review built into the process which removes any potential bottlenecks.
Adopting trunk based development can take some time and learning but it isn't the only way to solve the problem. The point here is to try and merge frequently and code review effectively so that it doesn't hold up releases. Using short lived feature branches with small batches of work and adopting a non blocking code review style can get you pretty far in terms of solving these issues.
How to Deploy it
DORA's research shows that implementing continuous delivery into your development process is one of the key drivers for higher software delivery performance as well as better organisational culture.
Continuous delivery relies on a large number of technical capabilities and I highly recommend going over Google's complete guide to continuous delivery for a full over view of the topic.
The core goal here is to get software development teams to the point where they can deploy on demand and where they have fast and easy access to feedback on the quality of the deployment and the system as a whole. Acting on this feedback should be the team's main priority.
In practice this requires teams to actively reduce the amount of time it takes to get committed code into production and to reduce the amount of time it takes you to restore code to a working state once a problem has been identified.
This process relies on 4 key capabilities:
- Test automation: the use of automated testing suites that reliably identify real failures and will only pass on releasable code.
- Continuous integration (CI): A development practice where code is regularly checked in, and each check-in triggers a set of quick tests to discover regressions, which developers fix immediately. The CI process creates canonical builds and packages that are ultimately deployed and released.
- Deployment automation: The degree to which deployments are fully automated.
- Comprehensive monitoring and observability: Allows teams to understand the health of their systems. Effective solutions enable teams to monitor predefined metrics, including system state as experienced by users, as well as allowing engineers to interactively debug systems and explore properties and patterns as they emerge.
Four Key Metrics
This might all seem a bit overwhelming depending on where you are in the process. In order to help simplify this DORA have managed to identify 4 key metrics that indicate the performance of a software development team.
These are broken into two parts:
Velocity:
- Deployment Frequency—How often an organization successfully releases to production
- Lead Time for Changes—The amount of time it takes a commit to get into production
Stability:
- Change Failure Rate—The percentage of deployments causing a failure in production
- Time to Restore Service—How long it takes an organization to recover from a failure in production
By measuring these values and by continuously iterating and trying to improve on them teams can achieve significantly better business outcomes.
If you do one thing, then this should be it.
So what about Agile?
Agile and The Agile Manifesto has seen huge adoption across the software development industry. It's given us a standard set of values and principals for how the software development process should be run and its served many organisations incredibly well.
Today however Agile, as a term, has been eroded. Many frameworks and ways of working were established and marketed off the back of Agile's fame and today the word has lost much of its original meaning.
The Agile Manifesto in it's simplicity has fallen victim to interpretation. Agile was never about Scrum or story points and backlogs. It was never about certifications and frameworks. It was established as a set of values to guide us in our work.
This is where I think DORA's research makes its largest contribution. Not only do it give us a set of objective best practices to follow it also gives us concrete ways to measure them.
Does this mean that DORA's research contradicts Agile? Well in my view, if you're:
- Following Lean Product Development working in small batches and continuously delivering valuable software
- Welcoming Customer feedback
- Implementing Continuous Delivery and deploying frequently
- Working closely with business people to ensure that developers have good Visibility of work in the value stream
- Working in a culture of high trust and empowerment as defined by generative organisations in Westrum Organizational Culture
- Prioritising good and effective information flow in terms of Westrum Organizational Culture
- Placing the most value in working software via Continuous Delivery and Continuous Testing
- Promoting sustainable software development practices through good culture which reduces burn out
- Prioritising good architecture design and ensuring your systems are easy to change
- Using customer feedback to ensure that you're not building stuff that they don't value in terms of Lean Product Development.
- Implementing self organising and independent teams
- Regularly iterate and try to improve based on the 4 key metrics
then you're Agile!
Conclusion
Software development has advanced hugely over the last decade. More and more our practices and ways of working are moving beyond anecdotal accounts of success and moving towards a real research based discipline.
Today we can answer the question of how things should be done from a place of authority and confidence. For that I'm extremely grateful to Nicole Forsgren, Jez Humble and Gene Kim.
Maybe the 17 people who met at Snowbird in 2001 were on to something...
Photo by Kristopher Roller on Unsplash
