Continuous Measurement in Software Development

Making software means changing code, and this is generally an unpredictable affair.

Maybe that new requirement means making a change to code that may or may not have been written with that requirement in mind. Even if it were, maybe the person making the change is new to that part of the code and does not know the simplest way to make that particular change. Or maybe the code itself is brittle and that little change that you made, set off a chain reaction that has you chasing failed tests for hours on end.

It's hard to know what you'll find until you start making changes. Which is why developer estimates suck and we hate to make them. And yet, we are asked for estimates and held to them when we are "behind plan".

The only real constant in software development is variability. So how about we try and acknowledge the fact, and start making software using systems that recognize this? For the past few years, we have been working on this problem at exathink, and one of the most promising ideas we have found is a very simple one.

Visualize how teams change code and measure  progress more frequently.

Why Estimates Exist

We are asked for estimates because people we work with need some idea of how long things will take before we start the work, butwe have very few ways to tell how long things  will take, or even more importantly, they are actually taking without asking engineers incessantly. Once work begins, it goes into an invisible black hole and we need stand-ups, quick "check-ins" and other productivity destroyers that only make what was taking long, longer.

Currently, if we measure software development work at all, it's only for work that has completed, in the form of burn down charts, cumulative flow diagrams etc, based on cards that have been marked closed in a work tracking system. There is no easy way to measure work in progress in software, because much of the "work" happens in the world of code, and it's not something anyone outside the black box of engineering has easy access to.

The Solution

Version control systems have a really precise picture of what is going on when people are changing code, which after all, is what "work in progress" really is. So if we can link the work that we are producing for users,  with the code changes needed to implement them, and measure progress at the commit granularity, then all of a sudden, we can start visualizing progress to folks outside our black box, without them having to read code, hold standups, monitor Slack, or wrangle reports that only tell you what happened in the past, etc.. It also leaves us more time to write code and less time reporting status.

Continuous Measurement is the idea that we measure and visualize progress in software development at the commit granularity, showing things  like lead and cycle time, internal latencies on individual work in progress items due to hand-offs and context switching, and code reviews, WIP costs etc in near "real-time": anytime someone on the team pushes code, updates a pull request or changes the status on a card in the work tracking system.

By seeing where everything in the pipeline is, in real time, at this level of granularity, situational awareness goes up exponentially across the team and we are able to react to changes and adjust scope as needed, with hard accurate data, instead of guesses. It means that folks outside engineering can starting seeing what's going on inside the black box, and we can finally start having a common vocabulary to start talking about the work of making software that does not involve pointing people at code repositories and saying "have at it".

Think of it this way. Our CI/CD pipelines, Code Quality tools, etc are able to ensure that our software stays working and is healthy at the commit granularity. But this part only considers moving finished code to production.  Why should we not be able to say the same about the state of our "plan"?

This part is all about the time it will take people to  change code. If your plans and reality are not at the same place, it's probably best to be able to find out as soon as possible and fix it instead of pretending it's all ok. Respond to change instead of following a plan, and all that...

Check out some of the related work on our website if you want to see how some of this looks like in action.