Every day in New York City, the Department of Transportation records the number of street lights that are reported to have issues. An issue could be that bulbs are out, wiring is exposed, a light is aimed in the wrong direction. And over the past week, there were 1,355 reports of street light related issues.

With over 315,000 street lights in the City, that number of issues is impressively low. That’s probably because New York City takes its street lights seriously. There is actually a city-wide project underway to upgrade all of the street lights to brighter, more reliable, longer lasting LED lights.

And as interesting as that may be, after reading this you may be thinking what does that have to do with software engineering?

Connecting street lights to code

The answer is a lot. Let’s focus on that New York City Department of Transportation metric of 1,355 street light related issues. Tracking that metric is similar to how many software engineering leaders track the number of open code reviews or number of lines of code committed.

Let’s be honest: without context, what’s the relevance of the number of broken street lights in New York City? And similarly, without context what’s the relevance of how many lines of code were committed in your repos last night? In the normal course of operations, those numbers will change every day, which is expected as no two days are the same.

What you should instead care about is the relevance of those metrics in the context of a problem or goal.

Making metrics relevant

For the New York City Department of Transportation, a week of 1,355 street light outages just doesn’t mean much. But if you match the creation and resolution times for each of those reports, you have a measure of responsiveness. Align the creation rate of street light related issues with the pace of the LED replacement project, and you can start to measure the reliability of the new lighting technology.

Do you see my point? Individual metrics on their own have limited value, they are just counts and ratios. But when you align them with a goal or with a problem you are trying to solve – if you’ll excuse the pun – it’s like turning the lights on and getting a glimpse into the shadows of your business.

Please don’t say you’re measuring lines of code committed

It’s no different when it comes to software engineering metrics. The number of open PRs is pretty much worthless on its own. But when you relate it to Cycle Time (essentially the time it takes for code to successfully move through your software development life cycle), it allows you to answer valuable questions. It helps you understand the relationship between code reviews and code throughput. It allows you to see that if you don’t deal with code reviews in a timely manner, you are slowing down the rate you deploy new features to customers.

Same thing with the number of lines of code committed – on its own it’s potentially the worst and least valuable metric in all of software engineering (and I cringe every time an engineering leader tells me they measure it). But take that number and divide it by the time it takes to go through a review or relate it to the number of errors found in production after it is deployed, and you have a way to assess Speed and Quality.

Look for the relationship between metrics

Peter Drucker was right when he said, “you can’t manage what you can’t measure.” The key though is how you measure it and what insights you are trying to receive from that measurement. And maybe more importantly, what problem are you trying to solve?

Let’s go back to the streets of New York City. More than a few years ago, they realized something that eventually made them take street lights as seriously as they do now. Instead of looking at street lights as just equipment, they started to look at the benefits that came from them.

They came to realize that areas of the City that were well lit experienced crime rates that were significantly lower than those that weren’t. Among other findings, the 2019 study concluded that increased levels of lighting led to a 36% reduction in “index crimes,” a subset of serious crimes. And after that study New York City started to maintain a daily log of street light outages.

They weren’t measuring just to measure, they were doing so because measuring helped them solve a problem. And it wasn’t just one metric that was important, it was the relationship between metrics that helped them find solutions.

Most software engineering leaders don’t have these answers

All of this leads to my question to engineering leaders: What problem are you trying to solve by measuring your software engineering operations? And what are the inter-metric relationships that are critical to finding the solutions to those problems?

Don’t feel bad if you don’t have answers to those questions – most engineering leaders don’t – even if they are already measuring right now. Many leaders simply have been taught that measuring is good, so they do it – without rhyme or reason. At one time in my career I was forced to measure software engineering performance and came to realize that measuring and forecasting do work, but only with a strategy.

It’s one of the reasons why I founded Treno: to help other software engineering leaders define and leverage their strategies.

There is no one magic metric

If your software engineering organization is starting to look at metrics and measurements, first take the time to dig deeper into your organizational goals. Look at the issues you are currently facing. Is it your team’s speed, quality, or output, or is it something else altogether?

And instead of trying to find one magic metric that represents your problem, take a step back and look at the business impact you are trying to drive. And finally, work backwards to determine how measurement of your software delivery life cycle will help you see things differently and give you insights that will help you accomplish your goal.

Point the light at business value

I imagine the New York City Department of Transportation didn’t start out by looking at the city’s assault rates and immediately think, “We can solve that problem by fixing street lights!” They likely started by looking at how they could optimize the value their street lights were delivering, like simply “lighting up dark pathways.” And then likely looked at the benefits of well-lit areas and eventually discovered that less crime happened when bright lights are around.

As engineering leaders we can and should do the same when looking to drive value with the services delivered by our teams. Measurements and metrics are valuable tools that help us start on the right track, but the ultimate goal is to drive business value.

Treno helps you leverage metrics to focus on results

At Treno we have hundreds of metrics that we calculate from every engineering transaction we monitor. But we believe the real value is in how our delivery model learns how your team works, suggests which metrics should be important to you, and identifies how those metrics are related.

Drop us a line to learn more about how Treno helps companies make engineering decisions using data. We’d also love to hear about the software engineering problems that you’re using metrics to solve, and especially about the relationships between metrics you’ve discovered while solving those problems. As always, you can reach me directly at [email protected], I look forward to hearing from you.

And whatever you do, keep the lights on!