Most corporate board meetings are a well-choreographed dance between the divisional execs in a company and the members of the board, people who have been there and done that and show up quarterly to share their knowledge. Regardless of the type of company – public, private, enterprise or startup – board meetings are all pretty much the same, and by this time in my career I had been the CTO or VP of Engineering at many companies – large and small, public and private – and I knew the board meeting game well.

Usually there is tons of prep that happens before a board meeting, but in this case, I knew things were going to be a little different. Only a few weeks earlier, I had just joined a startup rocket ship and my sole purpose at the board meeting was simply to listen and learn. I had met most of the board members during the interview process and since I had only been there a short time, there really wasn’t much I could share about the roadmap or operations of the team. So, when it came time for the Engineering Update, my slides were mostly about the size of the organization and general demographics. I figured that was it, and as I was so new in the job I wasn’t expecting any questions – I was sorely mistaken!

How good is your engineering team?

Over the next fifteen minutes, I was told that I was the third CTO in 18 months and none of the previous had been able to meet the board’s expectations. They questioned the quality of the product, asked why it took so long to release new features, and shared – in the most explicit way – their genuine dissatisfaction with the way my 110-person engineering team delivered software. They finished with “For the next board meeting, we want you to tell us: How good is your team? How do you get better? How can you improve both the quality and pace of releases in meaningful ways? And your answer cannot be to ask for more money or more engineers!”

Needless to say, I was left a bit shaken. Primarily because I didn’t know how to go about answering their questions. Aside from some basic burndown charts and anecdotes from my team, I had no real way of understanding and analyzing what was happening in real-time. I knew what the issues were – slow pace and poor quality – but I didn’t know why they were happening.

Our SDLC was a black box

It quickly became evident that what I needed was the equivalent of an Application Program Monitoring (APM) tool, but for my software development life cycle (SDLC). Our team had spent a good deal of money, time and resources in instrumenting our production environment with New Relic. The insight it provided us was amazing. We instantly knew when an application wasn’t performing well, when new features weren’t working as designed, when slowdowns were occurring and we could predict outages. It hit me that we were committed to monitoring and observing our applications in production and we should probably do the same while they were being developed.

Digging in to the data

Monitoring our SDLC would help us understand how we actually build software. Leveraging data would take out all of the opinion that comes from retros and allow our engineering managers to make decisions that actually improved results. So, over the next few months we broke down every part of our development cycle – ideation, design, development, QA, deployment and support. We then identified which parts of our SDLC cycle were dependent on others and we determined how changing one stage would affect (positively or negatively) another. The end goal was to find tangible ways to reduce the number and severity of issues found in production by doing things differently during development. And while it wasn’t a straight line and definitely wasn’t easy, in the end we succeeded.

By the time the next board meeting came around we had increased the number of deployments from once or twice a month to at least once a day. The number of features we released tripled and the number of errors reported by customers decreased by 1/3. And we were able to do this without adding any more resources and while reducing our overall spend. And while the results sound magical, the bottom line was that once we understood how we actually built software we were able to see areas of inefficiency and make changes that optimized our delivery process.

An unexpected effect of discovery

The improvement in our results was incredible, but the most interesting part was how this affected team morale. When we started, I was concerned that the team would feel demoralized when we shared how poor our delivery process actually was. What I didn’t understand was that they already knew it and were frustrated by how their hard work wasn’t paying off. Once we shared the results of our baseline analysis (an examination of what normal delivery looked like for us), the team was nodding their heads in agreement, happy that management had finally realized how bad things were. That led to a very open discussion where everyone agreed that changes needed to be made.

Monitoring processes, not people

Once it was clear that we weren’t looking to blame anyone, everyone felt empowered to make suggestions on how to improve and where to improve. It’s probably important to note that while measuring the process we never measured individuals. I am a strong believer that software engineering is a team sport, and that using metrics as a replacement for good personnel management is a great way to start a team revolt. We also never tried to hide anything from anyone. All of our research, data points, baselines, goals observations and recommendations were visible to the entire company, as were all our successes!

A few weeks later when we came back with the plan to improve the process, the entire team was excited. Every week we posted our improved results vs our baseline in Slack and the number of comments and thumbs up was overwhelming. When we reached (and exceeded) our goals, we took time out to celebrate and later identified new areas where we could get better. Word of our success got out and before too long CTOs from companies large and small were reaching out to learn more about how to improve their software delivery operations.

Why we built Treno

We developed Treno to help software engineering teams improve their delivery process and the products that come out of it. We believe it all starts with enterprises gaining observability into their software development operations. Instead of relying on meetings with teams or spending months tracking down data points, Treno arms software engineering management with real time information that allows them to make data decisions that foster continuous improvement.

Using our platform, our customers can see, in detail, every commit, every merge, every error generated, every review and every deployment. Our model shows how their teams actually work, identifies process improvements that increase productivity levels, and improves the speed and output of their development process and quality of their released software.
We understand that software delivery can easily get off track and with so many teams, tools, and processes it can be tough to figure out why.

Most engineering optimization platforms are simply analytics tools that overwhelm you with complicated metrics, focus on what’s already happened, and don’t give you observability into how your software projects and teams are performing now. At Treno we strive to predict software delivery issues and give you the insight and guidance you need to release faster and deploy more while maintaining quality and optimizing your engineering spend.

Taking the first step to deliver more software, faster

We all know that software engineering is both science and an art and that there is no magic bullet that will instantly improve all of your team’s software delivery shortcomings. But at Treno we believe that observability into your SDLC is the first step to unlocking your team’s engineering potential, optimizing at scale, and improving the value that comes from your team’s product development.

Now you know our story and what led us to build Treno. We would love to hear more about your engineering team and the challenges you’ve faced and how you’ve overcome them. We plan on using this space to ask, and attempt to answer, many of the “unanswerable” questions about complex enterprise software engineering. If you have a story that you want to share or a topic that you’d like for us to address please reach out to me directly at [email protected].