Understanding how a software engineering team builds and deploys software is the key to understanding how that same team can improve performance and quality, optimize resources, and foster a culture that promotes developer productivity.
There are numerous reasons for a company to monitor, track, and measure engineering. These reasons include:
Assisting engineering leadership in understanding team performance
Tracking performance over time to aid in strategic planning
Identifying areas that require improvement, such as quality or speed.
Optimizing the development environment to enhance developer effectiveness.
Estimating project costs.
Facilitating hiring decisions, ensuring sufficient capacity to meet demand.
Forecasting project delivery dates.
Measuring the impact of software engineering on overall business performance.
Improving the overall development process and culture.
Why is Measuring Software Engineering Performance Difficult?
While the reasons to measure are clear, measuring software engineering performance remains one of the most significant challenges in the corporate enterprises, and there are a number of reasons for that. Even in best-case scenarios, where there is clear communication between teams and where managers have access to proper tooling, there are so many different aspects of software delivery that can be monitored.
Without a strategy measurement efforts will surely fall. Many organizations fail because they try to find a single metric that can give a big-picture view of performance. Additionally they may over-index on high-level metrics frameworks such as DORA or SPACE without understanding how to best modify or account for the nuances that are unique to their teams, or processes.
There are several additional factors that contribute to the difficulty of measuring software engineering performance:
Complexity of the SDLC: The software development life cycle involves numerous people, processes, and tools, making it challenging to capture and assess performance accurately.
Multiple Teams and Diverse Processes: In many cases, multiple teams collaborate on the same projects, and each team may have its own unique set of tools, methodologies, and processes. This diversity complicates the measurement of performance across teams and requires a comprehensive approach to capture the complete picture.
Varied Stakeholder Requirements: Different stakeholders, such as project managers, team leads, and executives, often require specific reporting and insights tailored to their needs. Accommodating these diverse requirements while maintaining a consistent measurement framework adds another layer of complexity.
Multiple Levels of Measurement: It is crucial to assess performance at various levels, including the project, team, portfolio, and organization levels. Each level provides valuable insights and contributes to a holistic understanding of overall performance.
Diverse Metrics Across Tools and Teams: Every tool and team may have its own set of metrics to evaluate performance. This variation makes it challenging for managers to determine which metrics are most relevant and effective in assessing performance accurately and consistently across the organization.
Selecting the right metrics is difficult and often tool dependent
The challenge of measuring software engineering performance is further exemplified by the variety of widely used tools in the industry, including Jira, GitHub, and Datadog. Each of these tools provides distinct data that can be leveraged for performance assessment.
Jira, for instance, focuses on process methodologies like Scrum, employing metrics such as velocity and throughput to gauge performance. GitHub, on the other hand, offers counts such as the number of commits, pull requests, and closed issues as indicators of progress. Meanwhile, Datadog provides insights into application performance, including the number of errors, warnings, and exceptions.
With such diverse metrics available, it becomes exceedingly challenging to determine which ones offer the most accurate representation of the software engineering team’s performance or provide an accurate report on project status. Even if a specific set of metrics is chosen, the task of combining them into a comprehensive metric that offers a holistic overview of team performance, which can then be effectively communicated to executives, remains daunting.
Another challenge arises from the fact that several metrics used in software engineering performance measurement are primarily applicable to historical projects and not in-flight projects. For instance, metrics like velocity and throughput provide valuable insights into completed projects but are not as effective in gauging the progress of projects currently in motion.
This hampers a managers’ ability to make data-driven decisions during the course of ongoing projects. Without real-time insights, they may face difficulties in identifying potential issues, making timely adjustments, and mitigating risks. As a result, missed deadlines and costly failures become more likely due to the absence of up-to-date performance data to inform decision-making.
Output-based metrics vs. Outcome-based metrics
It is also difficult to understand the type of metrics on which to focus. The most logical path, without a strategy, is to focus on output-based metrics over outcome-based metrics. This is because output-based metrics are usually easier to collect – but they also can be misleading.
They measure activity, but that only tells part of the story. For example, a team might be able to produce a lot of code in a short amount of time (output), but this doesn’t necessarily mean the team is performing well, building the right things and that the code is delivering value to the end user (outcome).
Output is primarily a measure of activity. Activity is important, as a lack of activity may indicate a process bottleneck or an issue in capacity. But what good is a lot of activity that produces no quality results? That said, there are also issues with outcome metrics. They do measure results, but they give you no insight into what it took to obtain those results. For example, an outcome measurement may focus on the usage of a particular feature, but it tells you nothing about the effort or cost or amount of resources that were required to get there.
As both have different purposes and both can relay important signals – it is important to measure output and outcome.
Correlating Metrics is the Key to Quality Insights
Correlating metrics is essential for gaining valuable insights into software development. Failing to establish a strategy that accounts for the relationship between metrics usually results in several significant issues. Included in these are an incomplete view of performance and a failure to recognize the impact of different aspects of the development process.
For instance, an organization that solely focuses on the number of commits made by a team might perceive high commit numbers as an indication of good performance. However, if those commits are riddled with bugs, their quantity does not reflect actual performance or output quality. Similarly, a manager who prioritizes speedy delivery might perceive the team as being too slow without realizing that a lengthy QA process is the underlying cause, hindering the team’s ability to introduce new features. Alternatively, the manager might be satisfied with the rate at which issues are closed but fail to acknowledge that the team is spending a significant amount of time fixing bugs, impacting their capacity to deliver new features.
Success with measuring engineering performance starts with a strategy.
A Measurement Strategy Starts With “Why?”
Measuring the performance of software engineering is undoubtedly challenging, but it is crucial to emphasize that it is indeed possible. Once implemented, the measurement and monitoring of the software development lifecycle (SDLC) offer substantial benefits. To fully reap these benefits, a comprehensive strategy should be established. This strategy entails the following steps:
Identifying the stakeholders and their reporting requirements.
Documenting the business goals for the software engineering organizations.
Determining the key reasons for measuring and monitoring the SDLC.
Identifying the entities to be measured, including teams and projects.
- Assessing the available data that can be collected and utilized to create metrics and key performance indicators (KPIs).
- Implementing internal procedures for reviewing insights derived from metrics.
- Defining the ideal state and determining what success looks like.
It is worth noting that none of the above steps involve selecting metrics from a pre-defined list. Instead, the choice of metrics should be driven by the established strategy. Therefore, organizations should refrain from starting with specific frameworks like DORA or SPACE. The starting point should always be understanding the purpose of measurement and what the organization aims to achieve. The selection of individual metrics should emerge from these considerations. Ultimately, the goal should not be to adopt a framework for its own sake, but rather to gain observability into the software development lifecycle and achieve a unified view of software engineering performance.
Avoid the Pitfall of Evaluating Individual Software Engineers
It’s crucial to understand that assessing software engineering performance is distinct from evaluating the performance of individual engineers. Unlike many other aspects of the corporate world, software engineering is a collaborative endeavor. Relying solely on metrics to determine the value of an individual engineer is a risky practice.
The reality is that your top-notch software engineers often devote their time to enhancing the skills of others. They identify problems, mentor junior team members, assist with architectural challenges, and enhance code efficiency. All of these valuable contributions may not be reflected in metrics. As a result, your most valuable team members may have lower individual engineering metrics.
Don’t Forget the Human Factors & Promoting Developer Effectiveness
As we just discussed, measuring individual engineers can be very dangerous and lead to unexpected results. That said, it is a best practice to measure your process to ensure it is promoting developer effectiveness and productivity as opposed to hindering it.
For example, when measuring output there could be a tendency to believe that simply increasing the hours your team works and assigning more tasks would increase productivity. In reality, increasing the number of hours worked doesn’t necessarily increase the quantity or quality of the code. But it will inevitably result in a change in culture and deteriorate the quality of life of the engineers, resulting in a decrease in productivity and quality of the code. Additionally, if the engineers aren’t working on the right things, then increasing the number of hours worked will not necessarily increase the value to the business.
The end goal of any measurement program should not focus solely on the performance of the software development team, it should also focus on ensuring that the organization is maintaining an environment that promotes developer effectiveness. Developer Effectiveness is promoted by developer efficiency, an optimized SDLC that promotes developer productivity, and avoids developer burnout. Human factors are a key aspect of software engineering performance that is often overlooked, but it should be considered as one of the key areas to measure software engineering performance.
Introducing Software Engineering Intelligence: Treno
As we have discussed throughout this document, monitoring and measuring software engineering performance has become increasingly vital in the realm of software tooling. To achieve consistent visibility into software engineering performance, the most effective approach is to leverage a software engineering intelligence (SEI) platform like Treno. Treno seamlessly integrates with the various tools used throughout the software development life cycle (SDLC), models your organization’s software-building process, and establishes a connection between strategic business outcomes and engineering metrics.
Treno is purpose-built to address the challenges associated with measuring software engineering performance. It offers both the structure and flexibility necessary to assess the performance of complex SDLCs. The platform operates on the following principles:
Observability: Treno achieves observability by effortlessly integrating with the SDLC tooling. By utilizing metadata from these systems, it models your team’s code development and software product-building process, enabling in-depth analysis of every branch, epic, issue, or incident.
Analysis: Treno automatically analyzes the collected metadata, transforming it into actionable metrics and insights that identify software delivery issues
Improvement: Treno empowers engineering leaders to compare their teams’ performance with industry leaders and drive improvement by setting data-driven goals.
Treno equips managers with the tools and insights needed to make informed decisions that can optimize time, finances, resources, and ultimately enhance the overall development process and team culture. By seamlessly integrating with all the tools employed by software development teams in their SDLC and capturing data on every action taken by the software engineering team, Treno provides a comprehensive and detailed overview of the team’s performance.
Conclusion
Software engineering organizations have become more complex, with teams spread out, specialized tools being used, and a growing need for new software features and functionality. This complexity has made it extremely challenging for engineering leaders to track productivity and assess the value delivered by their teams. In today’s landscape, it is imperative to have monitoring and measurement systems in place for the software development life cycle (SDLC).
Treno.io is a Software Engineering Intelligence platform designed to address these challenges. Treno empowers organizations by providing visibility into their SDLC. It creates an environment that enhances developer effectiveness, generates reports on team and project performance, and offers recommendations for areas of improvement. With Treno, organizations can better understand and optimize their software engineering processes to drive success.
