close
close

Development productivity in the era of generative artificial intelligence

“Productivity is being able to do things you have never been able to do before.”

-In a second

With generative AI gaining the attention of almost every organization, it’s interesting to see how they use this technology. Today, most AWS customers are focused on achieving productivity gains (e.g., making it easier for customer service agents to acquire and digest information). A second, much smaller group of customers are looking for more disruptive applications of generative AI, but not to the point of redefining industries. These use cases range from rethinking customer experiences at Accor and Booking.com to accelerating pharmaceutical research using synthetic data. A virtuous process is likely to manifest itself in most organizations where efficiency frees people and investment to rethink and reinvent industries.

So far so good. However, companies are starting to ask themselves: how to measure productivity and is it important?

Instead of talking about productivity in general, let’s focus on developer productivity. We know that a huge amount of time is wasted in the average enterprise, as I discussed in my blog post on developer productivity. Here’s the challenge: if we could save twenty hours of developer time by changing processes and using Amazon Q Developer, would it be invested in significant areas, or would we simply find new, creative ways to waste our precious resources?

To learn more about this topic, I spent time with Joe Cudby, Principal Specialist on AWS’s Builder Experience team.

Joe, the term programmer productivity is relatively new and not familiar to me as a former programmer. AWS doesn’t typically use this term. Why?

My team regularly discusses software development journeys with AWS customers, including how they view and measure developer productivity. However, there is a clear shift from a narrow focus on individual developer productivity to a more expansive understanding of team development productivity at the organizational and SDLC levels. This shift, like the adoption of DevOps before it, requires a true cultural shift coupled with increased collaboration that extends beyond engineering to include business leadership, product management and end users. We believe that simply measuring an individual’s productivity without these broader considerations is typically not useful.

What led to the focus on teams rather than individuals?

There is no one simple metric that can provide a complete and detailed picture of overall productivity or progress. Metrics such as lines of code developed were discredited many years ago, but overly simplistic metrics such as story points, which are subjective and measure effort rather than value, are often still used. Using story points is a great way to plan and estimate work, but they are often misused as a measure of productivity. This problem becomes more pronounced when trying to compare the productivity of individuals and teams, even within the same company. Teams face very different challenges and opportunities depending on the heterogeneity or homogeneity of the technology stack and architecture. Within each of these applications and technology stacks, the number of languages, frameworks, and technical debt have accumulated to varying levels over many years. Even the difference between maintaining a system and developing a new one causes significant differences in productivity.

So how can an organization start thinking about developer productivity rather than developer productivity?

AWS leverages three core dimensions that go beyond just productivity. The first is system health, which measures end-user-oriented outcomes such as product quality, performance, availability, and feature utilization. After all, being productive but generating the wrong results is an oxymoron. Second, we look at the efficiency of CI/CD processes, namely volume and speed metrics such as deployment frequency, turnaround time between commits, and change failure rate. It can be assumed that this is only a speed measurement, but there is a hidden requirement to hit the quality and safety standards. Finally, we look at team health for factors such as job satisfaction, turnover risk and burnout rates, based on regular employee surveys and scoring. This may seem like a soft set of metrics, but burnout is highly correlated with loss of productivity.

There are many other frameworks that organizations can adopt, such as SPACE, which considers five dimensions of employee satisfaction and well-being, performance, activity metrics such as code and task performance metrics, communication and collaboration, and flow efficiency. . The DORA structure is probably the best known; connects technical capabilities with productivity by assessing test automation, architectural practices, and more. They influence productivity metrics such as implementation time and frequency, which are directly linked to overall organizational performance and employee well-being.

None of them are simple, single indicators. This reflects the fact that development productivity depends on the interplay and intersection of all these different factors at the individual engineer, team and company level. The key to understanding development productivity is a combination of objective data on system-level performance, engineering outcomes and activities, and regular surveys and assessments of subjective developer sentiment. This three-pronged insight is the basis for a sustainable culture that retains top talent. To paraphrase an old saying, it is often easy to measure the wrong things; measuring the right things can be difficult, but it pays off.

How can these development productivity metrics be misused?

These metrics are often intended to indicate health rather than hiring, firing and reward. They should be used to help the team improve and self-monitor. One anti-pattern to avoid is Goodhart’s law, where the goal becomes a measure of, say, development efficiency. A common misuse is to use metrics to compare teams despite the differences in technologies and challenges I described earlier. Use these metrics for the purposes they are intended for: enabling teams to identify areas for improvement, monitor the effectiveness of their working practices, and ensure it is all done in a sustainable manner.

What does the introduction of tools like Amazon Q Developer mean for developer productivity?

We see Amazon Q Developer being used by companies like British Telecom to help in areas such as recommending code, identifying quality and security issues, automating tests, and providing personalized productivity insights. Amazon Q Code Transformation is another tool that helps reduce technical debt on a large scale, such as a .NET or Java version upgrade. It allows development teams to get back to what they and their companies want: delivering business results. Be creative! AWS customers use these tools to create synthetic test data and unit test scripts, develop features, troubleshoot issues, remediate potential security vulnerabilities, and more. An important side benefit of these tools is that new research shows that the use of AI-based software tools correlates with improved well-being.

What advice would you give for improving developer productivity?

Improving development productivity should be viewed as a continuous, gradual journey with small gains that accumulate over time, rather than a search for a one-time silver bullet. Start by using the data and metrics you have today to develop a baseline. Much of this will come from CI/CD tools measuring speed, but don’t forget to look at code quality and security posture. Adding a regular subjective survey is an easy additional step. Often, the people who know how to improve productivity are the same people you measure, so ask them for their insights! Pilot changes with development teams and assess their impact. Use a combination of tools like Amazon Q Developer with work changes that minimize distractions, reduce work in progress and dependencies, and focus your team on the most value-added work.

Many AWS customers outsource development. Any advice for them?

I have to believe that while ultimately both the outsourcer and the AWS customer may have different goals, they want to ensure that their development teams are productive, working towards meaningful results, and using the tools available to them to continually improve team performance. Most of these more mature solutions understand that value is not just in charging for hours spent on development, but in truly delivering value. I’ve seen AWS customers and their partners sit down and create visibility into the metrics I’ve discussed in this blog, and then use them to look for improvements wherever they may be. A step in this direction is to ensure that contracts encourage the use of tools such as Amazon Q Developer.

Application

Being a programmer is an exciting time. For those who have been at it for a while, this is the next evolution of the tools we grew up with, improving our ability to focus more on delivering valuable results and business value. What I like about Joe’s focus on growth productivity is the implicit recognition that the unit of productivity in most organizations today is the team, not the individual. Perhaps we technologists can help colleagues in other parts of our organizations learn this lesson as well.

-Philip