What makes software engineers authentically productive?

600 developers from 92 companies competed to write the best code. The results are weird.

What can we learn about performance in engineering teams from those who really studied it.

From 1984 to 1986 a competition created by Tom DeMarco and Timothy Lister, was created to put software engineers against one another and measure what made certain teams perform better than others. The competition was simple. Who could write the best code, in a short time, with as few defects as possible.

The competition was known as the coding War Games and it worked like this.

Everyone was put into pairs from the same company. But they weren’t partners, their job was to compete against each other, as well as others pairs.

All the programmers performed the same activities and documented their time in a log book. All the coding was done at their usual desks, in their usual work spaces, where they would write the code as normal.

They wrote in whatever language they usually would, and used the same equipment and tools to get the job done.

Three important factors

There were 3 main insights that came up as it relates to individual performance.

The top performing individuals would outperform the worst by about 10:1

The best performers were about 2.5 times better than the median performers and the half that were better than the medium performers, could out-perform the other half by about 2:1

Netflix HR director Patty McCord, Author of Powerful, wrote famously in her Culturedeck that Netflix’s preference was to staff the entire company with those top performers, and weed out the rest.

“Sustained B-level performance, despite “A for effort”, generates a generous severance package, with respect.”

“Sustained A-level performance, despite minimal effort, is rewarded with more responsibility and great pay” — Patty McCord, Netflix.”

This recognition that top performers create, not just a 2x or 3x return, but a 10x return, is the reason that companies like Google and Netflix spend such a great deal of time trying to attract the world’s best programmers. The sentiment of this is sometimes referred to in the personal economics theory known as Tournament theory.

When the stakes are at their highest, such as circumstances where the winner takes all the rewards, and second place means you are left with nothing, as is often the case with some high growth software companies, then to a certain degree, it doesn’t matter what it costs to win. Rightly or wrongly, at high performing organisations, software engineering talent is sometimes being viewed through this paradigm.

Does quality actually take more time?

But what didn’t make any difference to performance during the Coding War Games? When the researchers went through the data, they found absolutely no correlation to performance in the following factors.

The programming language seemed to not make one shred of difference. People who were coding in COBOL and Fortran did the same as those writing in Pascal and C with the exception of Assembly language, who were simply slow on all counts.

Obviously, these languages are a little older than the ones we use today, and having one language deliver poor results is probably an indication one language could also make you faster. But that feels like a conversation for another day.

The second factor was how many years of experience a programmer had. They made an interesting observation. They uncovered that if you had over a decade of experience coding, you didn’t necessarily do better than someone with only two years of experience. But there was a bottom to this. If you had less than six months, you didn’t do as well as the rest of the group.

This isn’t to say that experience doesn’t count, it’s just to say for this coding test, it didn’t count. Read into that what you will.

If you’re a newbie programmer getting into coding today, you could take inspiration from these numbers. It could be depressing to think that senior developers out there with 10 years of experience will always be able to add more value than you can. But it would seem your ability to contribute great outcomes will come sooner than you think. Unless of course you’re one of those bottom 10%. In which case all hope may be lost.

The third interesting insight was when it came to bugs. Over 33% of the developers were able to write their code with absolutely zero defects. No bugs at all. And when you group these programmers together, these coders weren’t even slower. It would be safe to assume if someone is being more careful with their code, and trying to make it 100% bug free, it would take longer to do so. But that’s not what happened. In fact, they were even faster.

The third point is worth dwelling on, because it breaks down a myth that in all cases, quality takes time. This might be an oversimplification, but looking at the numbers, it appears that quality takes the right people, more than it takes more time.

Finally, the fourth observation relates to money and its role in performance. There is so much information out there about why paying for performance is a bad idea. Daniel Pink’s book is perhaps the most well-known on the topic, where he breaks down performance drivers into three things. Autonomy, being able to do your work your way. Purpose, having an end mission in mind, and finally, Mastery, an educational mindset that keeps people in a flow state for longer, because they find the work just challenging enough to be interesting, which is commonly referred to as deliberate practice.

With a few exceptions, these 3 things are the real factors driving motivation, but more money does not.

Then it’s then no surprise that salary played no role in productivity during the coding war games. There was no correlation between those who performed well, and how much they were paid.

So what really mattered?

It turns out Daniel Pink could be onto something. When the researchers tried to find a correlation, they found that who you were paired up with mattered a great deal. If you were matched with someone who performed well, you performed well too. So is being paired with someone an indication you are on the same mission? Does being in a team, give you both a sense of purpose?

If you were paired with someone slow, you were slow.

If you were paired with someone who wrote bugs, you did too. So what made people paired together do the same kind of things?

The gap between pairs in terms of performance never exceeded more than about 21%. Making them all quite close in terms of outcomes.

It makes you reflect on who your teammates are. Could being paired with the highest performer elevate your performance? The same could be said in reverse. What if you get lumped with someone terrible? Does that bring your grades down?

Perhaps the relationship between the pairing is less about purpose and more about mastery. If someone is slightly better than you, and you’re paired with them, you are going to step up your game and are bound to pick up a few things along the way.

Is the environment key?

In the experiment, all pairs were from the same organisation. So they shared all kinds of things, workplace culture being a big one. When they looked at pairs, they found most performed similarly. It also meant that there was an overwhelming observation that top performers tended to appear in the same organisations. So the question is, do the best organisations exist because of the top performers who work there, or do top performers naturally gravitate towards the best companies?

When researchers looked through all their data, they started to wonder how physical environments played a role in performance. What if the actual spaces were influencing the outcomes of the tests?

So they went back out to the developers and surveyed them, asking questions about their immediate environments. They asked things like how quiet they were and how much they were interrupted.

Then they compared those answers with the very top performers and the very bottom performers of the coding test.

The results?

For the top performers, their spaces were more private. There was less open space planning. They had less interruptions, and were left alone to do their work more frequently. It’s no wonder then over the last few years, open plan offices have seen a reversing trend. Here is an example of the responses based on the two groups.

How much dedicated space do you have?
Top performers = 78 sq. ft.
Low performers = 46 sq. ft.

Is your space quiet?
Top performers = 57% yes
Low performers = 29% yes.

Is your space private?
Top performers = 62% yes.
Low performers = 19% yes.

Do people often interrupt you needlessly?
Top performers = 38% yes
Low performers = 76% yes.

It’s unwise to confuse causation with correlation, but perhaps giving your team a quiet place to work, avoiding interruptions, and leaving them the hell alone may provide huge dividends on performance. It’s no wonder people are looking for more remote working options than ever before.

A modern interpretation

It may be easy to dismiss the findings as too old to be relevant. By saying that programming languages have changed a lot since the 80’s, workplaces mostly have headphones keeping distractions at bay, and that we have a vastly different understanding of how to deliver software today, makes this study a little outdated.

But when Google ran a similar experiment, code named project Aristotle, they also found that factors like programming languages, money and years of experience, didn’t matter that much when it came to performance. They also found that being near each-other had no impact on performance. Instead, they found that 5 factors contributed to the highest performing teams more than anything else

Impact

Team members felt their work mattered.

Meaning

Work was personally important to team members

Structure and Clarity

Team members have clear roles, plans and goals

Dependability

Team members get things done on time and meet a high bar for excellence, and finally, the biggest of them all.

Psychological safety

Team members feel safe to take risks and be vulnerable in front of each other.

My own research

When we built OHNO.ai (The last company I started), the product captured what problems were stopping teams from meeting their mission, their objectives, and upholding the values that were important to the organisation.

In essence, we were trying to find friction in teams that was slowing them down.

We could tell in real time, what held teams back, and we found some interesting things. Two topics come up more than anything else. Time, and focus.

When people talk about time, it’s not that there isn’t enough time, it’s that the time they have is mostly wasted on things that aren’t really productive, distracting them from what’s really valuable.

When it comes to focus, the biggest complaint we saw is not having some simple goals that everyone on the team understands. You would be amazed how fast a company moves when you can get everyone rowing in the same direction.

Setting goals can be a real motivator, but only if the goals don’t keep changing. You would be surprised how often problems like ‘lack of direction,’ or ‘no clear understanding of the goals’ came up in our survey results with customers.

When goals are set, then start changing, it’s difficult for teams to take them seriously. So they ignore them.

Each week, when we collected surveys from our users, we aggregated that data across industries, and discovered the real reasons teams were stuck. It’s became clear to us that having a clear set of objectives and a sense of purpose, appears to get teams moving in the right direction.

To look at the newsfeeds on the internet, we’d be led to believe that where companies spend a lot of time worrying, it’s on topics like disruption, innovation, AI and what the competition are up to. But when we aggregated the data at OHNO, the problems we saw were much more basic than that. Not understanding purpose, lack of focus, and wasting huge amounts of time on activities that don’t seem to matter, dominated the trends of what teams felt was slowing them down. The science tells us that these internal, cultural factors appear to be the real issues worth dealing with, and companies ignore them at their peril.

When it boils down to it, letting your teams have some autonomy, setting some clear objectives, holding people accountable, and ensuring everyone believes in the mission will get you further than you think. Don’t underestimate the basic stuff, and it’s impact on team performance.

“If you want to build a ship, don’t drum up the men and women to gather wood, divide the work, and give orders. Instead, teach them to yearn for the vast and endless sea.” — Antoine de Saint-Exupery

Perhaps the real reason we ignore the science, and instead, focus on launching new products or coming up with innovation strategies, is because fixing the cultural elements is much harder.

It’s difficult to create a culture where we feel safe to be vulnerable. It’s hard to set clear goals when different departments can barely agree, and it’s a tall ask to give co-workers meaning in their work when we aren’t sure our leaders find meaning in theirs. But what is clear, is that those who put the time in on these areas and work on those things, are clearly out-doing us all.

Research from this article comes mostly from Peopleware. Productive Projects and Teams, written by Tom DeMarco & Timothy Lister. It’s an incredible book, and if you have an interest in high performing software teams. It’s worth a read. You can get a copy right here.