Survivorship Bias Business

“The biggest problem we continue to deal with is not being able to attract and increasingly retain top talent in our workforce” Kevin told me. Kevin was a senior executive at a larger information management company.

“So, what are you doing to address the issue?” I asked Kevin in response.

“That’s the problem – nothing we are doing seems to be working. Every year we continue to conduct more surveys of our employees and even adopted NPS scoring. But everything we seem to be trying as a result of that feedback isn’t fixing the problem.” Kevin said. “It’s very frustrating!”

Kevin is not alone in dealing with this problem – many larger businesses suffer from this same problem every day. Skilled talent is an invaluable asset in limited supply and as I point out in my most recent book iQuit! – How The World of Work is Changing and What it Means for You, trends facing the United States in the years and decades to come will only help make this problem worse – A LOT WORSE!

Kevin wants to attract and KEEP top talent and is looking for answers in how to do it. The problem is he continues to look for these answers in the wrong places. Don’t get me wrong here; nothing is wrong with surveying your current employees. But if your problem, like Kevin’s, is attracting and then keeping top talent from leaving, then he is seeking answers from the wrong sources. In short, survivorship bias is slowly killing his business.

So, what is survivorship bias?  

The best way to help illustrate survivorship bias comes from a military research project carried out in WWII. At the time, the US Army Air Corp wanted to find ways to improve the survivability of their airplanes on combat missions. To help do this, they started by analyzing the bullet / shrapnel holes from returning aircraft to see where they were most commonly hit (see adjacent image – red dots).

Now, for a little background here, it is important to note that airplanes were not initially designed to be shot out of the sky. As a result, they need to be light with tradeoffs between performance and protection – the more protection (i.e. weight) added to the plane, the poorer the performance. Therefore, when it comes to adding protection, engineers must be selective as to where they can add a limited amount of it. So, when they created a chart showing all the holes in the surviving aircraft, most people naturally thought to add armor to those areas. That was until they brought in a non-military mathematician named Abraham Wald who told them they were wrong.

Wald’s conclusions were not based on where the bullet holes were on the surviving airframes but rather where they were not. Why? Because the surviving airplanes studied with the holes had made it back while everybody else did not. Therefore, instead of protecting the areas with all the holes, Wald deduced that the airplanes should instead be armored where they were not (see green circles). In short, the initial bias towards datapoints of the survivors in many respects were the antithesis of those that were lost.

The principles behind survivorship bias can be applied to nearly every aspect of our lives – including our business world. And what companies like Kevin’s suffer from is not the ability to want to address his talent issues that plaque them, but the datapoints used in order to make their decisions. Let’s break down this problem for Kevin in more detail.

First, most employee surveys have a low completion rate ranging from 30%-85% – with 75% or more being considered as optimal. In short, only a third (worse case) or 75% (optimal case) of your sampling of employee data is even captured. But how many of your most talented employees actually complete these and do so accurately? Moreover, when a growing number of employees don’t like answering these surveys, think they are pointless, think no results come out of them, and know the majority of managers don’t find them valuable, many consider these employee surveys as nothing more than a waste of their time – leading to skewed results.

Second, if you are trying to attract and retain better talent, why get insights only from people that are already working for you versus those that have left? This is no different than the US military looking at holes in returning planes and trying to initially protect those holes. Moreover, if you are losing “top” talent and cannot replace them with the same caliber of employees, what does that say about your existing workforce and the opinions of the people who are sticking around?

Third, what do you really know about the “top talent” in the marketplace? Naturally, it is difficult to seek insights from people you cannot initially hire but what about those you did that have left? What sights did you gain from them and how do you incorporate that valuable information into your decisions? Many organizations try to capture this information from exit interviews.

Depending on the size of a company, exit interviews are conducted by 50%-90% of companies today, of which around 70% are conducted by HR personnel (is that motivating?), 19% by the employee’s direct supervisors (conflict of interest anyone?), and so forth. In recent studies conducted by exit-interviewing companies, fewer than a third of the companies studied indicated actions taken as a result of these interviews – meaning two-thirds of the time, priority, action, and application of the data received was largely useless.

Over my career, I have proactively conducted countless exit-interviews with many “top” employees in my business, including many times with people outside my supervisory control – whether my company promoted such actions or not. I always provided them with “safe harbor” during our conversations and what amazed me was the disconnect between what they shared with me in private versus what many senior leaders independently concluded about their leaving the company. Instead of acknowledging and addressing their real issues, I would instead commonly hear excuses for their leaving and usually in a manner that covered peoples assess instead – frequently to the detriment of the company.  

Exit interviews conducted by HR people can be problematic as well. In many companies, these tasks are delegated to more junior HR people with little understanding of the business, conducted in a “check-box” manner, and often received low participation rates from top performers, and for good reason – most felt that their time was wasted talking to a low-level person that would likely result in nothing useful coming out of it. Many of the top people I talked with over the years also told me that they refused their HR-led exit interviews for varying reasons…while many others were never asked to begin with – usually to their surprise and dismay. Nearly all these people went on to become top performers with their next employers.

There are many tips and best practices around exit interviews and a good place to start is reading a recent Harvard Business Review on this topic. For the sake of time, I will save many of my own learnings and suggestions on the details of this topic for another article – for it is a moot point until the pathology of this problem is first understood and acknowledged. Which leads me back to my previous conversation with Kevin…

“Kevin, what are your exit interviews of your top talent people within your company telling you about your company?” I asked.

“Exit interviews?” Kevin responded with a curious look on his face. “I’m not sure we conduct exit interviews when people leave us. Let me find out…”

Nearly two weeks passed when Kevin and I met again and here is how the conversation continued: “Kevin, did you find out about an exit interview process at your company?” I asked.

“Yes.” Responded Kevin. “In fact, I finally heard back from somebody in HR who informed me that it is standard practice to conduct exit interviews when people leave our company.”

“Great” I responded. “And what happens with that information once received and how is it used to help people like you address your recruiting and retention issues?”

“Well, that’s a good question.” Kevin said. “And I still can’t get a solid answer to it. The senior head of HR, who is newer, said a summary of this information is provided in their annual reports to us, but I have never seen it. When I did finally get a copy of last year’s report, I couldn’t even make sense of it. It almost seemed like this was something they had to do instead of wanting to do right. When I expressed my concerns, the same HR leader reminded me that we conduct “lengthy” employee surveys instead twice a year. So, if I need useful employee information, I should get it from those surveys instead.”

All of this makes me wonder what this HR manager would say in WWII when looking at all those returning airplanes with bullet holes in them.

Good thing she chose a different profession…. except for Kevin.


iQuit! Book


Comments are closed