Reading Between the Lines: The Unspoken Truths of Employee Surveys
In my email newsletter I recently wrote about an excellent report which used a long form discovery technique (naturalistic interviews and thematic analysis of the transcripts). This kind of discovery work is typically how I start my work with clients. One key reason for this is that the management of companies often gets a very incomplete picture of how things are on the ground for their staff, regardless of how close to the work they are or how much time they invest in designing and running employee surveys. This isn’t their fault, it’s time intensive & resource intensive to conduct detailed long-form discovery work, and there is often a perception that this work is “fluffy and subjective”.
Around this time of year, many companies are looking to launch an end of year employee survey in part to meet this demand for “discovery” data. Employee surveys are easy to run and can offer a “numerically-precise” picture of the world, which people perceive as hard data even if the data is equally subject to bias and poor reliability. I find surveys are highly effective and I use them all the time, but they have some significant limitations. The aim of this post is to highlight those limitations so you can be more deliberate about designing surveys and interpreting their results.
In general, surveys are good at getting a little bit of information from a lot of people. But:
Surveys rely on self report measures which can be biased and/or unreliable
There are challenging trade-offs between data richness and data quality
Bias & poor reliability in self report measures
Self-report measures have well known limitations. “Everyone lies” by economist and data scientist Seth Stephens-Davidowitz highlights the broad evidence base which highlights the gap between what people say (largely measured through survey data) and what they do (based on search data and social media data). Even in anonymous surveys, people feel like they will be judged for giving true answers so they skew their responses towards what they consider to be the “norm” - this is known as social desirability bias.
An amazing new paper by a joint team from the University of Leeds and University of Cambridge captures this complexity well. They used a “Think Aloud Protocol” to get participants to elaborate on what they’re thinking about when answering survey questions. Around 1 in 4 participants appear to have misinterpreted what survey questions were asking or misunderstood an item in the survey. They also found that participants struggled to answer survey questions in a way that captured their “multiple selves” and felt like the surveys missed out on nuance. While their work didn’t focus on employee surveys specifically, the complexity in answers points to these wider challenges in self report measures, which can further drive bias and poor reliability.
Within the world of employee survey data, the picture is even more complicated.
Complex incentives and interpersonal dynamics may encourage people to misreport how they think things are going. They may worry about their responses reflecting poorly on their productivity or worry about backlash from those that they work with. It’s also not clear how companies tend to use the information, which can be made work in organisations where there is distrust between staff and the people teams gathering the data (e.g., following recent rounds of lay-offs).
But even if employees want to report how things are going truthfully they may struggle to do that effectively. Employees are often busy doing their day-to-day work and are responding to surveys without the mental bandwidth or time to really reflect on how things are going. This can lead to their responses being more subject to biases - negative topics or things that happen frequently tend to be more salient, even if they aren’t representative of people’s wider experiences. And participants may be more subject to framing biases in how the survey questions are posed.
Trade-offs between data richness and data quality
Okay but even if you could get people to be honest, reflective and reliable in their self report, surveys still only tell a narrow picture of the world.
The majority of questions in employee surveys tend to be closed prompts where participants score their level of agreement: “rate how much you agree with the following statements”. These questions are good at allowing participants to quickly express how they feel about the area of the prompt but it doesn’t provide much richness beyond an indication of the level of feeling towards that prompt. In other words, the data provides a snapshot of how things are in that one specific area but rarely offers much explanation to why things are the way they are or how to change things.
But that isn’t a bad thing - if anything it’s a feature not a bug. Like we said earlier, surveys are good at getting a little bit of information from a lot of people. Counter-intuitively the more detail and richness you try to extract information from a survey, the worse your data quality gets.
Normally employers who want to understand more include more questions or add several free-text response options to their surveys. This can be helpful but there are severe diminishing returns. First, as surveys get longer the completion rate from participants falls significantly - Surveymonkey found that a 30 question survey has 3 times the drop-out rate of a 10 question survey. This participant drop-off (also known as attrition) leads to a survivor bias, where the kinds of people who fill out the survey are different to those who drop off. In this case, the type of participants willing to endure a 20 minute process are typically the most engaged (or the most vocal) and the people you tend to lose first are those who are disengaged. In a wide array of studies poor workplace engagement has been found to precede retention challenges. As a result, this selection bias skews the results in ways that are hard to see and you lose information on the people you most want to target interventions on.
Worse still, as you increase the number of questions the people who do fill the survey tend to start “button clicking” where they speed-run through the questions without properly thinking about them (worsening some of the salience biases we mentioned earlier). This trend is known as “satisficing”. Surveymonkey found that moving from 10 questions to 30 questions can cut the amount of time participants spend per question by more than half, inevitably resulting in poorer quality data.
Third, as you introduce more questions which ask about the same topics in different ways, the results are more subject to data-mining issues. You have so many questions to choose from that you can find the results to retrospectively tell any story that you want. This is rarely intentional, but it’s very hard to resist. What may start as innocent analysis of the results can easily lead to over-interpretation. In psychology this is typically known as HARKing - which stands for Hypothesising After the Results are Known and even research specialists find it hard to avoid.
What does this really look like? How big is this problem?
Well, to understand the impact of some of these challenges - we commissioned a poll from Survation. They got 1000 people across the UK to answer a two part questions - first we ask them if they’ve worked at a company that sent them a survey and second we ask them about how they responded to the survey.
Part 1: Many companies send their staff employee surveys. Have you ever received an employee survey from either the company you work for now, or a company for which you have previously worked?
Yes
No
Not applicable – I haven’t worked / I have only ever been self-employed
[If they respond Yes…]
Part 2: Which of the following statements is closest to your experience?
I report how I feel about work accurately and honestly when filling out employee surveys
I exaggerate how I feel about certain areas of work when filling out employee surveys
I understate how I feel about certain areas of work when filling out the survey
I normally don't submit a response if I am sent an employee survey
The responses to the first question were not exactly what I expected. Only 43% of participants recalled working at companies that had sent them an employee survey. This number increases when you look at prime working age populations, but even still only half of 35 to 54 year olds had ever received an employee engagement survey. This already highlights the need to gather more employee data.
As part of the second question, 96% of participants say that if they receive and employee survey they tend to fill it out. If you showed the people team of most companies a 96% completion rate, they’d be astonished. Likely what is happening here is that many of our survey participants are unaware that their companies run employee surveys or have forgotten that they didn’t complete them.
The remainder of the second question was about how people tend to answer the questions in these surveys. The majority of participants said that they report how they feel accurately, however over a quarter of our sample admitted to mis-reporting their feelings. This split fairly evenly over those who tend to exaggerate how they feel and those who tend to under-report how they feel.
Note: This 26% is likely to be an underestimate given the social desirability bias we mentioned in surveys earlier!
The results also show that younger people were more likely to mis-report how they felt in a survey, with 30% of 18-34 year olds either exaggerating or under-representing their answers. We don’t have a breakdown by occupation but we do have the region a participant is from which might provide some proxy measures for differences in industry and job type.
If we first look at the regional breakdown, we can see the previous “symmetry” between under-reporting and over-reporting across 18 to 55 year olds disappear. This makes the bias in how people fill out the survey more problematic. If the same proportion of people always over-reported as under-reported, taking an average (or even the median results) can mitigate many of the measurement problems. But as you can see below, the results start to look very asymmetric as disaggregate along regional lines:
Beyond the fun of spotting regional differences (Londoners & Scots loving the drama of exaggeration, people in the Midlands being the most understated) the data highlights a need to have a more granular picture along industry line and job types, which also vary significantly by region.
What’s clear from the polling exercise is that participants aren’t always honest - in fact as many as 30% may be mis-reporting how they feel when directly asked.
So what can we do about this?
But things aren’t all doom and gloom, surveys are still a great way to gather data. You just have to be aware of their limitations and be deliberate about how you use them. Three tips to help you think about discovery work with your employees:
Redesign your surveys with their limitations in mind:
Cut down the number of questions dramatically
Use the survey to identify “hotspots” but don’t try to use it to find the root causes of issues - this requires a few questions that get a lot of coverage
Boost the signal-to-noise ratio of your questions by using a mix of different styles of questions, these tend to slow participants down and encourage them to think about the prompts more carefully
Pre-agree your analysis approach before you see the results:
Think through how you will use the data before you see the results - which questions are diagnostic, and how will you think about using other questions to drive your actions
This pre-agreement of approach is akin to “pre-registration” which is becoming the norm in high quality academic research
Triangulate survey results with other discovery techniques:
I try to a broad array of discovery techniques to get a more detail picture of what’s going on in companies, including but not limited to:
Long form interview techniques
Alternative survey formats (I like to use q-methodology for example)
Observational studies
There are two main reasons for triangulation:
You can validate the results of one technique by looking at the overlap from the findings of another technique
You can also lean into what each technique is best at
Learn by doing in addition to asking how people are feeling:
Piloting new ideas can be a great way of testing hypotheses about people issues and can generate a feeling that people are being listened to, which is almost as important as doing the discovery work
In an ideal world you can do a randomised trial to really test out what works (which I’ve done at a few clients) but even just introducing a “pilot period” for a new initiative and asking for feedback can be really effective
Taking these steps can help organisations of any size to have a much better idea of what their employee priorities should be and how they can address those priorities.
If you want to have a chat about some of these research approaches or talk about people topics - I’m starting to run “Office Hours”. If you’re interested in joining an office hour slot, drop me an email at Isar@Uncover.business.
