Running a survey is pointless unless you can make sense of the data. This post provides a practical perspective on how we tend to analyse employee survey data. After each survey we run for our clients we meet to review the survey data using our survey platform so this is an overview of how we do that.
In its pure form your survey data is just a bunch of numbers so it needs some work to convert it into information. This is about telling the story which the organisation can then use to gain insight around where it needs to take action.
Before we get into the detail it is important to remember that how you design your employee survey has a big impact on how you are able to analyse the data. The types of questions you use in your survey and the types of rating scales you use will determine how you will be able to analyse the survey responses. For example, it is much easier to make sense of how people respond to a quantitative rating scale question than it is to make sense of how people respond to a free text question.
Analysing your employee survey data effectively is also an important way you can measure the impact of your engagement strategies and initiatives.
As such, analysing your survey data needs to be seen as just one step in improving employee engagement in your organisation.
Key survey statistics
We tend to start by reviewing some of the key survey statistics.
This will include a review of the survey completion rate and a brief review of any demographic data captured.
Completion rate provides a sense of how many people took part in the survey (we actually calculate participation rate as not everyone will complete the survey fully). Most organisations want to feel that they have had a reasonable participation rate for the data to be seen as useful. We often get asked what the response rate needs to be to be ‘valid’, but there is actually no definitive answer to this. Employee surveys are based on asking people’s opinions of how they feel about working for the organisation so we tend to take the view that all views are valid no matter how many there are. Response rates will also vary depending on the type of business. For example, retail and hospitality businesses will usually be pleased with a 50% response rate, in the NHS it can be as low as 20%, whilst we have a couple of clients who are disappointed if they don’t achieve 80%.
Reviewing demographic data can provide an overview of how survey responses were distributed such as which part of the organisation respondents came from, what the gender or age range profile was etc. Demographic data becomes more useful when we get into slicing and the dicing the data to see if there are any differences between different groups of employees.
Survey scores by theme
Most of your survey questions are likely to be quantitative rating scale questions. These are the detailed questions where everyone rates how they feel and they are likely to be grouped into themes. We start by taking a high level view of the aggregated scores for each theme.
Rating scale question data can be analysed in two ways:
- We can calculate the mean average for each theme. In this survey we used a rating scale with 6 points on it (strongly agree – strongly disagree) so the maximum possible score is 6 and the minimum possible is 1. We tend to recommend with a 6-point scale that you aim for a minimum score of 4 as this is where most people are tending to agree with the survey question statements.
2. We can calculate the proportion of people who use each part of the rating scale.
Each type of chart uses the same data, it is just displaying it in two different ways. The stats can be explained by looking at the distribution curve of the data. The mean average chart shows the peak of the curve, the percentage chart shows the size of each rating scale ‘chunk’ of the curve.
The data for the survey themes shown in the charts above is giving us a clue as to where people are more or less satisfied.
Individual question scores
Now we have an overview of which themes in our survey are scoring high or low we can begin to drill into the detail so the next thing we do is start to explore the scores for each individual question in the survey.
Again we can view the data using either mean averages or the percentage distribution, but this chart shows the mean average scores for each question.
How do different groups of employees feel?
Reviewing the data at the question level allows us to really home in on the specific issues and we usually find at this point that clients want to start exploring to see if there are any differences between different groups of employees.
This is where the demographic data in the survey really helps as we can use it to slice the rating scale question data using the demographic questions in the survey. For example we can look at how people in different parts of the business feel, we can see how people with differing lengths of service feel etc. This is why it is important to use demographic questions in your employee survey.
In our platform we can either create multiple datasets by applying filters or we can use heatmaps to do the same thing.
So where do we focus our efforts?
The analysis we have outlined so far will provide some very strong pointers around where the issues might be. From a high level we know which themes we need to focus on, we have identified some specific questions where we can make improvements and we have explored to see if there are any variations based on demographic data.
There is a temptation to home in on the questions that have the lowest score, but this does not necessarily mean that it will have an impact on overall levels of engagement. Questions with low scores indicate that people are dissatisfied with them, but they might not actually have a big impact on overall engagement. For example, people might be dissatisfied with aspects of their job, but they might still be engaged with the organisation as a whole. In our experience, pay tends to fall into this category. Pay has undoubtedly become more of an issue for people in the last few months, but our analysis shows that although people would like their pay to be higher, other aspects of the employee experience tend to have a bigger impact on how engaged they feel.
We need to do some more analysis to understand which specific survey questions are having the biggest impact on overall levels of engagement. These are your engagement drivers.
Our Engagement Driver Analysis uses a bit of simple stats to do this by correlating how every employee rates each survey question compared with how they rate one ‘engagement outcome’ question. The most common engagement outcome question is “I would recommend the organisation as a great place to work”. We call it an engagement outcome question because if you score highly on all the other survey questions people will tend to also recommend the company as a good place to work.
We combine the results of this correlation calculation and use some colour coding to identify which specific questions are having an impact on overall engagement and how people feel about those questions. In the table below the ‘r Value’ column (nothing to do with Covid!) shows the strength of the relationship between the survey question and how much people would recommend the company. The higher up the list, the stronger the impact. The colour coding is based on how employees feel about each survey question (shown as average scores).
- Green indicates a question that has an impact on engagement and is scoring well.
- Amber indicates a question that has an impact on engagement, but could be improved.
- Red indicates a question that has an impact on engagement, but needs significant improvement.
Your red questions are the ones where you should focus – improving satisfaction levels with these questions will drive improvements in overall levels of engagement.
Establishing overall levels of engagement using eNPS
We started referring to Net Engagement Score a few years ago after taking learning from experts in customer experience management and seeing how we could apply it to employee experience management. The concept of Net Promoter Score has been around for a while in customer experience management, but in recent years Employee Net Promoter Score (eNPS) has become a more established term.
Our more detailed blog on eNPS goes into more detail, but in simple terms eNPS is based on how much employees would advocate your organisation as a good place to work. It provides a single metric of overall engagement.
Survey validity
We often get asked ‘is our survey valid?’. This is a complex subject but there are several ways we can explore the validity of your employee survey data. Validity is actually one of three aspects when it comes to determining how much you can rely on your survey data. Reliability and statistical significance are part of the same picture so just focussing on validity can only be part of the picture.
Survey validity is mainly impacted by survey design and sample size. Assuming the survey is well designed and you have a representative sample size you can then start to explore statistical significance. Our engagement driver analysis is designed to provide you with an indication of which questions in your survey are the most significant.
In conclusion
Analysing your survey data takes time and effort, but the value is not in the analysis, it is in the story that the analysis tells. There is no point spending weeks analysing data and producing overly complex information if your stakeholders can’t easily make sense of it or quickly understand where they need to focus their efforts. Our survey platform was built to significantly reduce that effort and help you make sense of your survey data as quickly and intuitively as possible.