What did you do?
The theory and practice of how to ask questions to accurately measure behaviours
Recently we looked at the way in which we need to think carefully about the metrics we use to measure outcomes – and talked about the value of asking about behaviour through self-report surveys. As well are all well aware, there can be pitfalls when asking people to do this (as much of course as there are also pitfalls in passive data reporting of behaviours). This week we are sharing some of our guidance on how to ask about behaviour within surveys. First we cover some of the broad principles of how we think about and recall behaviours before highlighting into some best practice pointers for how to construct survey questions to accurately measure behaviour.
Principles of self-reporting behaviour
Reference periods: We may want to ask questions about whether a behaviour happened in the last few weeks, months, or some other time interval and to calculate this with reference to the timing of the interview. For example, ‘As of the end of LAST WEEK, how many times did you go for a run?’. This requires an onerous task for the respondent to consider the reference period or date relative to the time of the interview. We cannot underestimate the demands of this sort of task – as the respondent may not previously have had to consider the information we are requesting.
Try to only ask only about first-hand events: Behaviours we usually ask about are ones in which the respondents themselves took part - for example, did they exercise or buy something. However, sometimes we ask the respondent to operate as a proxy for another – so a single adult may report for everyone in the household. Unsurprisingly, research suggests that people remember first-hand information more accurately than second-hand not least because it is typically aided by the presence of sensory details and their knowledge of the causes and effects.
Defining events: Survey designers assume that respondents are able to identify ‘events’ but there is little clarity about how these are defined. How do we define breakfast for example? Would this include the coffee in bed as well as the bowl of cereal at the kitchen table? In studying how we perceive our actions, psychologists have recorded actors engaging in common activities and then asked people to indicate where the events began and ended. These experiments have found somewhat limited agreement among subjects about ‘event boundaries’.
Generic knowledge: Much of the way we memorize information about ourselves is in the form of scripts, mental representations of commonplace sequences of behaviours, such as shopping for groceries, going to work, or going to see a film. The scripts consist of a causal sequence of behaviours - for example, entering a supermarket, finding a trolley, finding our items, waiting at the checkout, paying, and exiting. Much autobiographical knowledge consists of information about these general patterns, which means that we tend to remember and report our typical behaviour rather than our specific behaviour. Given that information about the overall pattern is often much more easily retrieved than information about the individuating details, care will need to be taken to distinguish one event from others of the same type.
Extended events: Many personal events are linked to the circumstances in which they took place. We think of the event as happening when you bought your first house or during your holiday or while you were living in London. These serve as ‘chapters’ in our autobiographical memory and provide a way to locate behaviours that fall under them. We think of our lives - past, present, and future - in terms of a series of fairly well-defined periods and milestones. This offers a rich structure that can be helpful to facilitate recall.
Telescoping: The telescoping effect refers to the way people perceive recent events as being more distant than they are (backward telescoping) and remote events as being more recent than they are (forward telescoping). In reality, forward telescoping is much more widespread issue as we are often asking about recent events (we are typically less interested in distant events although in some instances these are very important). With forward telescoping respondents get more hazy about dates as time passes and as a result tend round their answers to conventional time periods such as ten days ago, or a three months ago. This results in an overstatement as respondents who made purchases two or three weeks ago are more likely to report that they purchased it in the last ten days.
Asking about behaviour: best practice
Overall, there are no ‘silver bullets’ when it comes to designing behavioural surveys. The key issue is to understand the way in which respondents think about the topic and work with the limitation of recall. The more specific it is possible to be the better but this needs to be done in a way that does not require of the respondent impossible feats of memory (e.g. when in a period of seven days two weeks ago did you notice some advertising for a mobile phone brand that featured price). There are however some broad guidelines that can be followed that will assist in development of more accurate behavioural data.
Aided recall: A good way to get respondents answering questions about their behaviour more accurately is to provide them with a prompt that will anchor their response. This can help identify small but important aspects of their behaviour that relate to a specific event (rather than falling back on reporting on ‘typical behaviour’).
Keep it specific: The more specific the question, the easier it is for a respondent to answer. The ideal is for researchers to follow the journalist’s golden rule: ask ‘who, what, where, when and why?’ General questions require much more effort by the respondent. Think of the apparently straightforward question of ‘What brand of soft drink do you normally buy?’. To answer this accurately, the respondent needs to work out the relevant time period, and circumstances. So do we mean at work, at the cinema, at home? They then need to consider what is meant by ‘you’. Is the respondent themselves or all members of their household who they may be buying for. And how are we defining a soft drink? Is iced tea included? Mineral water? Most of us will buy several brands and therefore have to do a fair amount of work to answer this apparently simple question. This can mean that we fall back onto using brand awareness and salience rather than behaviour – which can mean significant over reporting.
Appropriate Wording: The answers that you get out of a piece of research depend on the questions – and the wording – that you put in. This is where an experienced survey designer is needed. They should be able to identify the right words that everyone will understand and which leave no room for confusion, a process that will require a fair bit of testing. Words to be avoided are those that consist of jargon or slang, those with multiple means in the context of the question being asked, and those which may have unexpected meanings to some of the respondents.
Keep wording as long as it needs to be: The common view is that shorter questions produce better responses. Yet while this may hold true in the context of attitude questions, the opposite seems to apply to behaviour topics. Indeed, here length helps reduce the number of omitted events and in turn improves recall. It is unclear why but there is some speculation that longer questions help aid recall as they are often more specific. Also, the longer they are, the more time it takes the researcher to ask them, and the more time the respondent has to think about the answer (although it is not clear whether this also applies online).
We collect information about behaviour from a range of sources, both survey-based but also from a host of other places such as passive data collection. Having the flexibility to ask questions about behaviour has a range of benefits (cost, flexibility, allows follow-ups, can collect more holistic and diagnostic information). Again, we can see the real value of marketing & social research methods alongside experimental design approaches – this integrative outlook is increasingly the cornerstone of effective applied behavioural science best practice.
For those that are looking for more detail, this book is a great resource.