Viewpoint: Separating methodologies?
Chris Barnham MMRS
Chris Barnham Research and Strategy
There have always been important differences between the research findings derived from quantitative research and those of qualitative research. The two methodologies have different approaches and their intended goals are not the same. In many ways, they also have competing visions of what constitutes truth. Despite these differences, however, the two methodologies often work in effective symbiosis with each other and each brings to the other a level of understanding that it would not otherwise achieve. As a professional qualitative researcher I have worked alongside quantitative researchers for many years and have, for several decades, seen qualitative research recommendations being subsequently, and very successfully, quantified.
This article puts forward the view, however, that these two methodologies are nowadays producing research results that are more divergent than they were a decade ago. This increasing level of difference is, I believe, also directional. My experience in the last ten years, across a range of different clients and different market sectors, suggests that quantitative results are consistently more positive than they were ten years ago. This means that concepts, packaging designs or advertising ideas that were borderline in terms of how well they performed in qualitative research are now often achieving relatively satisfactory scores in subsequent quantification tests. As a qualitative researcher, I have to admit that this is not altogether a negative outcome – it tends to place the qualitative research recommendations in a more positive light. But, professionally, one cannot help feeling that brand owners are being misled into thinking that potential marketing activities are being given more of a ‘green light’ than should be the case.
So what has changed in the last ten years? The answer, of course, is that quantitative research has become predominantly an online experience for respondents. As such, a quantitative survey is something that they do at home, in their own time and it is carried out in a way that is entirely unsupervised. This compares with the experience, ten or twenty years ago, which took place in a hall, or at home, and within a supervised interview context. As such, the quantitative interview experience for the respondent was, in the past, more similar to the qualitative one. Both experiences in the past were supervised, both involved interaction with an interviewer who could follow up with (at least some) open-ended questions, and both interview contexts created a degree of responsibility in the mind of the respondent towards the answers that he or she were giving. The interview, in both methodologies, was, critically, a personal interaction between two people, which arguably encouraged a degree of personal honesty in the respondent.
Many have suggested, of course, that the unsupervised interview scenario is, in fact, an advantage – when online quantitative research first emerged it was argued that it removed any influence that an interviewer might have on the respondent. It was suggested that respondent answers might be more truthful as a result. I would argue, however, that it has, in fact, reduced the sense of responsibility that the respondent feels towards the interview situation. But, whatever the effect of the interviewer on the respondent, at least the same effect was taking place in both quantitative and qualitative research environments.
Qualitative research (unless it is online) has largely continued with its conventional ways of doing depth interviews or groups, but the shift to online research in the quantitative sphere has now, I believe, opened up significant differences between the two methodologies. The online quantitative experience is now quite different to the qualitative face-to-face experience because the former is solitary, unsupervised and there is no interaction with an interviewer. In contrast, the latter remains interpersonal and, in a sense, supervised.
At the one-day IJMR Conference in November last year, BrainJuicer showed how the emotional state of the respondent sometimes has an impact on their responses to research questions. They explained how a more content consumer will answer questions in a more positive frame of mind. Orlando Wood also alluded to research in the legal world that demonstrated that it is far better to be in front of the judiciary in the UK system if the judge has just had lunch. A replete judge is more likely to be lenient in his or her sentencing than one who is feeling hungry. Frame of mind and context have significant effects on how questions are answered or how decisions are made.
If this learning is applied to the shift from supervised surveys to online questionnaires we can see how this effect might have consequences in the world of quantitative research. A quantitative respondent sitting in front of a computer screen is in the comfort of his or her own home. They have not been pulled off the street when they were trying to do their shopping (with their kids in tow), nor have they been interrupted by an unexpected knock at the front door. They have probably filled in similar questionnaires before and they are more confident as a result. Additionally, they are responding at a time of their own choosing. They probably have a cup of coffee in their hand and can approach the interview as a form of end-of-day relaxation. All of these factors will have an effect on the respondent’s mood and state of mind. They will create a research context that is generally more positive than those associated with quantitative surveys of the past. And they are, I would argue, more likely to deliver positively inclined results.
No one likes bad news, and market researchers are much happier giving their clients good news. Over the years, however, I have found a significant shift towards quantitative research delivering more positive results for concepts, pack designs or advertising ideas. Many of these concepts have performed adequately, but not that strongly, in qualitative research. There has been, as a result, genuine surprise when they have researched well in subsequent quantitative research testing. This experience has occurred across a range of product fields and clients, so one can only assume that the quantitative agencies were also different.
I have not, in this article, touched on other issues that surround the subject of quantitative online research. These include recruitment and the identity of respondents in online surveys, etc. All of these issues have been well rehearsed and discussed in the literature. In many cases, evidence has also been produced to show that online quantitative research delivers results that are not dissimilar to traditional methods. What is usually being compared in such studies, however, is two different sorts of quantitative research. The focus of this Viewpoint article has been, in contrast, to argue that we should look at the comparison between quantitative and qualitative research as well.
International Journal of Market Research 54(6), 2012