Viewpoint: Survey research: two types of knowledge

Patten Smith

I shall argue here that, in the UK, there is a major divide in the kinds of knowledge held by survey experts in research agencies and in academia, and that this works to the detriment of survey research. As befits a Viewpoint article, I shall perhaps portray this divide over-starkly, but I think it is right to do this – there is a serious point to be made.

To my mind, those of us working in agencies who claim survey expertise are strong on practice and weak in theory, while academic survey experts show exactly the opposite qualities. To borrow Gilbert Ryle’s terminology, agency practitioners are strong on knowing how while academics are strong on knowing that.^[1] In the agencies we know how to write questionnaires, design samples, collect data, and report results efficiently and quickly. But we are often hazy in knowing that the accuracy of our results should be assessed in such and such a way according to advanced statistical theory. Furthermore, most of us know very little about the latest theories and findings relating to how questions are (mis-)answered – those concerning response order and question order effects, for example. On the other hand, survey specialist^[2] academics have vices and virtues that are the mirror images of ours. They know the theory and the published findings, they have a rigorous framework for assessing survey error,^[3] and can point to many weaknesses in the surveys we run. But – in my experience at least – many survey expert academics would be hard put to write a useable 45-minute interview questionnaire in two days flat, let alone swiftly set up and implement a survey that delivers acceptable results to a reasonable timescale. And with this practical exiguity sometimes comes a raft of unrealistic expectations about the sorts of data a survey can reasonably be expected to collect.

In the above I have, perhaps, portrayed the relationship between agency practitioners and academics as akin to that between novelist, and critic. However, we should not let this metaphor tempt us into complacently thinking that we in the agencies are the only ones who actually produce something useful, and that academics are somehow parasitic on our endeavours. Such an attitude is surprisingly common in agencies, and usually involves the supposition that we practitioners base our surveys on ‘pragmatic’ decisions that somehow magically produce ‘fit for purpose’ data, whereas captious academics, working in their ivory towers, produce vast amounts of information of little relevance to a posited ‘real’ world. Such complacency has, of course, no grounding in logic (although it does have one in self-interest). In reality we can only judge whether our data are accurate or not by judging them, or their means of generation, against one of two sorts of criteria:

a priori criteria based upon statistical theory and logic – for example, we can make defensible statements about likely levels of sampling and non-response error for a random probability survey with a good response rate
a posteriori ones, showing us that when we have done similar surveys in the past they have delivered data that align with trusted external data.

Often we in the agencies cannot put our hands on our hearts and say that the results of out latest survey are vouchsafed by criteria of either sort. Instead, if pushed, we tend to say that we are confident in the results of our current survey because it uses methods that have ‘worked’ in the past, where by ‘working‘ we mean that when the survey was done before, respondents answered our questions and we came up with results that were not obviously implausible. Unfortunately, however, it is perfectly possible for a survey that ‘works’ in this way to produce wildly inaccurate data, and, given this, any belief we have in the accuracy of our data often requires from us a good measure of faith. If we want, as we should, to do better than this we have to take these criteria seriously, and this is where academics come in with their sophisticated understanding of how the criteria can be applied.

In short, we need guidance from academics as much as they need our craft skills to generate data. Unfortunately, however, in the world of surveys in the UK the two kinds of survey expert live in a kind of semi-detached symbiosis with one another, and this leads to significant problems:

practitioners make needless mistakes because they lack depth in their understanding of how survey errors work
the bulk of surveys in the UK (those not using random probability samples for a start) receive almost no serious academic methodological attention, and suffer as a result
academic commentary and expectations can be very unrealistic.

Clearly it would be better if we in the agencies knew some of what the academics know, and if academics knew some of what we know. Academics’ critiques and commentaries will be enriched by a deeper understanding of the practical exigencies of survey research and, perhaps more importantly, as we became more critical of the veracity of the results we produce, we would be motivated to improve our methods.

How might we learn from each other? A few obvious ideas would include:

having academics take secondments in agencies and agency staff take academic secondments
establishing formal links between agencies and academic departments with resource sharing – giving academics access to new data and practitioners access to electronic libraries
encouraging academics and agency practitioners to co-author papers
improving the quality of formal survey training for both academics and practitioners.

Will this be achieved? I rather doubt it. So long as clients are happy not to interrogate the veracity of the data supplied by agencies and so long as academics can make a career out of analysing survey data without having to get their hands dirty collecting it, where will be the motive for real change?

References
[1] Ryle, G. (1949) The Concept of Mind. London: Hutchinson.
[2] By which I mean the relatively small number of UK academics who have specialised to some degree in survey methodology. My discussion does not relate to the much larger group of academics who have at one time or another analysed some survey data.
[3] As set out in, for example, Groves, R.M., Fowler, F.J., Couper, M.P., Lepkowski, J.M., Singer, E. & Tourangeau, R. (2004) Survey Methodology. New York: Wiley.

Patten Smith is Director of Research Methods at Ipsos MORI.

International Journal of Market Research 51(6), 2009

Would you like to respond to this Viewpoint? Or perhaps you have an idea for another? Responses and new submissions are welcome. They should be emailed to the IJMR, where they’ll be considered for publication.

What's New - Membership - Company Partner Service - Members' Area - Code/Guidelines - Qualifications - Training - Awards
Events - Networking - Publications - Media Info - Market Research - Search - Site Map - A-Z Directory - Contact Us - Home