Accurate confidence intervals and measures of differences between sub-groups are often needed.

Particularly in situations where you are working on large scale surveys involving random (probability) sampling and where a high degree of reliability and precision is needed.

Very rarely are these the same as those from the regularly-quoted “p(1-p)/n” formulae. A (random) sampling scheme will be very likely to contain elements of stratification or clustering or both and these will make the intervals or precision levels greater or less than that from the quoted formula. If weighting occurs or if the population is small in relation to the sample (e.g. when sampling from small, specific groups), then the precision levels will be different again. This course demonstrates how intervals and significance testing may be accurately calculated when these effects occur.

The way in which confidence intervals and comparisons between two groups is calculated may be relatively simple if based on percentage measures. However, many measures that require analysis are more complex than that, such as mean scores and net scores (e.g. brand preferences, net satisfaction and NPS). If one needs to compare sub-samples with overall samples, if there is overlap between one’s two comparator groups and/or if one is interviewing the same individuals in part or in total, then further complexities would need to be borne in mind.

This course navigates delegates through all of these complexities and illustrates how valid, robust and convincing inferences can be made from the data, even if intricacies have been introduced after and despite appropriate survey design. It aims to make accessible and meaningful to researchers, many of the formulae which are covered in the traditional, often rather mathematically-positioned textbooks.

Who would benefit

Delegates will be researchers or client-side research commissioners with at least 2 years of quant experience who are comfortable with basic mathematical principles; especially those working on large scale public-sector surveys or where results are required with a high degree of accuracy. Buyers/receivers of research will find this course useful too, so that they will understand what questions to ask about the means by which the information was collected; thus helping them take a more critical stance.

Objectives

  • To equip advanced quant researchers with the tools and confidence to ensure that their large-scale surveys are robust and will produce accurate and reliable quoted results.
  • To add weight to the research carried out.
  • To give commissioners of research the tools to ask the right questions and hence to ensure that their suppliers are ones which fully consider the technical implications of their survey and sample design.
  • To appreciate the purpose and limitations of the “p(1-p)/n” formula in calculating confidence intervals (i.e. precisions) and significance testing.
  • To understand what the practical implications are for changes to the precision levels of surveys where weighting, stratification, clustering exists or when the populations are small/finite.

Learning outcomes

  • Understanding and appreciation of the concepts of the Design Effect and Effective Sample Size.
  • To be able to calculate confidence intervals and significance testing for each of the above and where comparison samples overlap or partially overlap, such as in “Panel Surveys”.
  • To be able to determine the differences between percentages, mean scores and net scores.
  • To be able to calculate confidence intervals and significance testing for each of the above and where comparison samples overlap or partially overlap, such as in Panel Surveys.

Level
Advanced

Andrew Zelin is a Freelance Data Scientist, with 25 years’ experience as a professional statistician, running analytical projects in Market Research, Central Government, Health and the Telecomms Sectors.  This includes over a decade of leading Analytics teams at senior level at Ipsos.  He has given training in a range of statistical and survey-design related practices since 2001 within his organisations, the MRS and the Rod Laird Training Company.  He is a Fellow of the Royal Statistical Society (RSS) who has presented at RSS, AAPOR and SAS conferences, has published two award-winning methodology papers and is a Mentor for the MRS.

Additional Information

Get the latest MRS news

Our newsletters cover the latest MRS events, policy updates and research news.