Abstract
A sample of 1452 adult respondents in the United States was drawn using a highly respected commercial Internet-based service provider. The 85-question survey addressed issues related to anti-consumption behavior among consumers. The instrument incorporated four quality control checks designed to identify inattentive respondents and cleanse the resultant data set. These quality control (QC) checks were a uniform identifier (where one specific response should result), an instructional manipulation check. A common knowledge question, and a time check. A total of 514 (35.4%) respondents failed at least one of the four QC checks with nine respondents failing all four. Comparisons of inattentive respondents to those who passed all four QC checks consistently identified statistically significant differences on each of the metrics used to determine the potential impact of inattentive respondents. These results indicate that inattentiveness does not produce “random noise” as some authors have stated; rather the inattentive respondents “create both random and systematic measurement error which impacts estimates of the reliability and validity” of the metrics used to assess various consumer phenomena. The results of this study are important, not only in projects designed to help marketers, but also for research in other fields where accuracy is of paramount importance.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Ajzen, I. 1991. The theory of planned behavior. Organizational Behavior and Human Decision Processes 50 (2): 179–211. https://doi.org/10.1016/0749-5978(91)90020-T.
Akbulut, Y. 2015. Predictors of inconsistent answers in web surveys. Internet Research 25 (1): 131–147.
Anonymous. 2021. Online surveys: What are they & examples, Market Research Blog, https://cfrinc.net/cfrblog/online-surveys-advantages-disadvantages/. Accessed 30 January 2023.
Anonymous. 2009. Herbert Simon. The Economist, https://www.economist.com/news/2009/03/20/herbert-simon.
Armstrong, J., and T. Overton. 1977. Estimating nonresponse bias in mail surveys. Journal of Marketing Research 14 (3): 396–402.
Aruguete, M., H. Huynh, B. Browne, B. Jurs, E. Flint, and L. McCutcheon. 2019. How serious is the ‘carelessness’ problem on Mechanical Turk? International Journal of Social Research Methodology 22 (5): 441–449. https://doi.org/10.1080/13645579.2018.1563966.
Bhat, A. 2023. What are online surveys? https://questionpro.com/blog/what-are-online-surveys/. Accessed 30 January 2023.
Brown, R. 2004. Consideration of the origin of Herbert Simon’s theory of “satisficing” (1933–1947). Management Decision 42 (10): 1240–1256. https://doi.org/10.1108/00251740410568944.
Buchanan, E., and J. Scofield. 2018. Methods to detect low quality data and its implication for psychological research. Behavior Research Methods 50 (6): 2586–2596.
Deutskens, E., K. de Ruyter, and M. Wetzels. 2006a. Comparing the generalizability of online and mail surveys in cross-national service quality research. Marketing Letters 17 (2): 119–136.
Deutskens, E., K. de Ruyter, and M. Wetzels. 2006b. An assessment of equivalence between online and mail surveys in service research. Journal of Service Research 8 (4): 346–355.
Dupuis, M., E. Meier, and F. Cuneo. 2018. Detecting computer-generated random responding in questionnaire-based data: A comparison of seven indices. Behavior Research Methods 51: 2228–2237.
Evans, J., and A. Mathur. 2018. The value of online surveys: A look back and a look ahead. Internet Research 28 (4): 854–887.
Frankenfield, J. 2022. Satisficing: Definition, how it works, and an example. Investopedia. https://www.investopedia.com/terms/s/satisficing.asp. Accessed August 9, 2023.
Fullerton, S., and T. McCullough. 2022. How quality control checks can help overcome the pitfalls of collecting primary data via an online platform. Proceedings of Society for Marketing Advances 5: 538–547.
Fullerton, S., D. Taylor, and J. Watson. 2009. Web-based instruction and online delivery of classes: Where are we now? American Journal of Business Education 2 (1): 91–100.
Google. 2023. reCAPTCHA protects your website from fraud and abuse without friction https://www.google.com/recaptcha/about/. Accessed 3 January 2023.
Greszki, R., M. Meyer, and H. Schoen. 2015. Exploring the effects of removing ‘too fast’ responses and respondents from web surveys. Public Opinion Quarterly 79 (2): 471–503.
Grieve, R., and J. Elliott. 2013. Cyberfaking: I can, so I will? Intentions to fake in online psychological testing. Cyberpsychology, Behavior, and Social Networking 16 (5): 364–369.
Groves, R. 2011. Three eras of survey research. Public Opinion Quarterly 75 (5): 861–871.
Hooley, T., J. Marriott, and J. Wellens. 2012. What is online research? Using the Internet for social science research. New York: Bloomsbury Academic.
Howell, B. 2020. Dealing with bots, randoms and satisficing in online research. https://www.psychstudio.com/articles/bots-randoms-satisficing/#bots. Accessed 20 November 2020.
Huang, J., P. Curran, J. Keeney, E. Poposki, and R. DeShon. 2012. Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology 27 (1): 99–114.
Ladini, R. 2021. Assessing general attentiveness to online panel surveys: The use of instructional manipulation checks. International Journal of Social Research Methodology 25 (2): 233–246. https://doi.org/10.1080/13645579.2021.1877948.
Lee, S., J. Park, and J. Lee. 2015. Explaining knowledge sharing with social capital theory in information systems development projects. Industrial Management & Data Systems 115 (5): 883–900. https://doi.org/10.1108/IMDS-01-2015-0017.
Li, Z., Li, M. and Li, J. 2008. An empirical research on the influencing factors of knowledge sharing willingness among virtual community members, 2008 In 4th International Conference on Wireless Communications, Networking and Mobile Computing, pp. 1–4, doi: https://doi.org/10.1109/WiCom.2008.2622.
Loomis, D., and S. Patterson. 2018. A comparison of data collection methods: Mail versus online surveys. Journal of Leisure Research 49 (2): 133–149.
Maronick, T. 2009. The role of the Internet in survey research: Guidelines for researchers and experts. Journal of Global Business and Technology 5 (1): 18–31.
Meade, A., and S. Craig. 2012. Identifying careless responses in survey data. Psychological Methods 17 (3): 437–455.
Neto, P. 2020. Trust is the path to accessing better consumer data. The Marketing Insider, 4 December, https://www.mediapost.com/publications/article/358385/trust-is-the-path-to-accessing-better-consumer-at.html?utm_source=newsletter&utm_medium=email&utm_content=headline&utm_campaign=120702&hashid=QcDYypzu0lpI4JhIX-Z3m2YYqK8. Accessed 11 December 2022.
Nichols, A., and J. Edlund. 2020. Why don’t we care more about carelessness? Understanding the causes and consequences of careless participants. International Journal of Social Research Methodology 23 (6): 625–638. https://doi.org/10.1080/13645579.2020.1719618.
Oppenheimer, D., T. Meyvis, and N. Davidenko. 2009. Instructional manipulation checks: Detecting satisficing to increase statistical power. Journal of Experimental Social Psychology 45 (4): 867–872.
Paas, L., and M. Morren. 2018. PLease do not answer if you are reading this: Respondent attention in online panels. Marketing Letters 29 (1): 13–21.
Pollfish. 2021. History of survey platforms & technology. https://resources.pollfish.com/market-research/online-survey-platforms-history/. Accessed 30 January 2023.
Rosenblatt, K. 2023. ChatGBT passes MBA exam given by Wharton professor. NBC News, January 24, https://www.nbcnews.com/tech/tech-news/chatgpt-passes-mba-exam-wharton-professor-rcna67036. Accessed 31 January 2023.
Said, M. 2022. Predicting the Sustainability Integration Effectiveness within the Government-Linked Universities in Malaysia. Global Business and Management Research, Supplemental Special Issue 14 (4): 132–141.
Scott, C. 1961. Research on mail surveys. Journal of the Royal Statistics Society 124 (2): 143–205. https://doi.org/10.2307/2984129.
Silber, H., D. Danner, and B. Rammstedt. 2019. The impact of respondent attentiveness on reliability and validity. International Journal of Social Research Methodology 22 (2): 153–164. https://doi.org/10.1080/13645579.2018.1507378.
Sincero, S. 2012. Online surveys, October 16, https://explorable.com/online-surveys. Accessed 30 January 2023.
Suh, M. 2010. How accurate are online polls? Pew Research Center, 20 December, https://www.pewresearch.org/2010/12/29/how-accurate-are-online-polls/. Accessed 11 December 2022.
Tella, A. 2015. Electronic and paper based data collection methods in library and information science research. New Library World 116 (9/10): 588–609.
Terwiesch, C. 2023. Would Chat GPT get a Wharton MBA? A prediction based on its performance in the operations management course. Mack Institute for Innovation Management at the Wharton School, University of Pennsylvania.
Trustmary. 2023. Online survey advantages and disadvantages. Trustmary Team, 23 January, https://trustmary.com/surveys/online-survey-advantages-and-disavantages/. Accessed 30 January 2023.
Twin, A. 2021. Market research, 10 September, https://www.investopedia.com/terms/m/market-research.asp, Accessed 16 April 2022.
Vasquez, J. (2011) The history of marketing research, 11 February, http://marketresearchworld.net/content/view/3754/49/. Accessed 11 December 2022.
Zhang, C., and F. Conrad. 2013. Speeding in web surveys: The tendency to answer very fast and its association with straightlining. Survey Research Methods 8 (2): 127–135.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
Please accept the manuscript titled Using Quality Control Checks to Overcome Pitfalls in the Collection of Primary Data Via Online Platforms for consideration for publication in the SMA Special Issue of the Journal of Marketing Analytics. I confirm that there is no conflict of interest between the authors and any interested party regarding outcomes. No funding by any interested party was provided. As such, it can be declared that On behalf of all authors, the corresponding author states that there is no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendix A: Scale items used for current study
Appendix A: Scale items used for current study
Consumer ethics
It is acceptable for a customer to eat a candy bar in a supermarket and check out without paying for it.
It is acceptable for a consumer to report a lost item as “stolen” in order to collect money from an insurance company.
It is acceptable for a customer to give misleading price information to a retail clerk in order to get a lower price.
It is acceptable for a customer to return damaged merchandise to the retailer even though the damage was the customer’s own fault.
Optimism
I am confident that I can help to solve the problems that lie ahead of us.
The major problems that we face today are not a threat to our future because I can help solve them.
When I see a big problem the world faces, it does not concern me that much because I know I can help to eventually solve the problem.
When I think of social or environmental issues we face, I am not concerned about the future. I can help to find ways to solve the problems we face. I always have.
Political inclination
I am politically more liberal than conservative.
In any election, given a choice between a Republican and Democratic candidate, I will usually vote for the Democrat.
I cannot see myself ever voting to elect conservative candidates.
On balance, I lean politically more to the left than to the right.
Each item used the following response set so as to conform to a Likert scale format
-
Strongly Disagree
-
Disagree
-
Slightly Disagree
-
Slightly Agree
-
Agree
-
Strongly Agree
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Fullerton, S., McCullough, T. Using quality control checks to overcome pitfalls in the collection of primary data via online platforms. J Market Anal 11, 602–612 (2023). https://doi.org/10.1057/s41270-023-00249-z
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1057/s41270-023-00249-z