Abstract
A core feature of behavior analysis is the single-subject design, in which each subject serves as its own control. This approach is powerful for identifying manipulations that are causal to behavioral changes but often fails to account for individual differences, particularly when coupled with a small sample size. It is more common for other subfields of psychology to use larger-N approaches; however, these designs also often fail to account for the individual by focusing on aggregate-level data only. Moving forward, it is important to study individual differences to identify subgroups of the population that may respond differently to interventions and to improve the generalizability and reproducibility of behavioral science. We propose that large-N datasets should be used in behavior analysis to better understand individual subject variability. First, we describe how individual differences have been historically treated and then outline practical reasons to study individual subject variability. Then, we describe various methods for analyzing large-N datasets while accounting for the individual, including correlational analyses, machine learning, mixed-effects models, clustering, and simulation. We provide relevant examples of these techniques from published behavioral literature and from a publicly available dataset compiled from five different rat experiments, which illustrates both group-level effects and heterogeneity across individual subjects. We encourage other behavior analysts to make use of the substantial advancements in online data sharing to compile large-N datasets and use statistical approaches to explore individual differences.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
Data Availability
The datasets generated and analyzed in the current study are available on the Open Data Commons for Traumatic Brain Injury: https://odc-tbi.org/data/703. The R code used to perform statistical analyses are provided on GitHub: https://github.com/VonderHaarLab.
References
Annicchiarico, I., & Cuenya, L. (2018). Two profiles in the recovery of reward devaluation in rats: Latent class growth analysis. Neuroscience Letters, 684, 104–108. https://doi.org/10.1016/j.neulet.2018.07.013
Austin, P. C. (2010). Estimating multilevel logistic regression models when the number of clusters is low: A comparison of different statistical software procedures. International Journal of Biostatistics, 6(1), 16. https://doi.org/10.2202/1557-4679.1195
Baer, D. M. (1976). The organism as host. Human Development, 19(2), 87–98. https://doi.org/10.1159/000271519
Bangdiwala, S. I., Bhargava, A., O'Connor, D. P., Robinson, T. N., Michie, S., Murray, D. M., & Pratt, C. A. (2016). Statistical methodologies to pool across multiple intervention studies. Translational Behavioral Medicine, 6(2), 228–235. https://doi.org/10.1007/s13142-016-0386-8
Baron, A. (1999). Statistical inference in behavior analysis: Friend or foe? The Behavior Analyst, 22(2), 83–85. https://doi.org/10.1007/BF03391983
Barrus, M. M., & Winstanley, C. A. (2016). Dopamine d3 receptors modulate the ability of win-paired cues to increase risky choice in a rat gambling task. Journal of Neuroscience, 36(3), 785–794. https://doi.org/10.1523/jneurosci.2225-15.2016
Baum, W. M. (1979). Matching, undermatching, and overmatching in studies of choice. Journal of the Experimental Analysis of Behavior, 32(2), 269–281. https://doi.org/10.1901/jeab.1979.32-269
Bechara, A., Damasio, A. R., Damasio, H., & Anderson, S. W. (1994). Insensitivity to future consequences following damage to human prefrontal cortex. Cognition, 50(1–3), 7–15. https://doi.org/10.1016/0010-0277(94)90018-3
Beltz, A. M., Wright, A. G., Sprague, B. N., & Molenaar, P. C. (2016). Bridging the nomothetic and idiographic approaches to the analysis of clinical data. Assessment, 23(4), 447–458. https://doi.org/10.1177/1073191116648209
Bono, R., Alarcón, R., & Blanca, M. J. (2021). Report quality of generalized linear mixed models in psychology: A systematic review. Frontiers in Psychology, 12. https://doi.org/10.3389/fpsyg.2021.666182
Boyle, P. P. (1977). Options: A Monte Carlo approach. Journal of Financial Economics, 4(3), 323–338. https://doi.org/10.1016/0304-405X(77)90005-8
Bray, B. C., & Dziak, J. J. (2018). Commentary on latent class, latent profile, and latent transition analysis for characterizing individual differences in learning. Learning & Individual Differences, 66, 105–110. https://doi.org/10.1016/j.lindif.2018.06.001
Burke, D. A., Whittemore, S. R., & Magnuson, D. S. K. (2013). Consequences of common data analysis inaccuracies in cns trauma injury basic research. Journal of Neurotrauma, 30(10), 797–805. https://doi.org/10.1089/neu.2012.2704
Burroughs, V. J., Maxey, R. W., & Levy, R. A. (2002). Racial and ethnic differences in response to medicines: Towards individualized pharmaceutical treatment. Journal of the National Medical Association, 94(10 Suppl), 1–26.
Busemeyer, J. R., & Diederich, A. (2010). Cognitive Modeling. SAGE Publications.
Calude, C. S., & Longo, G. (2017). The deluge of spurious correlations in big data. Foundations of Science, 22(3), 595–612. https://doi.org/10.1007/s10699-016-9489-4
Carrasco-Ramiro, F., Peiró-Pastor, R., & Aguado, B. (2017). Human genomics projects and precision medicine. Gene Therapy, 24(9), 551–561. https://doi.org/10.1038/gt.2017.77
Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2002). Applied multiple regression/correlation analysis for the behavioral sciences ((3rd ed.). ed.). Routledge.
Colaizzi, J. M., Flagel, S. B., Joyner, M. A., Gearhardt, A. N., Stewart, J. L., & Paulus, M. P. (2020). Mapping sign-tracking and goal-tracking onto human behaviors. Neuoscience & Biobehavior Reviews, 111, 84–94. https://doi.org/10.1016/j.neubiorev.2020.01.018
DeHart, W. B., & Kaplan, B. A. (2019). Applying mixed-effects modeling to single-subject designs: An introduction. Journal of the Experimental Analysis of Behavior, 111(2), 192–206. https://doi.org/10.1002/jeab.507
Di Ciano, P., Manvich, D. F., Pushparaj, A., Gappasov, A., Hess, E. J., Weinshenker, D., & Le Foll, B. (2018). Effects of disulfiram on choice behavior in a rodent gambling task: Association with catecholamine levels. Psychopharmacology, 235(1), 23–35. https://doi.org/10.1007/s00213-017-4744-0
Dinh, A., Miertschin, S., Young, A., & Mohanty, S. D. (2019). A data-driven approach to predicting diabetes and cardiovascular disease with machine learning. BMC Medical Infomatics & Decision Making, 19(1), 211. https://doi.org/10.1186/s12911-019-0918-5
Dockstader, C. L., & van der Kooy, D. (2001). Mouse strain differences in opiate reward learning are explained by differences in anxiety, not reward or learning. Journal of Neuroscience, 21(22), 9077. https://doi.org/10.1523/JNEUROSCI.21-22-09077.2001
Dorsey, M. F., Weinberg, M., Zane, T., & Guidi, M. M. (2009). The case for licensure of applied behavior analysts. Behavior Analysis in Practice, 2(1), 53–58. https://doi.org/10.1007/bf03391738
Ellis, S. E., & Leek, J. T. (2018). How to share data for collaboration. The American Statistician, 72(1), 53–57. https://doi.org/10.1080/00031305.2017.1375987
Eysenck, H. J. (1984). The place of individual differences in a scientific psychology. In J. R. Royce & L. P. Mos (Eds.), Annals of theoretical psychology (Vol. 1, pp. 233–285). Springer.
Fisher, A. J., Medaglia, J. D., & Jeronimus, B. F. (2018). Lack of group-to-individual generalizability is a threat to human subjects research. Proceedings of the National Academy of Sciences of the United States of America, 115(27), E6106–E6115. https://doi.org/10.1073/pnas.1711978115
Fishman, G. S. (1996). Monte Carlo. In concepts, algorithms, and applications. Springer. https://doi.org/10.1007/978-1-4757-2553-7.
Frankot, M., Mueller, P. M., Young, M. E., & Vonder Haar, C. (2023). Statistical power and false positive rates for interdependent outcomes are strongly influenced by test type: Implications for behavioral neuroscience. Neuropsychopharmacology. https://doi.org/10.1038/s41386-023-01592-6
Friedman, B., & Nissenbaum, H. (1996). Bias in computer systems. ACM Transactions on Information Systems, 14, 330–347.
Garner, J. P. (2014). The significance of meaning: Why do over 90% of behavioral neuroscience results fail to translate to humans, and what can we do to fix it? ILAR Journal, 55(3), 438–456. https://doi.org/10.1093/ilar/ilu047
Garofalo, S., & di Pellegrino, G. (2015). Individual differences in the influence of task-irrelevant pavlovian cues on human behavior. Frontiers in Behavioral Neuroscience, 9, 163. https://doi.org/10.3389/fnbeh.2015.00163
Gilroy, S. P., & Kaplan, B. A. (2019). Furthering open science in behavior analysis: An introduction and tutorial for using github in research. Perspectives on Behavior Science, 42(3), 565–581. https://doi.org/10.1007/s40614-019-00202-5
Gilroy, S. P., Strickland, J. C., Naudé, G. P., Johnson, M. W., Amlung, M., & Reed, D. D. (2022). Beyond systematic and unsystematic responding: Latent class mixture models to characterize response patterns in discounting research. Frontiers in Behavioral Neuroscience, 16. https://doi.org/10.3389/fnbeh.2022.806944
Gosselin, T., Le Guisquet, A. M., Brizard, B., Hommet, C., Minier, F., & Belzung, C. (2017). Fluoxetine induces paradoxical effects in c57bl6/j mice: Comparison with balb/c mice. Behavioral Pharmacology, 28(6), 466–476. https://doi.org/10.1097/fbp.0000000000000321
Guthrie, E. R., & Horton, G. P. (1946). Cats in a puzzle box. Rinehart.
Hagopian, L. P. (2020). The consecutive controlled case series: Design, data-analytics, and reporting methods supporting the study of generality. Journal of Applied Behavior Analysis, 53(2), 596–619. https://doi.org/10.1002/jaba.691
Hagopian, L. P., Rooker, G. W., & Yenokyan, G. (2018). Identifying predictive behavioral markers: A demonstration using automatically reinforced self-injurious behavior. Journal of Applied Behavior Analysis, 51(3), 443–465. https://doi.org/10.1002/jaba.477
Hagopian, L. P., Rooker, G. W., & Zarcone, J. R. (2015). Delineating subtypes of self-injurious behavior maintained by automatic reinforcement. Journal of Applied Behavior Analysis, 48(3), 523–543. https://doi.org/10.1002/jaba.236
Hamaker, E., Dolan, C., & Molenaar, P. (2005). Statistical modeling of the individual: Rationale and application of multivariate stationary time series analysis. Multivariate Behavioral Research, 40, 207–233. https://doi.org/10.1207/s15327906mbr4002_3
Hoekstra, R., Kiers, H. A., & Johnson, A. (2012). Are assumptions of well-known statistical techniques checked, and why (not)? Frontiers in Psychology, 3, 137. https://doi.org/10.3389/fpsyg.2012.00137
Hudson, K., Lifton, R., & Patrick-Lake, B. (2015). The precision medicine initiative cohort program: Building a research foundation for 21st century medicine.
Kochanek, P. M., Dixon, C. E., Mondello, S., Wang, K. K. K., Lafrenaye, A., Bramlett, H. M., Dietrich, W. D., Hayes, R. L., Shear, D. A., Gilsdorf, J. S., Catania, M., Poloyac, S. M., Empey, P. E., Jackson, T. C., & Povlishock, J. T. (2018). Multi-center pre-clinical consortia to enhance translation of therapies and biomarkers for traumatic brain injury: Operation brain trauma therapy and beyond. Frontiers in Neurology, 9, 640. https://doi.org/10.3389/fneur.2018.00640
Kostick-Quenet, K. M., Cohen, I. G., Gerke, S., Lo, B., Antaki, J., Movahedi, F., Njah, H., Schoen, L., Estep, J. E., & Blumenthal-Barby, J. S. (2022). Mitigating racial bias in machine learning. Journal of Law, Medicine & Ethics, 50(1), 92–100. https://doi.org/10.1017/jme.2022.13
Kyonka, E. G. E., Mitchell, S. H., & Bizo, L. A. (2019). Beyond inference by eye: Statistical and graphing practices in jeab, 1992–2017. Journal of the Experimental Analysis of Behavior, 111(2), 155–165. https://doi.org/10.1002/jeab.509
Kyonka, E. G. E., & Subramaniam, S. (2018). Translating behavior analysis: A spectrum rather than a road map. Perspectives in Behavior Science, 41(2), 591–613. https://doi.org/10.1007/s40614-018-0145-x
Lamata, P. (2020). Avoiding big data pitfalls. Heart & Metabolism: Management of the Coronary Patient, 82, 33–35. https://doi.org/10.31887/hm.2020.82/plamata
Liang, S., Deng, W., Li, X., Greenshaw, A. J., Wang, Q., Li, M., & Li, T. (2020). Biotypes of major depressive disorder: Neuroimaging evidence from resting-state default mode network patterns. Neuroimage Clinical, 28, 102514. https://doi.org/10.1016/j.nicl.2020.102514
Michael, J. (1974). Statistical inference for individual organism research: Mixed blessing or curse? Journal of Applied Behavior Analysis, 7(4), 647–653. https://doi.org/10.1901/jaba.1974.7-647
Molenaar, P. (2004). A manifesto on psychology as idiographic science: Bringing the person back into scientific psychology, this time forever. Measurement: Interdisciplinary Research & Perspective, 2, 201–218. https://doi.org/10.1207/s15366359mea0204_1
Mouri, A., Koseki, T., Narusawa, S., Niwa, M., Mamiya, T., Kano, S., Sawa, A., & Nabeshima, T. (2012). Mouse strain differences in phencyclidine-induced behavioural changes. International Journal of Neuropsychopharmacology, 15(6), 767–779. https://doi.org/10.1017/s146114571100085x
Nakagawa, K., & Kajiwara, A. (2015). Female sex as a risk factor for adverse drug reactions. Nihon Rinsho, 73(4), 581–585.
Nakagawa, S., & Schielzeth, H. (2013). A general and simple method for obtaining r2 from generalized linear mixed-effects models. Methods in Ecology and Evolution, 4(2), 133–142. https://doi.org/10.1111/j.2041-210x.2012.00261.x
National Institutes of Health. (2022). All of us research program https://allofus.nih.gov/
NIH Precision Medicine Initiative Working Group (2015) The precision medicine initiative cohort program – building a research foundation for 21st century medicine. https://acd.od.nih.gov/documents/reports/DRAFT-PMI-WGReport-9-11-2015-508.pdf
Normand, M. P., & Kohn, C. S. (2013). Don't wag the dog: Extending the reach of applied behavior analysis. The Behavior Analyst, 36(1), 109–122. https://doi.org/10.1007/bf03392294
Nuzzo, R. (2015). How scientists fool themselves—And how they can stop. Nature, 526(7572), 182–185. https://doi.org/10.1038/526182a
Oberski, D. (2016). Mixture models: Latent profile and latent class analysis. In J. Robertson & M. Kaptein (Eds.), Modern statistical methods for hci (pp. 275–287). Springer International Publishing.
Open Science Collaboration. (2015). Estimating the reproducibility of psychological science. Science, 349(6251), aac4716. https://doi.org/10.1126/science.aac4716
Préfontaine, I., Lanovaz, M. J., & Rivard, M. (2022). Brief report: Machine learning for estimating prognosis of children with autism receiving early behavioral intervention—A proof of concept. Journal of Autism & Developmental Disorders. https://doi.org/10.1007/s10803-022-05641-9
Radabaugh, H., Bonnell, J., Schwartz, O., Sarkar, D., Dietrich, W. D., & Bramlett, H. M. (2021). Use of machine learning to re-assess patterns of multivariate functional recovery after fluid percussion injury: Operation brain trauma therapy. Journal of Neurotrauma, 38(12), 1670–1678. https://doi.org/10.1089/neu.2020.7357
Revusky, S. H. (1967). Some statistical treatments compatible with individual organism methodology. Journal of the Experimental Analysis of Behavior, 10(3), 319–330. https://doi.org/10.1901/jeab.1967.10-319
Richter, S. H., Garner, J. P., Auer, C., Kunert, J., & Würbel, H. (2010). Systematic variation improves reproducibility of animal experiments. Nature Methods, 7(3), 167–168. https://doi.org/10.1038/nmeth0310-167
Robinson, W. S. (1950). Ecological correlations and the behavior of individuals. American Sociological Review, 15, 351–357. https://doi.org/10.2307/2087176
Scheerer, N. E., Curcin, K., Stojanoski, B., Anagnostou, E., Nicolson, R., Kelley, E., Georgiades, S., Liu, X., & Stevenson, R. A. (2021). Exploring sensory phenotypes in autism spectrum disorder. Molecular Autism, 12(1), 67. https://doi.org/10.1186/s13229-021-00471-5
Shaver, T. K., Ozga, J. E., Zhu, B., Anderson, K. G., Martens, K. M., & Vonder Haar, C. (2019). Long-term deficits in risky decision-making after traumatic brain injury on a rat analog of the Iowa gambling task. Brain Research, 1704, 103–113. https://doi.org/10.1016/j.brainres.2018.10.004
Shull, R. L. (1999). Statistical inference in behavior analysis: Discussant's remarks. The Behavior Analyst, 22(2), 117–121. https://doi.org/10.1007/BF03391989
Sidman, M. (1960). Tactics of scientific research: Evaluating experimental data in psychology. Basic Books.
Sisodiya, S. M. (2021). Precision medicine and therapies of the future. Epilepsia, 62(S2), S90–S105. https://doi.org/10.1111/epi.16539
Skinner, B. F. (1938). The behavior of organisms: An experimental analysis. Appleton-Century.
Sturman, O., von Ziegler, L., Schläppi, C., Akyol, F., Privitera, M., Slominski, D., Grimm, C., Thieren, L., Zerbi, V., Grewe, B., & Bohacek, J. (2020). Deep learning-based behavioral analysis reaches human accuracy and is capable of outperforming commercial solutions. Neuropsychopharmacology, 45(11), 1942–1952. https://doi.org/10.1038/s41386-020-0776-y
Sutton, N. P., Grace, R. C., McLean, A. P., & Baum, W. M. (2008). Comparing the generalized matching law and contingency discriminability model as accounts of concurrent schedule performance using residual meta-analysis. Behavioural Processes, 78(2), 224–230. https://doi.org/10.1016/j.beproc.2008.02.012
Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning. MIT Press.
Todes, D. P. (2014). Ivan pavlov: A russian life in science. Oxford University Press.
Vanderveldt, A., Oliveira, L., & Green, L. (2016). Delay discounting: Pigeon, rat, human—Does it matter? Journal of Experimental Psychology: Animal Learning & Cognition, 42(2), 141.
Veldkamp, C. L., Nuijten, M. B., Dominguez-Alvarez, L., van Assen, M. A., & Wicherts, J. M. (2014). Statistical reporting errors and collaboration on statistical analyses in psychological science. PLoS One, 9(12), e114876. https://doi.org/10.1371/journal.pone.0114876
Vonder Haar, C., Frankot, M., Reck, A., Milleson, V., & Martens, K. (2022a). Large-n rat data enables phenotyping of risky decision-making: A retrospective analysis of brain injury on the rodent gambling task. Frontiers in Behavioral Neuroscience, 16. https://doi.org/10.3389/fnbeh.2022.837654
Vonder Haar, C., Martens, K. M., Riparip, L. K., Rosi, S., Wellington, C. L., & Winstanley, C. A. (2017). Frontal traumatic brain injury increases impulsive decision making in rats: A potential role for the inflammatory cytokine interleukin-12. Journal of Neurotrauma, 34(19), 2790–2800. https://doi.org/10.1089/neu.2016.4813
Vonder Haar, C., Martens, K. M., & Frankot, M. A. (2022b). Combined dataset of rodent gambling task in rats after brain injury.
Weller, B. E., Bowen, N. K., & Faubert, S. J. (2020). Latent class analysis: A guide to best practice. Journal of Black Psychology, 46(4), 287–311. https://doi.org/10.1177/0095798420930932
Wickham, H. (2014). Tidy data. Journal of Statistical Software, 59(10), 1–23. https://doi.org/10.18637/jss.v059.i10
Wickham, H. (2016). Ggplot2: Elegant graphics for data analysis. Springer International.
Wilkinson, M. D., Dumontier, M., Aalbersberg, I. J., Appleton, G., Axton, M., Baak, A., & Mons, B. (2016). The fair guiding principles for scientific data management and stewardship. Scientific Data, 3, 160018. https://doi.org/10.1038/sdata.2016.18
Wixted, J. T., & Ebbesen, E. B. (1997). Genuine power curves in forgetting: A quantitative analysis of individual subject forgetting functions. Memory & Cognition, 25(5), 731–739. https://doi.org/10.3758/bf03211316
Young, M. (2017a). A place for statistics in behavior analysis. Behavior Analysis: Research & Practice, 17, 1. https://doi.org/10.1037/bar0000099
Young, M. E. (2017b). Discounting: A practical guide to multilevel analysis of indifference data. Journal of the Experimental Analysis of Behavior, 108(1), 97–112.
Young, M. E. (2019). Modern statistical practices in the experimental analysis of behavior: An introduction to the special issue. Journal of the Experimental Analysis of Behavior, 111(2), 149–154. https://doi.org/10.1002/jeab.511
Young, M. E., Clark, M. H., Goffus, A., & Hoane, M. R. (2009). Mixed effects modeling of Morris water maze data: Advantages and cautionary notes. Learning & Motivation, 40(2), 160–177.
Young, M. E., & Hoane, M. R. (2021). Mixed effects modeling of Morris water maze data revisited: Bayesian censored regression. Learning & Behavior, 49(3), 307–320. https://doi.org/10.3758/s13420-020-00457-y
Zeeb, F. D., & Winstanley, C. A. (2013). Functional disconnection of the orbitofrontal cortex and basolateral amygdala impairs acquisition of a rat gambling task and disrupts animals' ability to alter decision-making behavior after reinforcer devaluation. Journal of Neurosceicne, 33(15), 6434–6443. https://doi.org/10.1523/jneurosci.3971-12.2013
Zimmermann, Z. J., Watkins, E. E., & Poling, A. (2015). Jeab research over time: Species used, experimental designs, statistical analyses, and sex of subjects. The Behavior Analyst, 38(2), 203–218. https://doi.org/10.1007/s40614-015-0034-5
Acknowledgements
We thank the numerous researchers who helped to collect the original data which was used to support the analysis and simulations in this article.
Funding
This work was supported by the NINDS (R01-NS1109).
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of Interest
The authors have no competing interests to declare that are relevant to the content of this article.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Frankot, M.A., Young, M.E. & Haar, C.V. Understanding Individual Subject Differences through Large Behavioral Datasets: Analytical and Statistical Considerations. Perspect Behav Sci 47, 225–250 (2024). https://doi.org/10.1007/s40614-023-00388-9
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s40614-023-00388-9