Abstract
Given the recent push for universal screening, it is important to take into account how well a screener identifies children at risk for reading problems as well as how screener and sample information contribute to this classification. Picking the best cut-point for a particular sample and screening goal can be challenging given that test manuals often report classification information for a specific cut-point and sample base rate which may not generalize to other samples. By assuming a bivariate normal distribution, it is possible to calculate all of the classification information for a screener based on the correlation between the screener and outcome, the cut-point on the outcome (i.e., the base rate in the sample), and the cut-point on the screener. We provide an example with empirical data to validate these estimation procedures. This information is the basis for a free online tool that provides classification information for a given correlation between screener and outcome and cut-points on each. Results show that the correlation between screener and outcome needs to be greater than .9 (higher than observed in practice) to obtain good classification. These findings are important for researchers, administrators, and practitioners because current screeners do not meet these requirements. Since a correlation is dependent on the reliability of the measures involved, we need screeners with better reliability and/or multiple measures to increase reliability. Additionally, we demonstrate the impact of base rate on positive predictive power and discuss how gated screening can be useful in samples with low base rates.
This is a preview of subscription content, log in via an institution to check access.
Similar content being viewed by others
References
Brace, H. (2003). Stanford achievement test—Tenth edition: Technical data report. Author.
Brown Waesche, J. S., Schatschneider, C., Maner, J. K., Ahmed, Y., & Wagner, R. K. (2011). Examining agreement and longitudinal stability among traditional and RTI-based definitions of reading disability using the affected-status agreement statistic. Journal of Learning Disabilities, 44(3), 296–307.
Catts, H. W., & Petscher, Y. (2018). Early identification of dyslexia current advancements and future directions. Perspectives on Language and Literacy, 44(3).
Catts, H. W., Fey, M. E., Zhang, X., & Tomblin, J. B. (2001). Estimating the risk of future reading difficulties in kindergarten children: A research-based model and its clinical implications. Language, Speech, and Hearing Services in Schools, 32, 38–50.
Catts, H. W., Petscher, Y., Schatschneider, C., Sittner Bridges, M., & Mendoza, K. (2009). Floor effects associated with universal screening and their impact on the early identification of reading disabilities. Journal of Learning Disabilities, 42(2), 163–176.
Gearin, B., Turtura, J., Kame’enui, E. J., Nelson, N. J., & Fien, H. (2018). A multiple streams analysis of recent changes to state-level dyslexia education law. Educational Policy. https://doi.org/10.1177/0895904818807328
Good, R. H., Kaminski, R. A., Smith, S., Laimon, D., & Dill, S. (2001). Dynamic indicators of basic early literacy skills (5th ed.). University of Oregon.
Individuals with Disabilities Education Act, 20 U.S.C § 1400 et seq (2012). https://uscode.house.gov/view.xhtml?path=/prelim@title20/chapter33&edition=prelim. Accessed Jan 2022
January, S.-A.A., & Klingbeil, D. A. (2020). Universal screening in grades K-2: A systematic review and meta-analysis of early reading curriculum-based measures. Journal of School Psychology, 82, 103–122. https://doi.org/10.1016/j.jsp.2020.08.007
Jenkins, J. R., & Johnson, E. (2008). Universal screening for reading problems: Why and how should we do this. RTI Action Network.
Jenkins, J. R., Hudson, R. F., & Johnson, E. S. (2007). Screening for at-risk readers in a Response to Intervention framework. School Psychology Review, 36(4), 582–600. https://doi.org/10.1080/02796015.2007.12087919
Kaminski, R. A., & Good, R. H., III. (1996). Toward a technology for assessing basic early literacy skills. School Psychology Review, 25(2), 215–227.
Kent, S. C., Wanzek, J., & Yun, J. (2019). Screening in the upper elementary grades: Identifying fourth-grade students at-risk for failing the state reading assessment. Assessment for Effective Intervention, 44(3), 160–172. https://doi.org/10.1177/1534508418758371
Meehl, P. E., & Rosen, A. (1955). Antecedent probability and the efficiency of psychometric signs, pattern, and cutting scores. Psychological Bulletin, 52, 194–216.
National Center on Improving Literacy. (n.d.). State of dyslexia. https://improvingliteracy.org/state-of-dyslexia. Accessed Jan 2022
National Center on Intensive Intervention. (n.d.). Academic Screening Tools Chart https://charts.intensiveintervention.org/ascreening. Accessed Jan 2022
Nunnally, J. C., & Bernstein, I. H. (1994). Psychometic theory (3rd). McGraw-Hill.
Petscher, Y., Truckenmiller, A., & Zhou, C. (2016). The earlier assessment for reading success [web application software]. Florida State University.
Riedel, B. W. (2007). The relation between DIBELS, reading comprehension, and vocabulary in urban first-grade students. Reading Research Quarterly, 42(4), 546–567. https://doi.org/10.1598/RRQ.42.4.5
Schatschneider, C. (2006). Reading difficulties: Classification and issues of prediction. [conference paper]. Pacific Coast Research Conference, San Diego, CA.
Schatschneider, C., Petscher, Y., & Williams, K. M. (2008). How to evaluate a screening process: The vocabulary of screening and what educators need to know. In L. M. Justice & C. Vukelich (Eds.), Achieving excellence in preschool literacy instruction (pp. 304–316). The Guilford Press
Spencer, M., Wagner, R. K., Schatschneider, C., Quinn, J. M., Lopez, D., & Petscher, Y. (2014). Incorporating RTI in a hybrid model of reading disability. Learning Disability Quarterly, 37(3), 161–171.
Stevenson, N. A. (2017). Comparing curriculum-based measures and extant datasets for universal screening in middle school reading. Assessment for Effective Intervention, 42(4), 195–208. https://doi.org/10.1177/1534508417690399
Thomas, A. S., & January, S.-A.A. (2021). Evaluating the criterion validity and classification accuracy of universal screening measures in reading. Assessment for Effective Intervention. https://doi.org/10.1177/1534508419857232
Van Norman, E. R., Nelson, P. M., Klingbeil, D. A., et al. (2019). Gated screening frameworks for academic concerns: The influence of redundant information on diagnostic accuracy outcomes. Contemporary School Psychology, 23, 152–162. https://doi.org/10.1007/s40688-018-0183-0
Youman, M., & Mather, N. (2018). Dyslexia laws in the USA: A 2018 update. Perspectives on Language and Literacy, 44(2), 37–41.
Funding
This research was supported by Grant P50 HD052120 for the Eunice Kennedy Shriver National Institutes of Child Health and Human Development and Grant R305B200020 from the Institute of Education Sciences.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare no competing interests.
Additional information
Publisher's note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
About this article
Cite this article
Edwards, A.A., van Dijk, W., White, C.M. et al. Screening screeners: calculating classification indices using correlations and cut-points. Ann. of Dyslexia 72, 445–460 (2022). https://doi.org/10.1007/s11881-022-00261-5
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11881-022-00261-5