Bolstering the Persistence of Black Students in Undergraduate Computer Science Programs: A Systematic Mapping Study

2.1 Eligibility Criteria

There exists an urgent need, as we have argued, to attend to the specificities associated with the experiences of Black students in undergraduate CS programs. To that end, we designed our inclusion and exclusion criteria (see Table 1) with a view towards identifying, foregrounding, and amplifying non-deficit-based empirical studies engaging with Black students in undergraduate CS contexts.

Beginning with the inclusion criteria: First, because English is the language in which all members of the research team are fluent, we opted only to include papers written in English. Second, we opted to include papers published as far back as 2001 to test our hypothesis that the vast majority of non-deficit-based studies focused specifically on Black students in undergraduate CS programs have been published in the past 10 years. Third, in line with our objectives outlined above, we opted only to include papers containing primary research results. During the search process, it should be noted, we made note of all literature reviews which satisfied most or all the remaining eligibility criteria. These papers were addressed in Section 1.3, Related Work. Fourth, rather than only including studies involving current undergraduate students, we broadened this inclusion criterion to permit the inclusion of studies engaging with former undergraduate students as well. For example, it is doubtless that Rankin et al.’s [7] “Black Women Speak: Examining Power, Privilege, and Identity in CS Education” contains findings capable of informing efforts to bolster the persistence of Black students in undergraduate CS programs. However, only one of Rankin et al.’s 24 participants is an undergraduate student. The other 23 participants include academics, professionals, and graduate students, each of whom offer insights concerning their experiences in undergraduate computing programs. Thus, we broadened the fourth inclusion criterion to ensure that we would be able to capture insightful studies like this one in our sample.

Turning to our exclusion criteria: First, as we are primarily interested in papers published in peer-reviewed journals, we opted to exclude books, book chapters, and papers not subject to peer review. This is not to suggest, of course, that books do not contain insights capable of bolstering the persistence of Black students in undergraduate CS or computing programs. Rather, this first criterion is reflective of our interest in the current state of this literature in peer-reviewed journal and conference papers. We acknowledge, however, that this exclusion criterion may result in the exclusion of important accounts by Black authors. Consequently, we may have reproduced the very exclusionary practices we hope to stand against. Accordingly, we plan to revisit this dimension of the literature in a future study. Second, in keeping with our commitment to specificity (see Section 1.1, Rationale and Significance), we opted to exclude papers in which undergraduate CS education was not the principal focus (e.g., [51–53]). Third, along similar lines, we excluded papers that were not principally focused on Black students (e.g., [47–49]). That is, Black students did not have to comprise the entirety of the researchers’ sample, but they had to be the primary focus.

Finally, we excluded studies taking a deficit-oriented approach to framing or addressing the underrepresentation of Black students in undergraduate CS or computing programs. As noted above, deficit-oriented studies are those that are predicated on the assumption that Black students are underrepresented in STEM fields because they are missing something, such as a lack of motivation, intelligence, literacy, interest, or information [34]. Accordingly, deficit-oriented studies typically assess the effectiveness of interventions designed to address these apparent deficiencies. To assess whether a given study was deficit-oriented, we employed Shaun Harper's anti-deficit framework for research involving students of colour [97]. That is, if a given paper's research questions more closely resembled Harper's [97] sample of deficit-oriented questions than they did the associated anti-deficit reframing, we classified the paper as a deficit-oriented study. For example, Cummings et al. [68] assess whether a conversational agent and virtual mentor is capable of increasing Black undergraduate students’ interest in pursuing professorships. At issue for Cummings et al., in other words, is the question: Why do so few Black students pursue professorships in STEM fields? This question more closely resembles one of Harper's [97, p. 68] deficit-oriented sample questions (i.e., “Why do so few Black male students enroll in college?”) than it does the corresponding anti-deficit reframing of this question (i.e., “How were college aspirations cultivated among Black male undergraduates who are currently enrolled?”). In other words, an anti-deficit reframing of Cummings et al.’s question would read something like: How were professorial aspirations cultivated among Black male professors in STEM fields? Accordingly, Cummings et al. [68] was excluded during the database-driven search described below. Ultimately, three papers were excluded by virtue of having been classified as deficit-oriented studies: one during the database search phase, and two during the snowballing phase.

It should be noted that both the inclusion and exclusion criteria were applied sequentially, from top to bottom. For this reason, not every paper was assessed for its ability to inform efforts to bolster the persistence of Black students in undergraduate computing programs because many papers failed to satisfy the preceding inclusion criteria. If a paper was published in 1999, for instance, there was no need to assess whether it satisfied this more stringent criterion. Along similar lines, if a paper was not peer reviewed, there was no need to assess whether it qualified as a deficit-oriented study. As such, the true number of deficit-oriented studies excluded during this review was likely much higher.

2.2 Information Sources

To capture studies from a range of disciplines with the potential to satisfy the eligibility criteria outlined above, we selected a total of seven databases (see Table 2) in which to conduct keyword searches. We selected these databases with a view towards ensuring broad coverage of papers in the fields of computing (ACM Digital Library), engineering (IEEE Xplore and Engineering Village), science (Scopus), psychology (APA PsycInfo), and education or social sciences (EBSCOhost Web and ERIC).

In addition to the databases listed above, we used Connected Papers (https://www.connectedpapers.com/) to snowball records meeting the eligibility criteria. Connected Papers is a very performant tool that supports the implementation of robust search strategies [91] and that is increasingly used by the scientific community to perform snowballing. As Solomons [91] points out, Connected Papers is very efficient at finding strongly connected studies by leveraging a similarity metric that exploits the concepts of Co-citation and Bibliographic Coupling. We therefore deemed it appropriate to perform snowballing during our search. After snowballing, we used a computer science bibliography website called DBLP² (https://dblp.org/) to assess the completeness of our search by cross-referencing the digital libraries of two experts identified during our search against the eligibility criteria. We also contacted experts to ask them to assess our list of studies and to suggest additional studies we may have missed. Following Harry Collins [100], we define experts for the purposes of this study not as generalists but as specialists – that is, as the small handful of researchers—i.e., the ‘core set’—who are at the cutting edge of research into a particular research problem. The experts in the context of this study, in other words, are either: (1) the most prolific asset-oriented researchers on the persistence of Black students in undergraduate CS programs; or (2) high-profile researchers specialized in diversity in the computing area. Ultimately, however, no papers were added to our sample through the expert-based searches.

2.3 Search Strategy

In searching the databases listed above, we employed the following Boolean search string, which consists of three sets of keywords joined by the “AND” operator:

("black students" OR "black people" OR "black women" OR "black females" OR "black men" OR "black males") AND ("computer science" OR "computer science education" OR "CS" OR "CS education" OR "computing") AND (success OR achievement OR underrepresentation OR participation OR equity OR diversity OR inclusion OR "asset based" OR persist*)

We designed this search string with a view towards striking an optimal balance between the need to attend to the specificity of the subject matter on the one hand, and our desire to ‘cast a wide net’ with our search on the other. With that in mind, we structured the first set of keywords with specificity in mind by ensuring that the records returned included specific references to Black students. The second and third set of keywords were designed to cast a much wider net. For example, even though the second set of keywords were designed to capture CS-centric records, we opted not to pair these keywords with terms like “college”, “university”, or “higher education” to ensure that, say, studies engaging with current computing professionals would be captured as well (see Section 2.1, Eligibility Criteria). Along similar lines, the third set of keywords includes a range of terms variously used in equity-, diversity-, and inclusion-oriented and asset-based studies. Taken together, we believe we struck an appropriate balance between specificity and breadth.

We conducted keyword searches in our chosen databases on October 5, 2022. As each of our chosen databases has its own interface, features, and optional parameters, the search terms listed above had to be adapted accordingly, as summarized by Table 3. To verify the reproducibility of our database-driven search, we consulted this table to complete the search again on October 28, 2022, yielding identical results in the process.

In addition to the parameters outlined above, it should be noted that we opted to use advanced search functions where available. We searched within abstracts for each database save for IEEE Xplore, where an abstract search returned limited results. For this reason, we searched IEEE Xplore using metadata. We considered completing metadata searches for the other databases as well but discovered that the other databases did not offer this functionality. To ensure reproducibility beyond 2022, we limited search results by publication date from 2001 to 2022. Where permitted to specify the month, we limited search results by publication date from January 2001 to October 2022. We also specified additional parameters which corresponded with our eligibility criteria (e.g., limiting search results to peer reviewed papers, excluding books, etc.) wherever available.

In order to ensure the completeness of our search results, we used snowballing and an expert-based search. First, after removing duplicate search results and applying inclusion and exclusion criteria to the remaining records, we used Connected Papers to snowball all eligible papers. Second, after removing duplicates from the newly imported records, and applying the inclusion and exclusion criteria to those that remained, we snowballed all eligible papers once again. Third, we repeated this snowballing process until it yielded no additional eligible papers. Finally, over the course of our search, we identified Jakita Thomas and Yolanda Rankin as experts in the persistence of Black students in undergraduate computing. Thomas and Rankin, who are credited as author or co-author of six and five papers in our sample respectively, are the most prolific researchers in our sample (see Figure 1). LaVar Charleston and Jerlando Jackson, it should be noted, were also identified as prolific researchers, having been credited as author or co-author of four and three papers in our sample, respectively. Ultimately, however, we opted not to consider Charleston or Jackson as experts for the purposes of the expert-based search because the median publication year for papers authored by them in our sample is 2013.5 and 2013, respectively. This suggests that Charleston and Jackson are no longer actively conducting research in this area and therefore cannot be considered experts for the purposes of our expert-based search. By comparison, the median publication year for Thomas's and Rankin's papers in our sample is 2020, which is also to say that they are at the forefront of asset-based scholarship involving Black students in CS education. In lieu of contacting Thomas and Rankin to ensure the completeness of our search results, however, we initially used DBLP to cross-reference our list of papers against their digital libraries. We initially made this decision not only in the interest of efficiency, reproducibility, unbiasedness, but also because we are cognizant of, and did not want to contribute further to, the burdens associated with the so-called diversity tax [69]. Since using DBLP to conduct the expert-driven search yielded not additional primary study, we therefore resolved to contact (by email) four experts and to ask them to assess the list of primary studies included in our systematic mapping study, and to propose potential additional primary studies to include in our systematic mapping study. Hence, we contacted: (1) Thomas and Rankin who are experts in the persistence of Black students in undergraduate computing; and (2) Alexander Serebrenik, and Ronnie de Souza Santos who are experts in diversity in computing (more specifically in the software engineering field). Two of these experts replied, assessed our list of primary studies, and suggested additional studies that may be relevant to our map.

Fig. 1.

View Figure

2.4 Selection Process

To identify and select primary studies, we employed a three-phase, 21-round approach, as summarized by Table 4 below. When completing the selection process, we used EndNote as our reference manager.

In phase one, which consisted of three rounds, we (1) searched the selected databases using the parameters outlined in the preceding section; (2) manually screened the results for duplicate records (e.g., identical papers found in multiple databases) and ineligible paper types (e.g., book chapters); and (3) applied the inclusion-exclusion criteria to the remaining records to select relevant papers. One member of the research team (i.e., a postdoctoral fellow [second author]) was responsible for searching the selected databases, screening the results, applying the inclusion-exclusion criteria, and documenting their decisions at every step along the way. A second member of the research team (i.e., a faculty member [lead author]) was responsible for selecting a random sample of records to independently validate their inclusion or exclusion. In the event of a disagreement concerning how to classify a given paper, the research team agreed to expand the validation sample to include the remaining records, and, if needed, to use the Delphi process [70–72, 106] to resolve the disagreement. After the validation process was completed, we proceeded to phase two. Noteworthy, the Delphi process is a widely adopted iterative method that allows gathering data from experts within their domain of expertise to aggregate opinions on a given topic and reach a consensus [70, 106].

In phase two, which consisted of 12 rounds, we completed four iterations of snowballing. In the first iteration, we (4) used Connected Papers to snowball the papers (studies) selected during phase one; (5) manually screened the results for duplicate records (e.g., studies considered during phase one) and ineligible studies (e.g., studies published before 2001); and (6) applied the inclusion-exclusion criteria to the remaining records to select relevant studies. In the second iteration of this cycle, we (7) snowballed the studies selected during the previous round; (8) manually screened the results for duplicate records and ineligible studies; and (9) applied the inclusion-exclusion criteria to the remaining records to select relevant studies. The third iteration of this cycle, which included rounds 10–12, resulted in the selection of additional relevant studies. It was not until the fourth iteration of this cycle—which included rounds 13–15—that no additional relevant studies could be found. One member of the research team (i.e., a postdoctoral fellow) was responsible for snowballing the studies in our initial sample, screening the results, applying the inclusion-exclusion criteria, identifying experts in the field, and documenting their decisions along the way. In line with the validation approach used during the previous phase, a second member of the research team (i.e., a faculty member) was responsible for selecting a random sample of records examined during this phase to independently validate their inclusion or exclusion. As with the above, the research team agreed to expand the validation sample and, if needed, to use the Delphi process to resolve any potential disagreements that arose. Once the snowballing results were validated, and the research team approved the chosen experts, we proceeded to phase three.

In phase three, which consisted of six rounds, we (16) imported the digital libraries of Yolanda Rankin and Jakita Thomas. These digital libraries are online electronic libraries that can be accessed online (on Google) by typing “DBLP” followed by the name of the expert; (17) manually screened the results for duplicate records (e.g., studies considered during phase one or two) and ineligible studies (e.g., studies that are not peer reviewed); (18) applied the inclusion-exclusion criteria to the remaining records to select relevant study; (19) emailed four experts to request potential additional studies we may have missed; (20) manually screened the additional suggested studies; and (21) applied the inclusion-exclusion criteria to the remaining records to select relevant studies. One member of the research team (i.e., a postdoctoral fellow) was responsible for importing records from Rankin's and Thomas's digital libraries, screening the results, applying the inclusion-exclusion criteria, and documenting their decisions along the way. Once again, a second member of the research team (i.e., a faculty member) was responsible for independently validating a random sample of records. In the event of a disagreement, the research team agreed to expand the validation sample and, if needed, use the Delphi process to reach a consensus. Once the results of the expert-driven search were validated, we proceeded to the data-collection phase of this study.

2.5 Data Collection

During the data-collection phase of this study, we completed a full-text review of all studies in the sample to identify and document (1) key study characteristics including research methods, theoretical frameworks, and research participants; (2) research findings; and (3) recommendations. To that end, one member of the research team (i.e., a postdoctoral fellow) was responsible for preparing a brief set of point-form notes for each study in the sample. This researcher was instructed by other members of the research team including the lead author (i.e., a faculty member) to capture key points for each primary study (i.e., key study characteristics, findings, and recommendations) with our research questions firmly in mind. That is, we did not attempt to create an exhaustive set of notes for each study. At the same time, if there was any doubt as to whether or not a given point was relevant, the researcher was instructed to err on the side of assuming its relevance. The researcher was also instructed to include page references with their notes to permit other members of the research team to consult the underlying study if necessary. These primary study notes would later serve as the basis for the data-analysis phase of this study.

To validate the primary study notes, (i) a meeting was held to discuss the key points for each primary study (i.e., key study characteristics, findings, and recommendations) and to suggest the investigation of some additional key points; (ii) one member of the research team (i.e., a postdoctoral fellow) then iterated over each primary studies to further extract the additional key points mentioned in the meeting; (iii) a second member of the research team (i.e., a faculty member) then independently selected a random validation sample of three primary studies, completing a full-text review of the selected studies, and validated the key points specific to each of these three primary studies. The research team agreed to address any significant discrepancies by expanding the validation sample and, if needed, using the Delphi process [70–72] to reach a consensus concerning the remaining studies.

2.6 Data Analysis

During the data-analysis phase of this study, we (i.e., the authors of this study) (1) reviewed primary study notes, tabulated study characteristics, and created master lists of findings and recommendations; (2) reconciled and consolidated tables and lists; and (3) arrayed these tables and lists into matrices to address our research questions.

First, one member (i.e., a faculty member) of the research team was responsible for independently reviewing a random sample of the primary study notes and consulting the underlying study for additional clarity where necessary.

Second, two researchers (i.e., a faculty member and a postdoctoral fellow) were then responsible for meeting to discuss tables, lists, and categories with a view towards collaboratively reconciling the potential differences spotted in them [73, 74].

Third, after finalizing the study characteristics table, thematic categories, findings, and recommendations, we arrayed the findings and recommendations into a series of matrices to address our research questions. This is useful to present data in a tabular form in accordance with the review research questions [108]. Presenting data in a tabular form is a practice that is strongly recommended when conducting systematic reviews [108]. We created three matrices to address RQ1 – that is, what factors are associated with Black students’ persistence in undergraduate CS programs? The first matrix is designed to map thematic categories (column A) and their constituent persistence factors (column B) to all primary studies (the remaining columns) in our sample. The second and third matrices (i.e., Tables 8 and 9 that are available in the Appendix) map thematic categories and persistence factors to two distinct subsets of primary studies – that is, those engaging exclusively with Black women and those engaging with Black women and men. In the second and third matrices, persistence factors with no connection to the associated subset of studies were removed. In all three matrices, thematic categories are sorted in descending order based on the number of constituent persistence factors. Where two categories contained the same number of persistence factors, the category with the greater number of connections to the primary studies was placed on top. The persistence factors within each category were sorted in descending order by the number of connections to primary studies. Finally, we took the same approach to creating a matrix to address RQ2 – that is, what recommendations have been made to further bolster Black students’ persistence in undergraduate CS education programs? Rather than organizing these recommendations by thematic categories, however, we opted to group them by the target audience for each recommendation.

3.1 Study Selection

Using the search and selection process outlined in the previous section, we identified a total of 16 primary studies, as summarized by Figure 2.

Fig. 2.

View Figure

The database-search phase of the selection process yielded a total of 132 records. After manually excluding 78 duplicate records, we assessed the eligibility of the remaining records. Only 10 of these 54 studies satisfied the eligibility criteria. We then snowballed these 10 studies to assess whether any related studies met the eligibility criteria. After three iterations of this snowballing process, we added a total of 6 new studies to our sample. The fourth snowballing iteration did not result in the identification of any additional eligible studies. Accordingly, we assessed the completeness of our search and selection process by cross-referencing the digital libraries of Jakita Thomas and Yolanda Rankin against our eligibility criteria and existing sample of studies. We also contacted four experts to get potential additional studies we may have missed. Ultimately, however, the expert-based search did not result in the identification of any additional eligible studies.

3.2 Study Characteristics

Our analysis of the characteristics of the 16 primary studies which together constitute our sample (see Table 5) produced several interesting insights.

First, despite designing our eligibility criteria to include studies published as distantly as the year 2001, the oldest primary study in our sample was published in 2012. More strikingly, the median publication year for the studies in our sample is 2020. Simply put, the studies in our sample are temporally skewed towards the present, as illustrated by Figure 3.

Fig. 3.

View Figure

Second, though all studies in our sample focused specifically on Black people in computing, only five of 16 studies (i.e., 31.25%) focused on both Black men and women. The remaining 11 studies (i.e., 68.75%) focused specifically on Black women. In other words, none of the studies in our sample focused specifically on Black men.

Third, though four studies in our sample did not articulate a theoretical framework, the vast majority (i.e., 12/16 or 75%) did. Figure 4 provides a breakdown of the particular theories most commonly employed in our sample. Intersectionality theory was used in a total of seven studies, including qualitative, quantitative, and mixed-methods studies, and it was often used in conjunction with other theories – including, most commonly, standpoint theory followed by black feminist thought. Intersectionality theory was only used in studies focused specifically on Black women. Culturally relevant pedagogy theory was used in three primary studies but was not used in conjunction with other theories.

Fig. 4.

View Figure

Fourth, 10 of 16 studies in our sample (i.e., 62.5%) employed qualitative methods. Nine of these 10 studies utilized a theoretical framework, with Rankin et al., 2019 being the sole exception. The six remaining studies were evenly split between quantitative methods (i.e., 3/16, or 18.75%) and mixed methods (i.e., 3/16, or 18.75%). Two of the three mixed-methods studies in our sample articulated a theoretical framework. Ross et al. 2020, on the other hand, was the only quantitative study to articulate a theoretical framework.

Finally, the studies in our sample were almost evenly split between conference papers (i.e., 9/16, or 56.25%) and journal articles (i.e., 7/16, or 43.75%). As illustrated by Figure 5, moreover, these papers were published in a variety of venues. Though several of these venues are explicitly geared towards equity-, diversity-, and inclusion-oriented scholarship (e.g., the Conference on Research in Equity and Sustained Participation in Engineering, Computing, and Technology, or R.E.S.P.E.C.T., and the Journal of Diversity in Higher Education), others are focused on computer science education more generally (e.g., ACM Technical Symposium on Computer Science Education and ACM Transactions on Computing Education).

Fig. 5.

View Figure

3.3 Persistence Factors (RQ1)

In this section, we turn finally to our research questions, beginning with RQ1 – that is, what factors are associated with Black students’ persistence in undergraduate CS programs? Based on data collected from the primary studies included in our map, we identified 13 persistence factors across four thematic categories, as summarized by Table 6. In what follows, we provide a brief overview of the persistence factors in each thematic category.

3.4 Recommendations (RQ2)

In this section, we turn to RQ2 – that is, what recommendations have been made to further bolster Black students’ persistence in undergraduate CS education programs? Based on the data collected from the primary studies in our sample, we identified 26 recommendations across six audiences or stakeholder groups, as summarized by Table 7. In what follows, we offer an overview of the recommendations for each stakeholder group.

4.1 Implications for Future Research

As evidenced by the results outlined above, Black students must employ a range of persistence, coping, or survival strategies to succeed in undergraduate CS programs in spite of the ongoing experience of systemic anti-Black racism in computing contexts more generally. Though it is tempting, as Rankin et al., 2021b [7] argue, to applaud the ingenuity, determination, tenacity, and grit of these students, it would be counter-productive to do so, as this “does not address the oppressive systems that require coping mechanisms that enable Black women to survive—but not thrive—in computing” (p. 14, emphasis added). It is critical, in other words, that “we do not normalize the hoops that Black women jump through just to have the same opportunities as their non-Black counterparts” (p. 14). Indeed, to persist in computing, as Yamaguchi & Burge, 2019 [4] point out, some Black women develop a “singular focus on achievement”, but this often comes at the cost of immense “pressure and stress” as they contend with the tremendous burden of having to represent “a whole subgroup” (p. 225). For this reason, we follow Rankin et al., 2021b [7] in arguing that “we must break down the systems that require coping and survival mechanisms” (p. 14).

The list of recommendations outlined in the preceding section offers a good starting point for this kind of systemic reform, but this list of recommendations is incomplete for several reasons. Most notably, even though non-deficit-oriented studies concerned specifically with the persistence of Black students in undergraduate CS programs have been on the rise in recent years, this research problem remains severely understudied. Indeed, despite designing our eligibility criteria to permit the inclusion of studies published as far back as 2001, the oldest study in our sample was published in 2012, and the median publication year for the studies in our sample is 2020. Moreover, there are no studies in our sample published between 2001 and 2011, or 2015 and 2017. Likewise, despite supplementing our database-driven searches with snowballing and an expert-based search, our final sample contained just 16 primary studies. It could be argued, of course, that this is reflective of a systematic mapping study that was much too narrow in its scope. In our view, however, this outcome is more accurately interpreted as a demonstration of the extent to which this research problem remains severely understudied. In order, then, to generate a more comprehensive list of recommendations for creating meaningful, systemic change for Black students in CS education, a more sustained, intentional, theoretically informed, and asset-based program of research is required.

Though London et al.’s [59] systematic mapping study ultimately reached a similar conclusion—i.e., that “there is a need for more research that is focused on the unique challenges and opportunities associated with this demographic” (p. 237)—we argue that London et al.’s broad approach obscured the true extent of this need. That is, because London et al. focused not just on Black students in undergraduate CS education, but also on marginalized students more generally in engineering and STEM education from K-12 education to graduate school and beyond, the extent of the gap thus revealed was understated. This problem is compounded by the presence of deficit-based studies in London et al.’s sample, which further understates the urgent need for additional research in this area. By accounting for these considerations, in other words, our systematic mapping study reveals the true extent of the gap to be addressed, making a distinct contribution to the literature in the process.

In line with the above, we follow Rankin et al., 2020 [31] in calling on CS education researchers to “conduct more authentic research” involving Black students in general, and Black women in particular, in computing – that is, research which “chronicles their successes and failures, the sociological factors that impact their ability to remain in the field, or in some cases, contribute[s] to them deciding to leave the field” (p. 6). Though researchers may be tempted to chase statistical representativeness by grouping Black CS students with CS students from other underrepresented backgrounds, we argue that the need for empirical specificity in this instance outweighs the usefulness of any findings derived from a representative sample of an undifferentiated mass of students. Ross et al., 2020 [50] argue, along similar lines, that researchers should “continue to expand, intentionally, into critical theory to design and analyze data” (p. 12). Though critical theoretical frameworks tend to be better suited to qualitative research methods, it is possible to employ these frameworks in quantitative and mixed-methods research as well. Indeed, Ross et al., 2020 [50] notes that intersectionality, in particular, “can be leveraged in both qualitative and quantitative research designs to seek knowledge to grow and diversify […] CS” (p. 13). After all, Ross et al. contend, given the enormous complexity of the problem at hand, “we need many different approaches to inquiry […] to unpack and understand the landscape of computing for those least represented in computing” (p. 13). As Slaton and Pawley [98] have argued, however, researchers’ predilection for quantitative methods in general, and statistical representativeness in particular, may be responsible for obscuring our collective understanding of the experiences of ‘small n’ demographics in STEM fields, like Black students in undergraduate CS programs. Lucy Arellano [99] argues, more pointedly, that prevailing statistical practices in higher education research are responsible for perpetuating systemic inequities and White supremacy. In order, then, to combat persistent, systemic anti-Black racism in computing, we suggest that a greater emphasis on qualitative methods is needed in order to capture the complex experiences of Black students in undergraduate CS programs [102]. No matter the approach chosen, we believe it is imperative that CS education researchers reject deficit-oriented approaches to engaging with Black students, families, and communities in favour of one that intentionally foregrounds and privileges Black perspectives.

Additionally, the results of our map highlight the value of an intersectional approach to addressing the underrepresentation of Black students in CS education programs. Although Black students are underrepresented in CS education programs, the percentage of Black women is remarkably low. The extremely low representation of Black women brings to the fore the intersection of their identities, which interact with systems of oppression [87]. Put differently, Black women are underrepresented because they are “Black” and “women”. The intersection of race and gender reinforces each other [85, 86], leading to their underrepresentation in CS education programs. Though we agree wholeheartedly with the various primary studies in our sample which emphasize the importance of attending specifically to the intersectional experiences of Black women in CS [4, 6–8, 24, 31, 50], our analysis of the characteristics of the primary studies in our sample also revealed the existence of a gap in empirical research focused specifically on the experiences of Black men in undergraduate CS programs. That is, none of the primary studies in our sample focused principally on Black men. Accordingly, we call on CS education researchers not just to conduct additional research involving Black women in CS, but also to attend to the particular systemic inequities faced by Black men in undergraduate CS programs.

4.2 Limitations

Finally, it should be noted that, despite having been carried out in accordance with a systematic methodology, our mapping study is associated with a few limitations.

First and foremost, it is doubtless that the composition of our research team shaped our approach to carrying out, and—by extension—the findings to arise from, this systematic mapping study. That is, we are an interdisciplinary team of researchers which includes computing scientists, social scientists, and experts in education. Accordingly, though we adhered to the prescribed guidelines for conducting systematic reviews [64–67], the inclusion of social scientists in our research team may impact the reproducibility of our research findings. For example, social scientists may be better attuned to identifying the features of deficit-based research. As such, our particular application of the provided eligibility criteria may differ from that of a research team with a different (inter)disciplinary composition. This is not to suggest, of course, that social scientists are better equipped to conduct this study, or vice versa. Rather, we are pointing this out to recognize the inescapable subjectivities associated with expert judgement, especially in interdisciplinary contexts. This is not the same as conceding, however, that this study is not objective, or that its findings are not valid [80, 81]. Still, to mitigate this limitation, we asked four experts in diversity in computing (education) to assess the studies included in our map and to propose potential studies our search may have missed. The feedback we received from two of them strengthened our confidence in the completeness of our study.

Second, as a result of our methodological emphasis on empirical specificity, we intentionally excluded studies which (a) lumped Black students together with students from underrepresented backgrounds more generally; (b) grouped Black students in undergraduate CS programs with Black students in other STEM fields; and (c) took a deficit-based approach to framing or addressing the underrepresentation of Blacks in computing. As a result of our decision to focus solely on studies that are concerned principally with Black students, we may have missed some important insights from studies involving not just Black students, but also students who are racialized, gendered, or minoritized more generally (e.g., [17, 38, 47]). Along similar lines, our emphasis on undergraduate CS education means that we may have missed out on insights offered by studies concerned only with Black students in graduate school (e.g., [82]) or K-12 computing education contexts (e.g., [83]). Consequently, it is possible that the persistence factors and recommendations identified in this paper are incomplete. For this reason, future mapping studies should consider taking a similar approach to identifying persistence factors and recommendations in both K-12 and graduate computing education contexts.

Furthermore, one element of our search strategy may have slightly reduced the scope of our search results. More specifically, when snowballing studies, we used Connected Papers, which only returns around 40 of the most closely related studies for each snowballed study. It is possible, therefore, that this limitation prevented us from identifying a few additional related studies. In our experience, however, relevant papers or near-misses only tended to appear in the first 10–15 studies returned by Connected Papers.

When developing the protocol that we describe in Section 2, we decided to meet and discuss our findings and opinions towards collaboratively reconciling the potential differences spotted in them. Besides, in case of significant disagreements between researchers, we also decided to use the Delphi method to reach consensus. Still, the Delphi process is not necessarily straightforward when completing a systematic mapping study. Besides, given the relatively small number of primary studies captured by our systematic mapping study, meetings proved to be the most efficient way to collaboratively reconcile the potential differences in our opinions. They therefore allowed us to reach a consensus efficiently. Nevertheless, in future work, we will explore the use of other statistical techniques (e.g., agreement indices like Cohen's Kappa, and Fleiss's Kappa [107]) to compute agreement based on inter-raters’ ratings and reach a consensus faster.

Finally, as mentioned in Section 2, five members of the research team developed/revised the different versions of the protocol used in this systematic mapping study until the most optimal one was defined. Two members of the team also made several checks at each key phase of the study to ensure the protocol was properly followed throughout the completion of study. Still, one member of the research team (i.e., a postdoctoral fellow) was responsible for completing most of the search and selection process, potentially impacting the validity of the results thus generated. As noted earlier, we accounted for this threat to validity by ensuring that all decisions made throughout this process were thoroughly documented, and by tasking a second member of the research team (i.e., a faculty member) with randomly validating a sample of records included or excluded during each round and phase of the search and selection process. Though we are confident that this random validation process sufficiently addressed this validity threat, it is doubtless that this threat could have been reduced further by tasking two researchers with independently completing the search and selection process. Ultimately, however, we opted to employ the random validation process instead because we determined that the alternative would constitute an inefficient use of the limited resources available to us. We opted, instead, to direct these resources towards the data-collection process, which we associated with a higher validity threat. Indeed, despite assigning two researchers to collect and review data from the primary studies, it nevertheless remains possible that the process of collecting data from the primary studies to address the research questions was skewed by the particular researchers assigned to this task. This is, of course, an ever-present risk associated with qualitative, thematic analyses more generally, and we are confident that we adequately accounted for the associated validity threats in the design of this study.

4.3 Additional Considerations

As we stressed in Section 4.2, one of the limitations of our work is that it focuses on undergraduate CS education, which means it may have therefore missed out on insights offered by studies concerned only with Black students in graduate school or K-12 computing education contexts, among others. To mitigate these issues and further ensure the success of Black students in CS undergraduate and graduate programs, among others, we think it is crucial that future work on broadening strategies also focus on the factors specified below. This may help tackle the “Diversity crisis” that hampers the CS area [90, 95].

5.1 Protocol

As stated earlier, several members of the research team developed/revised the different versions of the protocol used in this systematic mapping study until the most optimal one was defined. In the initial version of the protocol, the search strategy consisted in relying on (1) four databases to complete the database-driven search; and (2) the manual search to identify additional primary studies that were missed when completing the database-driven search. After the first round of revisions made by the reviewers on our first manuscript submitted in March 2022 to the ACM TOCE journal and the concerns these reviewers raised regarding some strategies used to conduct our study, we decided to amend the protocol accordingly. As explained in Section 2, our protocol now involves searching candidate studies based on three search techniques: database-driven search, snowballing, and expert-based search. Besides, our database-driven search now relies on seven databases. These databases respectively belong to the fields of computing, engineering, science, psychology, and education or social sciences. We used the revised protocol to redo our study from the beginning and to report the so-obtained study.