Ranking resilience: assessing the impact of scientific performance and the expansion of the Times Higher Education Word University Rankings on the position of Czech, Hungarian, Polish, and Slovak universities

Abstract

University rankings are gaining importance worldwide. While the top-ranked universities are the subject of numerous academic publications, very little literature is available on universities in the lower two-thirds of the ranking list. In this study, we analysed the sensitivity of year to year changes in position in the Times Higher Education World University Rankings (THE) of universities from the Czech Republic, Hungary, Poland, and Slovakia using hypothetical scenarios of potential changes in their research outputs. A regression model was developed that quantifies the effect of the inclusion of institutions that are new to ranking as well as dropouts on the ranking list. We also developed a method to allow customized sensitivity analysis of the subsequent year's rank from an institution's projected research output, given its current rank and research output, assuming that other ranking indicators remain unchanged over 1–2 years. It should be noted that most universities in the region have ranked below 800, and while research output has been increasing, it was observed that ranks worsened for most universities from these four countries. The Field-Weighted Citation Impact and the total volume of publications had the highest effect on rankings, while the proportion of Scimago Q1 publications did not have a consistent effect on the projected next year's ranks. In most universities, the growth in research performance indicators were below the level that would be required to offset the rank changes due to the inclusion of new entrants in ranking. The findings of this research suggest that universities have to consider complex strategies to improve their ranking focusing on other activities than research such as reputation, internationalization, or industry income. With due caution, our results might be generalized to most of the universities below 800 in the THE. The rank prediction tool presented in this article is available online at https://hecon.uni-obuda.hu/en/rank-calculator/.

Introduction

University rankings attract global attention by serving as a reference point for students (Dearden et al., 2019; Wut et al., 2022). High rankings are recognised by governments and partner institutions, positively influencing student recruitment, research funding, and global partnerships (Anowar et al., 2015; Grewal et al., 2008). Despite some critical voices (Thorp, 2023; Vernon et al., 2018), improving university rank positions has become part of the political agenda in many countries across Asia or the European Union (Dong et al., 2020) (Whittle & Rampton, 2020) (Galgánková, 2020; Kowalska et al., 2018; Łącka & Brzezicki, 2020; Lannert & Derényi, 2021). The resulting competition drives innovative practices and investments in research, infrastructure, and facilities to maintain their top ranks (Huenneke et al., 2017). Governments often provide financial incentives for the attainment of top-tier rankings (Benito et al., 2019; OECD, 2007; Marope et al., 2013), motivating universities to launch higher educational excellence programs and improve their ranking positions (Taylor & Braddock, 2007).

Higher education excellence is shaped by a variety of economic, societal and cultural factors (Pietrucha, 2018; Tijssen & Winnink, 2018). The best-known ranking systems, Quacquarelli Symonds (QS) and Times Higher Education (THE) (Ciubotaru, 2022) are based on indicators of research reputation, the learning and teaching environment, citations, internationalisation, and industry income (QS, 2021; THE, 2022). There is a wealth of research on modelling global university ranks using their performance indicator components (Agarwal & Tayal, 2022; Anahideh & Mohabbati-Kalejahi, 2022; De Luna Pamanes et al., 2020; Kaycheng, 2015; Liu et al., 2022; Tabassum et al., 2017), or independent variables including financial, scientific, staff-related metrics, or past ranks (Cantwell & Taylor, 2013; Marconi & Ritzen, 2015; McAleer et al., 2019; Pride & Knoth, 2018) The existing applied techniques included a range of regression(Cantwell & Taylor, 2013; De Luna Pamanes et al., 2020; Kaycheng, 2015; Liu et al., 2022; Marconi & Ritzen, 2015; McAleer et al., 2019) and machine-learning models (Agarwal & Tayal, 2022; De Luna Pamanes et al., 2020; Liu et al., 2022; Popoola et al., 2018; Tabassum et al., 2017) A number of studies suggest that the performance indicators of THE ranking are highly correlated, and university ranks are mainly predicted by two factors: academic reputation and citations (De Luna Pamanes et al., 2020; Kaycheng, 2015).

University rankings are often analysed in the context of adjacent competitors of universities that are considered close competitors or peers (Anahideh & Mohabbati-Kalejahi, 2022; Grewal et al., 2008). In our research, we analyse rankings of universities of the Visegrad Group (we refer to V4 in the text) countries, namely Poland, Hungary, the Czech Republic, and Slovakia (Table 1) (Schmidt, 2016). Most institutions in this region occupy positions below top 800 in the World University Rankings (Vašenda, 2019). The V4 countries started to invest increasingly in higher education and research, aiming to enhance the scientific output, internationalisation, and ranking positions of their universities (‘V4 UniVisibility Program’, 2021) While improving visibility on the global reputation surveys, changing staff composition, internationalisation or building industrial relations may take years, changes in scientific output can impact ranking positions on the short-term (Robinson-Garcia et al., 2019). Being viewed as the most impactful short-term strategy to improve their ranking positions, most V4 universities managed to increase their publication output over the past years. Despite such efforts, the ranking positions of V4 universities have shown an almost uniformly declining trend over the same time period. Hence, understanding the sensitivity of ranking positions to changes in the scientific performance may provide valuable insights to decision-makers when they develop strategic targets for their institutions.

Table 1 Characteristics of examined countries

While research on the highest-ranked universities is abundant (Cantwell & Taylor, 2013; De Luna Pamanes et al., 2020; Kaycheng, 2015; Szluka et al., 2023), analysis on the lower end of the list is scarce. Top positions on the ranking lists are relatively stable (Grewal et al., 2008; Safón, 2019), but exact positions below rank 600 are volatile and difficult to predict (Liu et al., 2022; Osipov et al., 2020). The number of universities ranked by THE increased from 200 in 2011 to 1800 in 2023 (World University Rankings, 2023). Several reputational elements, such as the Matthew effect or halo effect have stabilising effects on the positions of top-ranked universities (Grewal et al., 2008; Safón, 2019), while the positions of lower ranked institutions are challenged by both their incumbent competitors and new entries on the list. Entering the competition for top positions in global university rankings is a priority in many countries’ higher educational policy (Lee et al., 2020), but there is limited research on quantifying the effect of new entrants on ranking positions, which is beyond the control of incumbent universities. Furthermore, while reputation seems to be a key factor in determining the rankings of top universities (Grewal et al., 2008; Safón, 2019), anecdotal evidence suggests that reputation is hardly measurable in the lower end of the ranking lists, leading to a proportionally greater influence of research output, a factor, that institutions can control. (Pandiella-Dominique et al., 2018).

While research output is increasing, the setback in the rankings of V4 universities represents a concern in the face of growing competition for higher ranking positions. There is a research gap in understanding the evolution and sensitivity of the V4 universities’ rankings to changes in their research output. This motivated our objective to analyse the relationship between the research output of V4 universities and their global rankings. We aimed to understand, how the ranks of the V4 region’s universities have evolved over recent years, what is the effect of new entries on rankings, and what are the relationships between changes in the institutions’ research performance indicators (RPIs) and their next year’s THE ranking positions. Furthermore, we explored how sensitive next year’s rankings are to various research performance patterns of V4 universities, assuming that other performance indicators remain stable over 1–2 years. Finally, we developed a method to allow customised sensitivity analysis of next year’s rank from institutions’ planned research output, given its current rank and research output.

Methods

We combine three separate datasets in our analysis covering four calendar years from 2019 until 2022: we calculate the exact ranks of V4 universities using published performance indicators by THE, we count new entrants and dropouts on the THE list and collect RPIs of universities from the V4 region. The evolution of rankings and RPIs over time are then visualized, which shows the relationship between rankings and RPIs, quantify the effect of new entries on ranking positions, evaluate the sensitivity of THE rankings to changes in RPIs using a regression model, and propose a rank prediction method.

Data sources

We collected published university ranks and THE performance indicator scores for exact rank calculations from the website of Times Higher Education World University Rankings (www.timeshighereducation.com) (THE, 2022). RPIs were collected from the SciVal database (www.scival.com). Both SciVal and THE process data from the Scopus database, so the RPIs used in this research reflected the part of the research output of universities, which impacts their THE rankings. Our analysis did not include regional or other specialised rankings. 2023 THE ranking data is the one published in 2022.

Exact THE ranks

From rank 201,401, 601 and 1201, THE list presents rankings in widening ranges of 50, 100, 200 and 300 ranks, respectively. Therefore, for all V4 universities on the list, we calculated exact ranks for five years from 2019 until 2023 using the THE method. We validated the accuracy of our method on exact ranks published for the top 200 universities (Pearson r > 0.999, p < 0.001). The THE methodology incorporates five key categories of indicators, including teaching, research, citations, international outlook, and industry income, with weights of 30%, 30%, 30%, 7.5%, and 2.5%, respectively (THE, 2022). The performance indicator scores were only used for the calculation of exact ranks but were not included among the RPIs used in the sensitivity analysis. Throughout this paper, smaller rank numbers will be referred to as better or higher ranks, and greater rank numbers as worse or lower ranks on the THE World University Ranking list. A decrease in rank numbers will denote improvement in an institution’s ranking position. We note that some universities may share the same exact rank, which is compensated at the end of each THE rank range. For example, in 2022 in the 601–800 range six ranks were shared by two-two universities. Ranks 795–800 were not assigned in this range and rank 794 was followed by rank 801. Within the 200 rank positions in this range, 194 ranks were assigned to 200 institutions. Between 2019 and 2022, the correlation between the unique rank positions and assigned THE ranks was near-perfect (r > 0.999) in the 400+ range, so we ignored the effect of shared ranks in our calculations.

Rank shift due to new entrants and dropouts in ranking

From the THE homepage (THE, 2023), we compared the lists of ranked institutions, recording the number of new institutions added to the ranking list (new entrants) and those removed from the list (dropouts) in each year from 2019 to 2022. Institutional name changes were matched and merged institutions were also identified. We collected data on new entrants and dropouts for each 100-rank interval on the list (e.g., new entrants and dropouts between ranks 1–100, 101–200, etc.). From the properties of ordered sets it follows, that new entrants and dropouts in higher positions change the rankings of institutions in lower positions on the list. For example, the position of an institution ranked 600 is changed upwards or downwards respectively by new entrants or dropouts between ranks 1–600, while an institution ranked 800 is also affected by new entrants and dropouts between ranks 601–800. We denoted the net change from new entrants and dropouts as “rank shift.” As new institutions typically enter THE rankings anywhere between ranks 200 and 1800, rank shifts at specific rank positions can be modelled via a continuous function (see below). Therefore, we fit a polynomial curve on the rank shifts recorded in 100-rank intervals in each year (see below).

Research performance indicators

To follow the THE ranking methodology, we collected five-year running average values of RPIs of each institution from SciVal (2023). The preceding five years’ average was calculated for year t (denoted with lower index _5t). The change of RPIs in year t was entered in our analyses in the form of growth ratios. Growth ratios for year t were calculated by dividing the preceding 5-year’s average RPI with the preceding 5 year’s average for year t − 1. Growth rates referred to the difference between the preceding 5-years average RPIs of two years divided by the preceding 5-year’s average of the RPI in the base year. (A growth ratio of 1.1 corresponds to a growth rate of 10%.) We denoted as year-on-year growth rate if two years values were compared, not 5-year averages.

Contributing to 30% of THE ranking (THE, 2022), the citation score is based on the Field-Weighted Citation Impact (FWCI). FWCI is calculated from citations received by an institution’s publications relative to the average number of citations expected from similar publications from the same year, having the same type and discipline in the Scopus database. FWCI reflects the research influence of an institution relative to the average of its scientific field (Scopus, 2022). We denoted the 5-year average FWCI in year t as FWCI_5t. Also, as a proxy to research volume and reputation, we included the 5-year average of the total volume of scientific output (V_T5t) including all publications indexed in Scopus with the institution’s affiliation. We also collected the 5-year average of the number of Scimago Q1 publications from the institution (V_Q5t) (SJR, 2023). To reflect the proportion of top-tier journals in research output, we computed the proportion of Q1 publications within the total volume of publications, as shown in Eq. (1).

$${P}_{Q5t}=\frac{{V}_{Q5t}}{{V}_{T5t}}$$

(1)

where ${P}_{Q5t}$ denotes the proportion of Q1 publications, ${V}_{Q5t}$ the number of Q1 publications over the last 5 years in Scimago database and ${V}_{T5t}$ the total publication volume over the last five years.

Descriptive statistics and data visualisation

We used heatmaps to provide a comprehensive assessment of the ranking performance of universities for the entire V4 region. One heatmap was designed to illustrate the ranking status of each university from 2019 until 2023 (Fig. 1). The exact ranks of the institutions in each year and the number of Q1 papers in the first and last years of the 10-year observed period were provided for each institution. Non-ranked years were depicted with grey colours. The change in the volume of published Scimago Q1 articles between 2013 and 2022 was shown for each institution on another heatmap (Fig. 2). On both heatmaps, shades from orange to dark red depicted unfavourable trends (e.g., ranking status below 800 or declining publication volume), yellow depicted intermediate status at rank 800 or stagnation, and colours from yellowish green to dark green depicted favourable trends (e.g., ranking status beyond 800 or rapid growth).

Fig. 1

Ranking status of universities in the region

Fig. 2

Year-on-year growth rates in the number of Q1 publications by institution

We also tabulated the number of ranked institutions, the minimum, median and maximum ranks and 5-year running averages of the key RPIs by country and by year (Appendix 1, Table 4).

Furthermore, we plotted the growth ratios of the 5-year running averages of RPIs by the actual rank of each institution. The trends are shown by fitting fractional polynomial curves over the scatterplots (Fig. 3).

Fig. 3

Growth ratios of RPIs by rank

We assumed that the rank of an institution is impacted by both its scientific output and the dynamic changes to the list caused by new entrants and dropouts. Our rank prediction model is shown in the Eq. (2).

$${R}_{Pt}={R}_{Et}+{R}_{Gt},$$

(2)

where R_Pt denotes the predicted rank in year t, and R_Et denotes the expected rank in year t. Expected rank is the rank that the university would achieve if the list would only be changed by the new entrants and dropouts compared to the previous ranking cycle, and the ordering of already ranked institutions on the list would be preserved. Rank gain in year t, denoted by R_Gt is the difference between the predicted and the expected rank.

Predicted ranks were derived in a two-step process. First, R_Et was calculated from the institutions actual rank in previous year (R_A(t−1)) and the rank shift (R_St) due to new entries/dropouts (Eq. (3)).

$${R}_{{\text{E}}t}={R}_{{\text{A}}(t-1)}+{R}_{{\text{S}}(t)},$$

(3)

where ${R}_{{\text{E}}t}$ denotes the expected rank, ${R}_{{\text{A}}(t-1)}$ is the institution’s rank in previous year and ${R}_{{\text{S}}(t)}$ is the rank shift due to new entries and dropouts. ${R}_{{\text{E}}t}$ of an institution could only be calculated for those years, which were preceded by a ranked year.

We estimated R_St as a function or R_A(t−1), by fitting a 5th order polynomial curve on the rank shifts in each 100-rank band, as shown in Eq. (4). For the estimation of model parameters to predict R_Gtvalues (see below), R_St values were calculated separately for each year between 2020 and 2023, while for the prediction of future ranks, R_St+1 values were predicted from the average of rank shifts between years 2020 and 2023.

$${R}_{{\text{St}}}={\alpha }_{1}{R}_{{\text{A}}(t-1)}^{2}+{\alpha }_{2}{R}_{{\text{A}}(t-1)}^{3}+{\alpha }_{3}{R}_{{\text{A}}(t-1)}^{4}+{\alpha }_{4}{R}_{{\text{A}}(t-1)}^{5}+\varepsilon ,$$

(4)

where ${R}_{{\text{St}}}$ is the shift in rank due to new entrants or dropouts, R_A(t−1) denotes the actual rank of the institution in the previous year.

In the second step, we modelled rank gains (R_Gt) as a function of scientific output. Our assumptions for this step were the following: the absolute performance level of an institution in year t − 1 predicts its actual rank in year t − 1 and hence, its rank shift, and its expected rank in year t (Eq. (3)). The ranks of two adjacent institutions will be shifted by nearly the same amount, and if the performance of all ranked institutions increases at the same rate, their rank order remains preserved. Therefore, rank gains (R_Gt) (i.e., deviations of the actual ranks versus the expected rank order in year t) depend on the growth ratios of an institution’s performance compared to the growth ratio of others. Institutions with performance growth ratios greater compared to their neighbours’ on the list improve their position (i.e., their R_Gt value becomes negative). R_Gt values were computed as the difference between R_At and R_Et of each institution in each year t that was preceded by a ranked year (Eq. (5)).

$${R}_{{\text{G}}t}={R}_{{\text{A}}t}-{R}_{{\text{E}}t},$$

(5)

where R_Gt represents the difference between the actual (R_At) and expected rank (R_Et).

We modelled R_Gt in year t using the growth ratio of RPIs (i.e., the ratio of 5-year averages of RPIs in year t and preceding year t − 1). The vector of predictors is denoted by x_t,

$${x}^{\prime}=\left[\frac{{FWCI}_{5t}}{{FWCI}_{5\left(t-1\right)}},\frac{{V}_{T5t}}{{V}_{T5\left(t-1\right)}},\frac{{P}_{Q5t}}{{P}_{Q5\left(t-1\right)}}\right],$$

(6)

where x_it^g = x_it/x_i(t − 1) represents the growth ratio of a single RPI denoted with i in year t (Eq. (6)). (We note that this specification provided a better fit than expressing percentage growth in logarithmic form).

We assumed that three types of interactions could modify the effect of predictors. First, the growth ratio of other RPIs (e.g., the growth ratio of publication volume may impact ranking positions differently at high or low growth ratios of publication quality). We denoted the vector of all pairwise interactions between the predictors as x^g_t. A single term is written as x^g_it⋅x^g_jt (i ≠ j). The absolute levels of the indicators may also modify the effect of predictors (e.g., the growth ratio of FWCI_5t may have a different effect at high or low overall publication volume). We denote the vector of interaction terms with the absolute levels of predictors in year t as x^a_t. As a single term in x^a_t can be written as x^g_it⋅x_jt (i and j are allowed to be the same indicator). Finally, to capture the changing importance of scientific performance at different rank levels (e.g., the contribution of the reputation survey is increasing at better rank positions), we included a vector of interactions of all previous terms with R_A(t−1), denoted as x^r_t. The elements of x^r_t include x^g_it⋅R_A(t−1), x^g_it·x^g_jt⋅R_A(t−1) and x^g_it·x_jt·R_A(t−1) Finally, to reflect the serial correlation of R_Gt (e.g., strategic initiatives with lasting effect on an institution’s performance), we added R_Gt with 1-year lag R_G(t−1). Altogether, our hypothetical model to predict R_Gt can be written as

$${R}_{Gt}=\alpha +{\beta }_{1}{R}_{G(t-1)}+{x}_{t}{\prime}{\beta }_{2}+{x}_{t}^{g{\prime}}{\beta }_{3}+{x}_{t}^{a{\prime}}{\beta }_{4}+{x}_{t}^{r{\prime}}{\beta }_{5}+\varepsilon ,$$

(7)

where α is the intercept and β denotes coefficients for the predictors.

At first, given the non-normal distribution of many predictors, we observed the Spearman rank correlation matrix for all predictors, to analyse the bivariate associations between the raw predictors and the constructed interaction terms (Bishara & Hittner, 2015). Then the estimation of model coefficients was performed via OLS regression with errors clustered on the level of institutions. We developed six regression models and present our findings based on the model that was found to be optimal (M₁) through the estimation and selection process detailed in Appendix 3. Predicted ranks were calculated by adding previous year’s rank (R_A(t−1)), R_St as calculated using the estimated coefficients of Eq. (4) and R_Gt, computed using the estimated coefficients of selected variables from Eq. (7).

Exploring change patterns of RPIs via strategic scenarios

To understand if some change patterns of RPIs are associated with better ranking positions, we predicted ranks in various strategic scenarios detailed in Table 2.

Table 2 Strategic scenarios tested in the analysis

In each scenario, we made predictions from baseline rank positions of 800,1000 and 1200. We computed the mean values of the RPIs for each baseline rank, and applied growth rates of 5%, 10% and 15% for the 5-year averages of RPIs.

Results

Descriptive results

The number of ranked institutions in the V4 region increased from 35 in 2019 to 68 in 2023. While Czech Republic had the most of ranked universities in 2019 (N = 14), by 2023, Poland took over the lead. In 2023, Poland, Czech Republic, Hungary and Slovakia had 32, 18, 11 and 7 ranked universities, respectively (Appendix 1, Table 4). Most institutions from the region were ranked below 800. This proportion increased from 81.1% in 2019 to 93.7% by 2023, showing an overall declining trend. The median rank decreased from 1054 in 2020 to 1372 in 2023 in the four countries (Appendix 1, Table 4). While the total publication volume of V4 institutions stagnated between 2019 and 2022, both the research influence and proportion of Q1 publications improved over the same period. Median FWCI_5t increased from 0.94 in 2020 to 0.98 in 2023. The median proportion of Q1 publications increased from 0.34 in 2020 to 0.44 in 2023. A detailed summary table of ranks and RPIs by country are summarized in Appendix 1—Table 4.

The heatmap in Fig. 1 shows rank changes over time for each institution. The colour shift towards dark red tones from 2019 to 2023 illustrates that most universities lost ranks, but a few demonstrated improvements. Semmelweis University from Hungary emerged as the most prominent performer, displaying a consistent upward trajectory across all assessed periods. Additionally, Jagiellonian University from Poland exhibited a favourable trend, albeit with a notable setback in 2023 compared to 2022.

Figure 2 provides insight into the year-on-year growth rates of published Q1 articles from 2013 to 2022. The predominant green colours of the heatmap indicated a gradual yet steady growth in the number of published Q1 articles in most V4 universities. The pandemic years of 2020 and 2021 were particularly remarkable, when the number of published Q1 articles grew rapidly in nearly all V4 universities compared to previous years. Country-specific heatmaps are presented in Figures. 7–10 in Appendix 2.

The growth ratios of the 5-year averages of RPIs by rank are shown in Figure 3. For all indicators, we observed the highest growth ratios at the lowest ranks, suggesting that the competitive dynamics of universities change with their ranks and universities with lowest baseline performance can achieve highest growth ratios, probably due to their lower baseline performance. The association between the growth ratios of FWCI_5t and P_Q5t and the institutional rank had reverse J-shape, with stagnation between rank positions of 1000 and 500, and a modest increase in growth ratios of the institutions at the highest ranks. Growth ratios of total publication volume (V_T5t) varied greatly at lower ranks, while the total publication output beyond rank 1000 tended to show growth ratio between 1.0 and 1.1 (i.g., single-digit growth rate).

Correlations between variables

The correlation matrix of the variables is shown in Table 5 (in Appendix 3) and supports the assumptions of our rank prediction strategy. R_Et is nearly perfectly correlated with R_A(t−1) (r = 0.97), suggesting that the actual rank of an institution in year t − 1 predicts nearly perfectly its expected rank in year t, shifted by new entries and dropouts in the list. However, R_Gt shows no significant correlation with R_Et (r = − 0.04), but it exhibits weak to moderate correlations with the RPIs in the predictor vectors. This suggests that changes of an institution’s scientific performance (partly) predict its position relative to its expected rank. However, the serial correlation between R_G(t−1) and R_Gt was weak (r = 0.19), suggesting that the previous year’s rank gain will have only minimal influence on how the institution will perform in its new position.

Predicted rank shifts due to new entries and dropouts

Between 2020 and 2023, after deducing dropouts, there were no new entries at positions above rank 200, while the number of new entries per year below rank 200 increased linearly by 1.2 per 100 range. For example, on average, there were 1, 8.2 and 14.5 new entrants between ranks 201–300, 801–900 and 1201–1300, respectively. The mean rank shift due to new entrants and dropouts per year in the same ranges was 1.3, 38 and 80.5, respectively. The parameters of the rank shift (R_St) calculator at specific rank positions are shown in Eq. (8).

$${R}_{St}=5.016\times {10}^{-5}{R}_{A\left(t-1\right)}^{2}-4.155\times {10}^{-8}{R}_{A\left(t-1\right)}^{3}+5.835\times {10}^{-11}{R}_{A\left(t-1\right)}^{4}-2.102\times{10}^{-14}{R}_{A\left(t-1\right)}^{5}+\varepsilon ,$$

(8)

where R_St the shift in rank due to new entries/dropouts, and R_A(t−1) denotes the actual rank of the institution in the previous year.

The fit of the 5th-order polynomial curve was nearly perfect (R² > 0.999). The prediction error term for R_St was several magnitudes smaller than that of rank gain (R_Gt), therefore we ignored it in the final rank predictions. Rank shift (R_St) at specific ranks as a continuous function of R_A(t−1) is depicted in Fig. 4.

Fig. 4

Rank shift due to new entries and dropouts

Predicted rank gains

There were 200 observations with R_Gt values available after at least one ranked year, out of which 191 observations were included in the estimation sample. Intermediate results of the data preparation and variable selection processes are provided in the Appendix 4. Altogether, the 31 predictor variables in the full model explained 32.8% of the variance of R_Gt, while the three predictors selected for the final model explained 22.1%. The growth ratios of all three key RPIs were included in some form of interaction with other predictors. Negative coefficients indicated a contribution to better-than-expected ranks. In line with the THE methodology, the growth of FWCI_5t (FWCI_5t^g) had the greatest influence on R_Gt, which was enhanced by higher FWCI_5t values. The growth ratio of the total publication volume (V_T5t^g) also improved R_Gt, increased by higher FWCI_5t growth ratios, and by worse rank positions. (i.e., with the same growth ratio in publication volume, more institutions can be overtaken at lower ranks. However, institutions at lower positions also lose more ranks due to new entrants in ranking.) Unexpectedly, the positive coefficient of P_Q5t^g suggested that high growth ratios in the proportion of Q1 publications were associated with poorer rank positions. This effect was enhanced by high FWCI_5t values, and lower ranks. (Table 3). However, in the sensitivity analysis with alternative models the role of P_Q5t^g was rather inconsistent (Appendix 4, Table 6).

Table 3 Parameters of the selected prediction model of rank gains (R_Gt)

Predicting next-year-ranks

The final model to predict next year’s ranks (R_A(t+1)) is shown in Eq. (9). Using the parameters of Eq. (8) for predicting R_St+1 and the coefficients of Table 3 estimated for in-sample predictions of R_Gt+1, R_P(t+1) (i.e., predictions using the available historical ranks and RPI data for the estimation of model parameters) predicted actual ranks with R² value of 0.9997 and root mean squared error (RMSE) of 54.6, mean absolute error (MAE) of 42.5, and normally distributed error terms. The RMSE value of 54.6 suggests that 68% of in-sample predictions were within the actual rank ± 55 range. Within sample predictions of next year’s expected ranks (R_E(t+1)) assuming unchanged research output, and predicted ranks (R_P(t+1)) considering both rank shifts and changes in research output were plotted against current ranks in Fig. 5.

Fig. 5

In-sample predictions of next year’s expected ranks and actual ranks

$${R}_{{\text{A}}t+1}={{R}_{{\text{P}}t+1}+\varepsilon =R}_{{\text{A}}t}+{R}_{{\text{S}}t+1}+{R}_{{\text{G}}t+1}+\varepsilon .$$

(9)

To aid the interpretation of our results, we calculate the next-year predicted rank of an institution currently ranked 1000 with no change in its scientific performance (i.e., FWCI_5t^g = P_Q5t^g = V_T5t^g = 1). The mean FWCI_5t at rank 1000 is 1.15. The predicted rank gain (R_G(t+1)) using the coefficients of Table 3 is 9.4 (− 165.3 × 1.15 + 0.06 × 1.15 × 1000–0.14 × 1000 + 270.5). Using Eq. (8), the predicted rank shift (R_S(t+1)) is 46. Following Eq. (9), the predicted rank is 1055.4 (1000 + 46 + 9.4). Altogether, without change in its performance, the institution is expected to lose 55–56 rank positions by next year. To allow customised sensitivity analysis, we included a calculator as a supplementary file.

Exploring strategic scenarios

Following the strategic scenarios, in Table 2, we show rank predictions from various baseline rank positions and growth rates. (Fig. 6.) To offset the rank shifts, an institution at rank 1200 must increase its overall 5-year average scientific output by approximately 15%, while at rank 1000, 10% and at rank 800, 6% growth may be enough to preserve positions. While the growth of FWCI_5t has the most influence on an institution’s ranking, those universities were most successful, who increased both their FWCI_5t and total publication volume. Although FWCI_5t and P_Q5t were strongly correlated (r = 0.62) in our sample, the increasing proportion of Q1 publications did not contribute to better rankings. We note, that while high growth rates of P_Q5t suggest a rapid evolution of publications in top-tier journals, the observed growth rates of FWCI_5t and V_T5t remained below the threshold required to offset the rank shifts due to new entries, which explains the overall declining trend of university rank positions in the region.

Fig. 6

Exploring the regression model across various rank positions and growth rates

Discussion

In this paper we described that the THE World University Ranking positions of most universities in the V4 region have been decreasing over the past years, even though the citation impact and proportion of Q1 publications have been increasing over the same time period. We explored the sensitivity of next year’s ranks of V4 universities to changes in their research performance, if other performance indicators remain stable over 1–2 years. Simultaneous growth in the overall volume of publications and FWCI seems to be the scenario with greatest positive impact on ranks, while, contrary to the expectations, the growth in the proportion of Q1 publications did not have positive impact on rankings. Outside the control of ranked institutions, the large number of new entrants in the THE ranking list shifts the ranks of V4 universities downwards, and the growth in the research output of most universities cannot offset the negative rank shifts. To allow customised sensitivity analysis of next year’s rank from institutions’ planned research output given their current rank and research output, we developed a model with in-sample prediction performance characterised by R² of 0.9997 and RMSE of 54.6.

With the increasing use of international rankings for higher education decision making, more and more studies are examining the reliability and impact of rankings, the weaknesses and strengths of different methodologies, and their potential biases (Kováts, 2015). In addition to these studies, an increasing number of papers have been written on the trends and predictability of the determinants of rankings, such as FWCI (Wagner et al., 2018). Furthermore, models for predicting ranking position represent an active field of research. These approaches are typically based on the performance indicators of existing ranking systems, with a focus on research indicators. In our research we used THE’s publicly available league tables, scores and weights of the main performance indicators, and publicly available data on the RPIs from SciVal database of Czech, Hungarian, Slovak, and Polish universities. To the best of our knowledge, our approach is unique as we have considered the dynamic nature of rankings. We assessed the impact of scientific performance and new entries and dropouts on the THE ranking positions of universities from these four countries between 2019 and 2023. Another unique feature of our study is that while previous studies have attempted to predict the ranking of universities at the top of the rankings for example a group developed a predictive model for universities in the top 200 in THE world list (De Luna Pamanes et al., 2020) our research focused on universities of the V4 region where universities are typically ranked in the lower range of THE rankings with positions below rank 800. These universities also face similar geopolitical challenges that affect their competitiveness in the rankings, such as language barriers (van Raan et al., 2011), historic policy factors (Antonowicz et al., 2017).

The data suggest that, in terms of overall representation within THE ranking system, Poland has emerged as the clear leader in the region followed by Czech Republic. Hungary and Slovakia lagged regarding the number of ranked universities. On the other hand, it was observed that the number of universities ranked beyond 800 is approximately the same in three countries: Poland, the Czech Republic, and Hungary. In contrast, Slovakia has no universities ranked beyond 800. While being ranked or the number of ranked universities can be considered a significant indicator of the quality of higher education in a country, the positions at which the ranked universities stand are also of great importance (Banász et al., 2023). Poland has the most ranked institutions out of the four countries, although the number of universities ranked in the top 800 in Poland, Czech Republic and Hungary is almost the same. This suggests that a higher quantity of ranked universities may not mean that a country has a higher quality of higher education across all its universities. It is important for policymakers, decision makers and university executives to focus on enhancing the quality of education and research in all institutions, rather than just select a few, to ensure the long-term sustainability and competitiveness of their higher education systems (Dachyar & Dewi, 2015; OECD, 2007; Simina, 2022).

We chose heatmaps to visualise the performance of these universities in the THE world ranking. This showed that with few exceptions, the ranking of universities of the region showed a declining trend, despite the increasing research prestige and influence. The growth rates of the two most impactful indicators, FWCI and overall publication volume were insufficient to offset the loss in ranks due to new entries in ranking. We found that among the examined universities the growth rate of proportion of Q1 publications and FWCI was higher for universities beyond 1000. It is important to note that for universities with lower publication output a small change in the number of already small number of Q1 publications leads to a larger relative growth compared to the previous year. Our results suggest that this effect may weaken at the 500–1000 ranks, and then the growth is more modest for the top universities in the list. Furthermore, our results highlight that the impact of new entrants in ranking is stronger under the range 1000–1200, new entries push incumbent institutions downwards up to 50–70 positions in each year, that is a major threat on their ranking position, largely outside their control. Unless achieving a constant double-digit growth in FWCI and overall publication volume, progress in ranking for universities of the region seems elusive. Accordingly, it is imperative to take into account two pivotal factors. Firstly, the addition of new universities to the ranking each year can lead to a decline in the ranking of other universities. Secondly, universities must strive to make significant enhancements in their RPIs. Hence, universities need to consider the combination of “new entries” and “significant improvement in RPIs" as the two decisive factors.

In our sample, FWCI was strongly correlated with rank, in line with the THE methodology (FWCI weight of 30%). While universities with higher citations are more likely to have higher publication output, a higher volume of publications does not necessarily result in more citations (De Luna Pamanes et al., 2020). These findings are further nuanced by the results of our strategic scenario Q and V_T5t, which suggests that the proportion of Q1 publications did not have a positive effect on ranks. This observation can be partly explained by the Matthew Effect (Merton, 1968) which suggests that entities with more existing recognition often from developed countries or prestigious universities tend to receive more attention and citations for their work (Gomez et al., 2022). Consequently, publishing in high-tiered journals may result in fewer citations for less well-known universities or countries due to existing biases in the academic community. On the other hand, increasing the volume of publications, which includes contributions to both high and low-tiered journals appear to improve rank and makes it more likely to receive citations. Given that citations carry significant weight in academic rankings, an increase in the overall volume of publications can lead to rank improvement. THE's new methodology aims to moderate the impact of FWCI to ranks to better measure research impact and filter out FWCI potential bias (Davis, 2022). Despite the intention, it is questionable how the new methodology will change the dynamics of Q1 publications and the new citation indicators.

Despite the strong explanatory power of our overall model (R² = 0.9997), the moderate explanatory power of our rank gain model (R² = 0.221) suggests that there are elements that we have not taken into account, and the mean absolute error of 42.5 ranks could be further improved. Banász et al. argue that universities should consider an appropriate composite of indicators in order to move up in the rankings, while highlighting the salience of reputation emphasising its self-reinforcing effect (Banász et al., 2023). The statistical analysis presented by Benito et al. drew attention to the influential power of income, their work was based on an analysis of the top 300 universities in the QS and their results showed that funding explained up to 51% of the variability of the positions attained by the universities in the rankings (Benito et al., 2019; OECD, 2007; Marope et al., 2013). While income can be influenced by a targeted strategy, reputation is a challenging factor to control To address this challenge and get recognition, quality of the teaching and research has to improve, universities could prioritize investments in research infrastructure and faculty development, attract and retain talented researchers and educators, or collaborate with international institutions to promote cross-border knowledge exchange and innovation (Bilous, 2015; Dachyar & Dewi, 2015).

Although some of the studies seek to provide alternative solutions to the rankings, such as Banász's classification solution (Banász et al., 2023), which aims to compare similar universities with similar universities, or such as different national rankings adjusted for local specificities, that are not in line with the results of global rankings (Çakır et al., 2015). Globalisation and the increasing competition between universities, as well as the popularity of international generic rankings as opposed to more narrowly defined rankings (regional or subject), justify the focus of our research on the methodology of a widely recognised generic ranking, THE world university ranking. Understanding the dynamics and limitations of these rankings is of key importance when planning programs based on them, interpreting published league tables.

Strengths and weaknesses

One potential limitation of our study is that we have focused on a relatively small number of universities, and they exhibit homogeneity in their ranks over a five-year period. Including more universities from other regions and other rank ranges could improve the generalisability of our findings beyond the V4 region and ranks above 800. The explanatory power of our rank gain model could be improved by including additional factors beyond RPIs. However, to aid the predictions of next year’s ranks during strategic planning in the universities, it is important that non-standardised values of institution-level indicator data are uniformly available from data sources external to the THE rankings, similarly to the RPIs presented in the SciVal database. Indicators in this form should also provide meaningful input to decision-makers, which can be readily translated to strategic targets. Furthermore, the rank shift model assumed that future rank shifts follow the average of those in the previous years. However, with the expansion of the THE list, rank shifts may change over time in various rank ranges of the list. Modelling the dynamic changes of rank shifts could further improve the predictive power of our model and decrease mean absolute error. Finally, our model considered only 4 consecutive years. Therefore, we consider that the model results remain valid for the 1–3 years, when the stability of model parameters over time have to be re-examined. Finally, the model performance has been reported for in-sample predictions, while its out-of sample performance will have to be tested when 2024 ranks will be available for the included institutions.

The main strength of our approach is that we have identified an important factor that has been so far ignored in the literature, the impact of new entrants on the position of already ranked universities. This factor becomes more important as the popularity of rankings are increasing. Our research also shows that the effect of rank shifts is particularly strong for universities at the lower range of the ranking list. We have also developed an applicable, interactive tool that can predict with high confidence the next year's rank of a given university along the planned RPIs.

Conclusion

This work first analysed the sensitivity of the next year’s Times Higher Education World University Rankings positions of universities from Czech Republic, Hungary, Poland and Slovakia to the changes in their research output. A regression model was developed to quantify the effect of new entrants and dropouts on the ranking list, as well as the sensitivity of ranks to various research performance patterns. A method was developed that allowed customizing the sensitivity analysis of next year’s rank from an institution’s planned research output, given its current rank and research output, assuming that other performance indicators remain unchanged over 1–2 years.

A descending trend was observed in the ranks for most universities from these four countries. We observed that the Field-Weighted Citation Impact and the total volume of publications had the highest effect on ranks, while the proportion of Scimago Q1 publications did not have a consistent effect on projected next year's ranks. In most universities, the growth in RPIs was below the level that would be required to offset the rank changes due to the inclusion of new entrants in ranking. The findings of this research suggest that universities have to consider complex strategies to improve their ranking focusing on other activities than research such as reputation, internationalization, or industry income.

Given the results and findings of this work, we can conclude that establishing and maintaining complex ranking strategies requires awareness that experiences and knowledge about ranking might be transferred between different jurisdictions, such as regions, countries and universities, but not necessarily in a direct way and not without difficulties and limitations. Some elements of this complex system are likely to be easier to adopt while other ones might be more difficult to transfer. Complementary local data collection and local research are needed to adapt and to transfer ranking experiences to the local context. Each region and country must also develop its own strategies to have better formulated and evaluated initiatives and strategies to improve ranking positions. It is also proposed that all universities should have trained ranking specialists and that ranking-related training should be available at national level. This study utilizes existing knowledge from other regions and applies relevant findings to the V4 regions. To conduct a comprehensive analysis, it was necessary to collect and analyse local data from universities in the lower third of the ranking. With due caution, our results may be generalizable to most of the universities below 800 in the THE.

We recommend universities conducting similar research to incorporate the findings of this study, ensuring the proper utilization of local data. To do so, we provide our rank prediction tool presented via this link: https://hecon.uni-obuda.hu/en/rank-calculator/.

Appendices

Appendix 1: Country performance in terms of rank and scientific performance

See Table 4.

Table 4 Ranks and RPIs by country

Appendix 2: Heatmaps of universities of the four examined countries

See Figs. 7, 8, 9, 10.

Fig. 7

Heatmap on the performance of Czech universities—a ranking status and b growth rates in the number of Q1 publications by institution

Fig. 8

Heatmap on the performance of Hungarian universities—a ranking status and b growth rates in the number of Q1 publications by institution

Fig. 9

Heatmap on the performance of Polish universities—a ranking status and b growth rates in the number of Q1 publications by institution

Fig. 10

Heatmap on the performance of Slovak universities—a ranking status and b growth rates in the number of Q1 publications by institution

Appendix 3: Data management and model selection

For institutions with only one ranked year, R_G(t−1) values were imputed using a panel regression of R_G(t−1) values on R_At, and calendar year (model not shown). As a sensitivity analysis, we also explored results on the dataset without imputation. To detect the effect of outliers, we fitted robust MM regression and ordinary least squares (OLS) regression on R_G and the main predictors (Fig. 11). We observed unacceptably high leverage values (> 4/N) for some data points. To mitigate the effect of outliers, we deleted 5% of the cases with greatest Mahalanobis distances from the sample mean for single predictors and the multivariate sample, resulting in similar OLS and robust regression parameters. Scatterplots of R_Gt and key predictors before and after removing outliers are shown in Figs. 12 and 13.

As a next step, we selected relevant predictors from the hypothetical model. First, the bivariate relations between the dependent and predictor variables were explored via correlation analysis (Table 5). Also, we calculated the Shapley values to explore the contribution of the main predictor groups to explaining the variance of R_G. Next, we used backward, forward, and stepwise selection and the Furniwal and Wilson leaps and bounds algorithm (Furnival & Wilson, 1974) to identify candidate models. The final model was selected if the following criteria were met: all coefficients were significant (p ≤ 0.05), the Ramsay RESET test suggested no functional misspecification error, and multicollinearity was absent with VIF values < 10 for all variables. When meeting these criteria, preferably all three scientific output indicators (FWCI, V_T and P_Q) had to be included in the model in at least one interaction term, and the model should have higher R² and/or lower Akaike’s information criterion (AIC) values than its alternatives.

Fig. 11

Scatterplot of rank gains and main RPIs before removing outliers (N = 200)

Fig. 12

Scatterplot of rank gains and main RPIs after removing outliers (N = 191)

Table 5 Correlation matrix of variables

Appendix 4: Results of the data preparation and variable selection

Rank positions of two preceding years were not available for 64 observations, therefore R_G(t−1) had to be imputed. Nine observations were removed because of outliers. Altogether, 191 observations were included in the full estimation sample, and the dataset was complete for 130 observations without the imputated values of R_G(t−1). The scatterplots of R_G and the main predictors before and after removing outliers are shown in Fig. 12, respectively. RPIs explained altogether approximately one third of the variance of R_Gt. The R² of the full model with all predictors was 0.328. The Shapley values indicated the predictor groups R_G(t−1), x′, ${x}^{{{\text{g}}}{\prime}}$, ${x}^{{{\text{a}}}{\prime}}$, and x^r contributed by explaining 1.5%, 14.8%, 14.8%, 28.4% and 40.5% of the variance.

The variable selection methods provided a variety of proposed models on the imputed and non-imputed sample. The models are shown in Table 6. Model performance was evaluated on the full sample without outliers in both cases. Based on the selection criteria, M₁ was selected as the optimal rank predictor. The explanatory power of M₂ was slightly lower. M₃ had the best explanatory power from models selected from the imputed sample, so we included for comparison, although it did not meet the selection criteria because of multicollinearity. Although similar in performance, different predictors were selected when the sample was used without imputation. In this sample, M₄ showed best performance, followed by M₅. While M₆ had better explanatory power than the other models, because of multicollinearity it did not meet the selection criteria.

Table 6 Results of six regression models tested in the sensitivity analysis

Fig. 13

Comparison of six regression models across the strategic scenarios

We compared the six models for a hypothetical institution ranked 1000 with 10% growth rate in the strategic scenarios. Combined growth of publication volume and FWCI would provide the greatest rank gains in all models. The results suggest that approximately 10% growth is required across all indicators to preserve the institution’s previous year’s rank. However, the models were inconclusive about the contribution of growing publication quality to better ranking positions. While keeping all metrics unchanged, efforts to increase the proportion of Q1 publications seemed to be counterproductive in models M₁, M₃ and M₄, while in M₂ and M₅ publication quality was not a significant predictor. In M₆ with the highest explanatory power, increasing publication quality seemed to contribute positively to better ranking positions.