Introduction

The Bologna Declaration of 1999, which ushered in the European Higher Education Area (EHEA), proposed, among other questions, a change of the teaching–learning model and, to this end, introduced Continuous Assessment (hereinafter, CA) in European universities (Sanz-Pérez, 2019).

The academic literature has highlighted the advantages of CA from the perspective not only of students, since it enhances their motivation for learning through feedback on their knowledge (Day et al., 2018), but also of instructors, as it encourages them to obtain feedback from students on their learning (Myllymäki, 2013). A number of publications have also focused on the impact of continuous assessment on student engagement (Cole & Spence, 2012), perception-satisfaction (Carrillo-Peña & Pérez, 2012) and dropping out of the subject (García et al., 2014), with its impact on examination marks being one of the most controversial effects (Carrillo-Peña & Pérez, 2012; Cole & Spence, 2012; Gallardo & Montolio, 2011; García et al., 2014; González et al., 2015). However, one of the main barriers to the application of effective CA has always been class size (Broadbent et al., 2018) since, when there is a high number of students in the class, instructor feedback to students can become a highly repetitive and time-consuming process, making delivery of ‘timely’ feedback very difficult (McCarthy, 2017). Furthermore, CA seems to work better in small classes since in large groups teachers and instructors may be unable to give the individualised attention that the system requires for every student to develop the prescribed skills (González-Campos et al., 2018).

Despite its recent introduction in Europe at the end of the twentieth century, the use of CA in higher education is not a new idea. End-of-semester exams have been supplemented or replaced by several CA systems in Australia, the UK, and New Zealand over the last 40 years (Richardson, 2015). Universities in the USA have also used CA for decades. In fact, at Harvard University, since 2010, a final exam has only been held by special permission as a supplementary tool to CA (Harvard Magazine, 2010).

In the case of Europe, although the Declaration of Bologna established the year 2010 as the common horizon for the implementation of the EHEA, the implementation of CA as an element of the new teaching and learning paradigm was not carried out at the same rate across the many different countries. For example, the use of CA at Danish universities was allowed in 2016 (Bjælde et al., 2017). In the case of Spain, where this paper is framed, CA has constituted one of the central elements upon which the methodology of the new university model has been built (Quiroga et al., 2014), although the process This adaptation process has occurred progressively since 2010, as each university has introduced it into their University student statutes, whilst always considering that the evaluation of academic performance should converge towards a system that contemplates CA (Ministerio de Educación, 2010).

All things considered, this paper analyses the transition towards a CA system in the subject Principles of Economics, which is taught in the first year of the degree in Business Administration and Management at the University of Seville (Spain) with a high number of students enrolled, and specifically evaluates the contribution made by certain digital tools to optimise the operation of the CA system.

In accordance with the Spanish University regulation that came into force in the 2010–11 academic year and that required the implementation of the EHEA, the faculty of this subject began the gradual process of transition from traditional evaluation, based exclusively on an exam at the end of the course, into a system in which the CA was the priority choice by the student, thus following the mandate of the regulations on evaluation of the University of Seville, which, like the rest of the universities in its environment, considers CA to be the priority evaluation system today.

In an initial phase of this transition process towards the implementation of CA, students were encouraged to choose to take the subject through CA, a conventional system, based on written activities carried out throughout the semester, under supervision and tutoring, involving advanced readings, assignments and communication exercises (as recommended by Bridges et al., 2022). The faculty found that this CA alternative was not very successful, and that over 95% of the students continued to take the final exam.

This result obliged the teaching staff of the subject to open a period of reflection in search of other CA options that would enjoy a greater degree of acceptance by the students, in which the debate between Clark and Kozma on educational technology was settled in the 1990s (Clark, 1994; Kozma, 1994). The advantages of using digital tools such as digital platforms (Tormos et al., 2014), mobile web-enabled technologies (Castillo-Manzano et al., 2017; Jahnke & Liebscher, 2020) and interactive response systems (Castillo-Manzano et al., 2016) were analysed to mitigate the problems of increasing class sizes (Girma & Darza, 2020) and the demands on instructor time and resources (Wilson et al., 2011). These digital tools became especially relevant during the global COVID-19 pandemic since many educational institutions were forced to switch to e-learning (Maatuk et al., 2021), thereby generating different expectations regarding the possibility of returning to traditional methods in the post-pandemic scenario (Moore & Piety, 2022). As a result of this reflection, during the 2016–2017 and 2017–2018 academic years, pilot experiences were carried out based on taking digital multiple-choice tests (e-tests) throughout the semester (Playfoot et al., 2022), along with a final exam. The results of these pilot tests, the results of which are included in this paper, have been decisive in designing a CA system that, complying with the normative evaluation guidelines of the University of Seville, would optimise both student learning and the work of the teaching staff, who teach in more than one class group per semester (each class group has an average of 70 students).

In this way, in the 2020–2021 academic year, the implementation of an e-Continuous Assessment (hereinafter, e-CA), was carried out, which is currently followed by 90% of the studentsFootnote 1 and in which class activities are included, carried out through the virtual platform, combined with the completion of multiple-choice tests in electronic format (e-tests). This approach follows the thesis of Kozma (1994) in that technology, as an attribute of learning media, directly influences such media. In addition to contributing to greater student participation, the implementation of this e-CA system has made it possible to improve academic performance rates.

Bearing in mind the theoretical framework provided by Fig. 1 below, this paper collects the experience accumulated during the implementation of the e-tests, to promote the transition and consolidate a system of e-CA in higher education to enhance student learning and improve academic outcomes. It should not be forgotten that several scholars, such as Holmes (2018), Lopez-Tocon (2021), and Zorio-Grima and Merello (2020), have stated the pertinence of contributing new evidence in this regard. A vast amount of evidence shows that new technologies and tools have opened up new possibilities for teaching and learning paradigms, in general, due to their potentially beneficial characteristics for educational change. However, a better understanding is needed of the role played by digital technologies in support of the evaluation processes introduced by the EHEA.

Fig. 1
figure 1

Source: prepared by the authors

Research conceptual and theoretical framework

Full size image

In short, the goal of this paper can be given in terms of the following research questions:

  • Do e-tests lead to similar results in terms of academic performance to those of the traditional final examination?

  • Are e-tests an appropriate option to assess both theoretical and practical knowledge of an Economics subject?

  • Can e-tests become an effective tool for the design of an e-CA system that meets the demands and expectations of students?

  • Are e-tests useful, even if the student does not pass them, by reducing the probability of the student dropping out of the subject?

  • Are there other socio-demographic or academic factors that can influence the student’s academic performance and that must be taken into account to isolate the effect of the e-tests carried out?

Research design

Participants

The target population of this study is a sample of first-year students studying for the Business Administration and Management Degree at the University of Seville, enrolled in the subject of Principles of Economics, in two successive academic years: 2016–2017 (106 students) and 2017–2018 (149 students). Students were allowed to change groups if they did not want to take part in this research and all the remaining students were asked for their consent to take part and to complete an initial questionnaire regarding their profile (see the Appendix).

The subject analysed, Principles of Economics, is a core class at the University of Seville and thus compulsory. Its content is mainly theoretical.

Two strategies were adopted to minimise the ‘Hawthorne effect’, which is the potential bias that can occur when the rise in the interest of students and instructors on the application of a new strategy of teaching innovation generates a boost to performance (Bartsch, 2013). Firstly, the objective of our research was not explained to the students in advance. Secondly, the research phases were distributed independently; instructors of the groups that carried out e-tests did not design the tests, develop the database, apply the econometric model, nor interpret the results.

e-test implementation

The students that formed part of this research took three e-tests corresponding to the three programme blocks (Fundamentals of Economics and Economic Thought, Macroeconomics, Microeconomics). They chose to take an e-test for each block due to the thematic unit of the subjects taught in each part and to the organisation of teaching in a subject of 6 ECTS (60 h of classes), where 8 topics are taught that include theoretical content and practical exercises. All three blocks were structured in the same way: 10 multiple choice questions with three possible answers each, only one of which was correct (Rodríguez 2005). e-tests were carried out in the first 15 min of class once instructors had previously taught the content of the programme blocks.

As mentioned in the introduction, during these two courses in which the capacity of the e-tests to evaluate the knowledge acquired by the students and become a true e-CA system was being tested, the students were obliged to sit a final written examination to pass the course, regardless of the results achieved in the tests. This exam consisted of three theory questions (theory paper) plus a practical question (practical paper) that required them to solve a mathematical problem.

The ethical appropriateness of our research design was endorsed externally by the study being financed through a competitive call made by the University of Seville with evaluation by anonymous reviewers (Project: 21105, Support for Teaching Coordination and Innovation Programme).

Methodology

Two microeconometric models were used depending on the particular objective. Namely, both probit and bivariate probit models were used to test the formative efficiency of e-tests (see Table 3) taking into account students’ demographic, socio-economic and economic attributes included in Table 1. A probit model (rather than a logit) was utilised to analyse the first outcome variable (the likelihood of sitting the examination) since it maximises the log pseudo-likelihood, whereas a bivariate probit model was applied for the other two outcome variables (the likelihood of passing the theory and practical papers), since it is a model category specially designed for cases where two questions with very closely linked binary answers need to be answered. In this case, there should be a relatively strong a priori correlation between the two, as the factors that affect whether the student passes the theory and practical examination papers can be expected to be similar.

Table 1 Variables and descriptive statistics
Full size table

Variables

In order to isolate the effect that carrying out the e-tests has had on student performance, which is the objective of our paper, a set of variables related to students’ individual profiles has also been included in the microeconometric models given that, according to the previous academic literature, they can influence academic results. This data was taken, as aforementioned in Sect. ”Participants”, from a questionnaire (see the Appendix) filled out by students at the beginning of the course and has been classified into the following three groups:

  1. (a)

    Demographic and personal information: gender (Covarrubias et al., 2018); age (Dumford & Miller, 2018); and student’s vocation to study management at university (Salanova Soria et al., 2005).

  2. (b)

    Socio-economic information: worker (Covarrubias et al., 2018); family responsibilities (El Massah & Fadly, 2017); parents (Beattie et al., 2018); people living in the family household (Millea et al., 2018); non-local (Millea et al., 2018); wages (Covarrubias et al., 2018); and computers (Zou, 2013).

  3. (c)

    Academic information: Second session (Lara et al., 2009); Erasmus (Beattie et al., 2018); university access (Ballard & Johnson, 2005); first year at university (Herzog, 2018); grant (Glocker, 2011); number of final examinations sat and failed (Cappellari et al., 2012); and highest year (Beattie et al., 2018).

Correlation coefficients between the control variables were extremely low, which precluded any autocorrelation problems. The correlation matrix is available from the authors upon request.

Finally, a variable cluster was included to correct any heteroscedasticity problems due to any difference in the level of difficulty between the two academic years analysed, or any other differences, such as the number of holidays in the academic calendar.

Table 1 presents definitions and descriptive statistics of the dependent variables, explanatory variables, and the cluster variable.

Results and discussion

We have tested whether passing the e-tests and the number of e-tests taken on the virtual platform affect, firstly, the likelihood that students will sit the final examination (probit estimation, see 2nd column of Table 2) to measure the drop-out effect of the subject, and, secondly, whether this affects passing both the theory and practical final examination papers (bivariate probit, see 3rd and 4th columns of Table 2) to check whether the e-tests implemented can be converted into an appropriate e-CA that substitutes the final exam.

Table 2 Results of probit and bivariate probit estimations
Full size table

Since estimated coefficients in discrete choice models such as logit, probit and bivariate probit models cannot be interpreted directly, the marginal effects at the mean have been calculated (Table 3). The marginal effect measures the change in the likelihood of the dependent variable following a 1% change in one of the regressors, expressed as a percentage.

Table 3 Estimations of the marginal effects at the mean of the coefficients in Table 2
Full size table

In the first place, relevant findings can be derived from Tables 2 and 3 regarding the influence of the variables related to students’ individual profiles on both the likelihood of sitting the final examination and the likelihood of passing the theory and/or practical exam papers.

For example, regarding demographic and personal variables, unlike other studies such as Swope and Schmitt (2006), the results of this study show different gender effects on performance, but only for the theory paper. Tables 2 and 3 also show that vocation is a factor that has a positive relationship with academic performance and passing both the theory and practical papers (Hsieh, 2019).

With respect to student’s socio-economic information, the results for the relationship between working and living away from the family home and academic performance show that there is only a positive relationship with passing the practical paper (25% of the final mark), which would be insufficient to guarantee a pass mark for the whole subject (Beattie et al., 2018). An interesting result is the fact that having computers in the family household is one of the determining variables for our analysis as it positively and significantly affects all the outcome variables, that is, it increases the likelihood of sitting the examination and passing the theoretical and practical papers. And, once again this is not a fixed effect but increases as the number of computers in the student’s household increases, as a higher number enables access to a computer at any given time without having to compete for its use with any other family member in the household. So, students who have three computers in their households will see a 7% (= 2.328% * 3) increase in the likelihood of sitting the examination, a 12.35% (= 4.116% * 3) greater likelihood of passing the theoretical paper and an 11.5%. (= 3.826% * 3) greater likelihood of passing the practical paper compared to students who do not have any computers in their households. These results are similar to those of previous works such as Zou (2013).

Likewise, an analysis of students’ academic information shows that there are characteristics that positively correlate with the likelihood of sitting the final examination, such as students being in their first year at university; attending classes in the afternoon-evening (second session); repeating the academic year (final examination sat and failed) and studying other subjects in higher years (highest year). These last two characteristics may indicate a greater urgency or need to pass the subject and, therefore, to sit the final examination. If we examine the relationship of these variables with performance, attending afternoon-evening classes is found to reduce the likelihood of passing the final examination (Lara et al., 2009). However, being a new student is not related to performance (contrary to the findings of Kara et al., 2009). The number of final examinations sat and failed is seen to increase the likelihood of passing both the theory and practical papers, which contradicts Cappellari et al. (2012). Finally, financing university studies with scholarships increases the likelihood of passing the final examination due to an increase in the likelihood of passing the theoretical paper (Marcerano & Navarro, 2007; Whalen et al., 2009).

Regarding the objective of this paper, the results of Table 3 show that increasing the number of computing tests that students take has a positive relationship with the likelihood of sitting the final examination, that is, there was a 14.37% increase in the likelihood of sitting the final examination when students took three tests compared to students who took only one test (7.184% is the marginal effect of the number of tests. This value has to be multiplied by two to include the effect between having taken only one test and having taken the maximum number of tests possible, three).

We also found an additional favourable relationship for students tested on the virtual platform (Tables 2 and 3). It was observed that when the number of tests taken increased, the likelihood of passing the theory paper also increased (students taking 3 tests would have an 18.2% greater likelihood of passing the theory paper than students taking only one).

Based on the two results mentioned in the previous paragraphs, it can be concluded that when instructors opt for e-CA, there is a clear incentive to increase the number of tests. Given the lower cost involved in terms of instructor workload compared to pen and paper tests, this scenario would seem to be quite feasible. The cost in time is expected to decrease progressively; an electronic library of possible test questions will be compiled as academic years go by, and, once this e-library is sufficiently large, e-tests can be coded automatically with the random selection function that can usually be found on university virtual teaching platforms.

Furthermore, these e-tests could act as a predictor of a student’s final academic performance, thereby converting it into an appropriate instrument that organises the e-CA of the subject. Namely, the students with over half the multiple-choice questions correct are 14.16% more likely to pass the theory paper. The e-tests provide good feedback to students on how to assimilate the subject matter. Although it only refers to the theory papers, this result implies that e-tests constitute an excellent low-cost tool for teachers to tutor students. Thanks to these e-tests, the teacher would be able to see in advance of the exam which students are likely to need more educational support during the course in order to be able to pass the subject. They are also an essential tool for students, who would be able to assess their learning process prior to the final examination.

However, a non-statistically significant effect is observed on the practical exam. This is in line with Alauddin & Khan (2010) and reveals that the effectiveness of the e-test format could be less appropriate for more practical subjects, such as those based around mathematical problems.

Conclusions

The European Higher Education Area focused on the Continuous Assessment (CA) of students’ knowledge and competences according to the performance of various evaluation tools that differed from the traditional system, which was based exclusively on a final exam. In our research context, that of the Business Administration and Management Degree of the University of Seville (Spain), this process of adaptation towards CA was initiated in 2010 by means of the application of a programme of evaluation techniques based at that time on conventional written activities. The reduced number of students (less than 5%) who finally opted to choose this system to take the subject, together with the modest results obtained in terms of grades, prompted the need for the application of other alternatives to develop CA, which, in addition to making the system more attractive to students, would prevent the premature dropping out of the subject, and would allow teachers greater agility in planning the assessment process, given the high number of students enrolled in the subject per academic year in recent years (approximately 800 on average in each course). The adoption of e-CA (through the use of a variety of tools such as digital platforms, mobile web-enabled technologies, and interactive response systems), widely integrated into higher education and whose transition has been particularly accelerated due to the recent COVID-19 pandemic, was presented as a relevant option in this context, given that, among its advantages recognised in the literature, it is known for providing immediate feedback and flexibility in adapting to the teaching/learning process, both for the teacher and the student.

Considering all the above, the current paper presents the results of the development of a pilot trial of e-CA based on online tests (e-tests) performed on a virtual platform, carried out on a sample of 255 students enrolled over two successive academic years (2016–2017 and 2017–2018) in the subject of Principles of Economics of the first-year in the Business Administration and Management Degree. From these results, and by applying microeconometric models, our research goal is to analyse the impact of these digital tools on students’ academic performance, in order to determine their degree of substitutability and/or complementarity with respect to traditional CA based on conventional written tests that had been performed for many years in the subject.

Regarding the research questions raised in the Introduction section, the results derived from our models lead us to conclude that not only has our e-CA pilot study based on e-tests rendered the choice of CA more attractive for the students thereby diminishing the probability of dropping out, but it has also made it possible to attain better academic results in the final exam, particularly with respect to the theoretical contents of the subject.

In summary, our findings provide evidence that a key role is played by technologies and digitalisation as support to maximize learning and to prevent the students from dropping out in Higher Education, for various reasons. Among these, it is first worth pointing out the fact that a considerable increase is detected in the number of students who choose to follow the subject through CA, which, indirectly, has enabled them to prepare more continuously for the subject, and has thereby positive repercussions on the results obtained in the final examination. Thus, students who pass the periodic online tests were 14.16% more likely to pass the subject, although this favourable effect becomes more diluted if we consider the practical part of the final examination based on practical exercises. This relationship between the implementation of the e-CA and the improvement in academic performance can also be observed in the average rate of students passing in relation to those presented in the final exam for the subject. In the two years prior to the application of the e-tests, the pass rate was 47%, while in the academic years analysed (2016–2017 and 2017–2018), this figure rose to 53%.

Secondly, the higher the number of online tests that students took, the greater the likelihood of their sitting the final examination (Abio et al., 2019). This finding is even more relevant in our case study, since the Business Administration and Management Degree has an initial dropout rate of 22.55% for first-year students at the University of Seville, which triggers the continuous search for teaching strategies to motivate students to persevere, thereby preventing at-risk students from failing and from not completing the course.

For all these reasons, and based on the results obtained, an e-CA plan for the subject has been designed and implemented, which is mainly focused on the e-tests on theoretical questions of the program, and complemented by practical exercises, as described in the Introduction section. Among the results obtained for the rest of the control variables used in the analysis, another manifestation can be appreciated of the success of digitisation as a learning tool through the socio-economic variable ‘having computers at home’ which is one of the most determining variables in our analysis. Estimates show that the existence of computers in the student’s family household is one of the determining variables that positively and significantly affects all the outcome variables, that is, it increases the likelihood of sitting the examination and passing the final examination. Its effect is not fixed but increases in line with any increase in the number of computers in the student’s household. Therefore, according to this analysis, the positive effect of a computing environment does not appear to be limited to the classroom, but also extends to students’ home environments.

Our principal limitation is that our results were obtained before the COVID-19 pandemic but the rapidly changing educational landscape throughout the world and new challenges that universities must address henceforth, such as the spread of online and remote teaching, show them to be relevant and timely. Hence, by taking the results collected here as a guide from the pilot experiences developed in previous academic years, and since the CA system has been fully implemented in the subject since the 2020–21 academic year, we propose a future line of research that employs a time series model with intervention analysis, once a sufficiently large sample of multiple academic courses becomes available, in order to determine the validity of these results in the medium and long term. This will, at the same time, enable testing the satisfaction of the student and the teacher with the system.