Determining the financial performance of the firms in the Borsa Istanbul sustainability index: integrating multi criteria decision making methods with simulation

Abstract

Regardless of the industry in which a company operates, evaluating corporate performance is one of the most critical and vital processes; the most essential and prominent performance evaluation is related to financial performance. Appropriate performance analysis is complex and critical for decision-makers in different financial performance factors; thus, a methodological framework is needed to solve such complex decision problems. Therefore, this research aims to rank the companies included in the sustainability index (excluding banks) in Turkey by considering their financial performance. The criteria weights were determined using the full consistency method (FUCOM) by considering the evaluations of four experts. The firms were ranked using nine multi-criteria decision-making methods. The consensus among the nine rankings was ensured with the Copeland technique. The decision matrix includes financial ratios and the stock market performance of the firms; 100,000 FUCOM weights were created with random evaluations to validate the results. The results indicate that the most crucial criterion is the current ratio by considering expert evaluations. Weight simulation indicates that alternative 16 (alternative 21) is superior (inferior) to the other alternatives, even though the weights are determined with random evaluations. Ranking with expert evaluations is similar to the mean of the weight simulation results. The results demonstrate that the proposed framework can be performed as a basis for financial performance ranking.

Introduction

In recent years, companies have widely used financial performance as an essential indicator of management performance (Cheng et al. 2012). Company managers must measure the financial performance of their companies (Tsolas 2020), and recently, it has become more critical to measure the financial performance of companies, especially in the financial sector (Yalçın and Bayrakdaroğlu 2012). In today’s competitive environment, reliable and accurate determination of a company’s financial performance is crucial for managers, creditors, current/potential investors, and companies operating in the same sector (Farrokh et al. 2016; Alossta et al. 2021). The financial performance evaluation of listed companies is critical for both shareholders and investors (Dong et al. 2018), especially with economic globalization and financial innovation; therefore, evaluating the financial performance of companies is a valuable research topic for investors and researchers (Inani and Gupta 2017; Muhammad et al. 2021).

Companies must be ranked by their financial performance to know their position against their competitors. Owing to these performance evaluations, companies can determine the strategies needed to increase their financial performance (Lam et al. 2021). Since the financial performance evaluation includes many evaluation criteria, it is considered a kind of multi-criteria decision-making (MCDM) problem (Dong et al. 2018; Yalçın and Ünlü 2018). MCDM analysis determines the best alternative by considering more than one criterion or factor that affects the other options (Lam et al. 2021). MCDM methods, widely used in business and engineering, enable decision-makers to make more rational and effective decisions regarding alternatives (Deng et al. 2011). Nonetheless, financial ratios are generally used as evaluation criteria, as they can fully reflect the information regarding the companies’ financial status (Dong et al. 2018), revealing financial strengths and weaknesses (Lam et al. 2021). Over the years, many studies have shown the effectiveness of financial ratios in performance measurements (Yalcin et al. 2012).

The criteria weights that directly affect the results of MCDM methods can be determined subjectively and objectively, and expert opinions generally determine subjective weights. In contrast, objective weights are mainly determined according to the data set’s essential characteristics; however, it is crucial to objectively determine the weights of the criteria to create more accurate rankings.

This research aims to rank the companies included in the 2021 sustainability index in Turkey by considering their financial performance. The dataset was collected from two different sources, and the criteria weights were determined with the full consistency method (FUCOM). Four experts completed surveys and created four different weight sets, with the arithmetic average of the four weight sets calculated to obtain a single weight set. There are 22 firms in the Borsa Istanbul (BIST) sustainability index (excluding banks since their financial statements differ from other firms). There are nine criteria, including the stock market performance of the firms. The dimensions of the decision matrix are 22 by 9 for 2021. Nine techniques were employed to evaluate the alternatives: combined compromise solution (CoCoSo), grey relational analysis (GRA), multi-attributive border approximation area comparison (MABAC), multi-attribute ideal real comparative analysis (MAIRCA), multi-objective optimization based on simple ratio analysis (MOOSRA), operational competitiveness ratings (OCRA), the technique for order of preference by similarity to ideal solution (TOPSIS), the Portuguese acronym for interactive MCDM (TODIM), and multi-criteria optimization and compromise solution Vise Kriterijumska Optimizacija I Kompromisno Resenje (VIKOR). Although significant efforts have been made to develop many multi-criteria techniques, no comprehensive methodology exists, and no single method appears to be better than its counterparts (Saaty and Ergu 2016; Varmazyar et al. 2016); each technique has its superiority in identifying the weights of factors. According to Kou et al. (2021a), a hybrid approach involves utilizing various MCDM models to assign weights to criteria and rank alternatives. As a result, hybrid methods enhance the objectivity of these outcomes. In this research, a consensus among the different MCDM techniques is ensured with the Copeland technique. Kou et al. (2014) state that a ranking obtained through the consensus of several MCDM methods is considered more reliable than a ranking produced by only one MCDM method; thus, a single score (and rank) is calculated for each alternative. If the two alternative scores differ significantly (in other words, if the difference between them is high), it is possible to say that the alternative with high scores is superior to the alternative with low scores. Conversely, if the difference between the scores is insignificant, it is difficult to determine whether there is a practical difference between the alternatives. Practical significant differences are always an issue of judgment and interpretation for the decision-maker; however, the discipline of statistics can guide the issue of statistically significant differences (Rosenbloom 1997). This study created 100,000 FUCOM weight sets of random values to examine the alternatives’ statistical superiority (or inferiority). The alternative that exhibits statistical superiority is determined using 100,000 scores.

The proposed framework presents a comprehensive and reliable performance evaluation tool to help determine financial performance rankings. This paper’s objectives can be summarized as follows.

• This study aims to realize a sustainability performance evaluation of companies using the stock market return and financial ratios of the companies listed in the BIST sustainability index. The companies’ stock market and financial performances were then integrated into a single decision matrix.

• We created a weight set using the FUCOM technique with four experts.

• This work applied nine MCDM techniques (CoCoSo, GRA, MABAC, MAIRCA, MOOSRA, OCRA, TOPSIS, TODIM, and VIKOR) to evaluate the alternatives.

• Using the Copeland technique, we ensured the consensus among the ranking results of various MCDM tools.

• A weight simulation process was applied to determine the statistically superior alternatives.

• The proposed methodology was repeated in the years 2019 and 2020.

This study is structured as follows. After this introduction, the second section presents a review of the related literature and discusses the importance of evaluating financial performance and the studies in which financial performance is determined using MCDM techniques. The third section is dedicated to the methodology, where the research flowchart, performance indicators, and MCDM techniques are presented. In addition, the analysis and processes to be carried out at each stage of the proposed framework are presented. The results are given in the fourth section, and the results are discussed in the fifth section with managerial emphasis. Finally, the sixth section is dedicated to the conclusion and directions for future research.

Literature review

Performance evaluation investigates whether the company’s goals and objectives have been achieved (Chang and Tsai 2016) and whether resources have been allocated efficiently. It is applied for operational control purposes in the short-term and strategic management and planning purposes in the long-term (Wu et al. 2009). In a competitive environment, companies generally aim to compete in the international market and be at the top of the sector in which they operate. Financial performance evaluation is one of the most important indicators of whether these targets have been achieved. At this point, analyzing companies’ financial ratios effectively reveals their strengths and weaknesses (Abdel-Basset et al. 2020). Financial performance is performed for stakeholders, including company owners, managers, investors, competitors, and creditors (Bağcı and Yerdelen Kaygın 2020).

Different methods have been used to determine the financial performance of companies. In financial performance evaluation, discriminant analysis (Mihalovic 2015; Keskin et al. 2020a), a balanced scorecard (Davis and Albright 2004; Cohen et al. 2008; Knápková et al. 2014), Generalized Method of Moments (GMM) analysis (Kılıç et al. 2022), and data envelopment analysis (Dekker and Post 2001) have been used. MCDM is another method frequently used in financial performance evaluation (Ersoy 2021; Dukić et al. 2022). Most financial performance evaluations are considered MCDM problems (Abdel-Basset et al. 2020). The use of MCDM in measuring companies’ financial performance has become widespread since the 1980s (Erdoğan et al. 2016). Researchers have used it since the early 2000s as it helps financial information users make more accurate decisions, especially in a complex environment where the number of criteria related to financial performance is high (Baydaş and Elma 2021). MCDM methods are essential in solving multidimensional and complex problems (Lee et al. 2012). The most important feature distinguishing MCDM from other methods is that it provides a suitable framework for the decision-maker in case of many alternatives and conflicting criteria (Ersoy 2021).

Financial ratios generally determine the financial performance of companies. Using financial ratios in performance evaluation has a long history, and there has been a significant increase in these ratios in recent years (Alimohammadlou and Bonyani 2017). Financial ratios are essential evaluation tools to understand the profitability of companies and analyze their financial situation (Aldalou and Perçin 2020; Bakır et al. 2021); however, having financial data alone is not enough to evaluate financial performance. Furthermore, financial statements offer only a momentary glimpse into a company’s financial position from the previous year and fail to depict its current operational state (Kou et al. 2021b). For this reason, firms and information users use financial ratios as data and calculate financial performance through statistical and econometric models, including regression analysis, correlation analysis, time series analysis, and MCDM methods (Bağcı and Yerdelen Kaygın 2020; Osintsev et al. 2021; Narang et al. 2022). However, with too many ratios or criteria, MCDM methods have proven to be quite successful in determining financial performance (Visalakshmi et al. 2015). Nonetheless, considering stock market indicators, such as stock returns and financial ratios, in evaluating financial performance is beneficial for users of financial information, as it can assist them in their decisions (Jokić et al. 2021).

Many studies use MCDM methods in financial performance evaluation, as shown in Table 1. These studies primarily aim to rank the alternatives according to financial performance criteria and identify the companies with the highest performance.

Table 1 Summary of related studies

Table 1 shows that different MCDM methods rank firms according to their financial performance. It has been understood that various methods are increasingly preferred in ordering alternatives. As shown, most of the studies use more than one method together, and the criteria weights are determined mainly by the fuzzy AHP, entropy, and AHP methods.

There are studies in which companies’ financial performances included in the BIST sustainability index are determined using different methods. For example, Ates (2020) investigated the effect of sustainability performance on financial performance using regression models. Similarly, Akben-Selcuk (2019) used regression models and examined the impact of corporate social responsibility on financial performance. Using the Hirose method, Karaömer and Oypan (2020) determined the financial performance of six banks in the BIST sustainability index. Acar and Temiz (2018) and Çimen (2019), investigated the effect of the sustainability index on firm performance by conducting an event study analysis. Using Mann–Whitney U, Kruskal–Wallis methods, and panel data analysis, Dinçer and Altınay (2020) examined the effect of disclosures in sustainability reports on financial performance. Using discriminant analysis, Keskin et al. (2020b) investigated the impact of sustainability on financial performance. In addition, Şahin et al. (2017) used a T-test analysis and examined the effect of sustainability on the financial performance of 15 firms in the BIST sustainability index.

The literature review revealed that FUCOM was not frequently used in determining the weights of the criteria in an MCDM framework to evaluate the performance of the companies in an emerging economy like Turkey. Furthermore, weight simulation was not employed with the FUCOM technique. The fact that the alternative evaluation was not conducted with random weight sets is an essential gap in the literature because, together with the simulation study, it is possible to determine the statistically superior (or inferior) alternative.

Methodology

This research determined firms’ financial performances using four different MCDM methods. Previous that determined companies’ financial performances used more than one financial performance indicator to make closer and more accurate assessments. In this context, the companies are ranked according to their financial performance using eight financial ratios and stock returns. According to Baydas and Pamučar, stock return is a significant financial indicator for research that determines financial performance together with MCDM methods (Baydas and Pamučar 2022). Table 2 shows the financial performance indicators and MCDM methods used in the research.

Table 2 Indicators and methods

This study used 11 MCDM techniques together. The FUCOM technique is used to determine the order of importance of the criteria, nine techniques are used to rank the alternatives, and finally, one technique is used to reach a consensus among the nine rankings. In addition, random weight sets are created to provide a statistical interpretation of the results. Repeating the analysis with random weight sets has the following benefits:

• It can be determined whether the evaluations made by the experts are an extreme value. If random assessments are generated, some weights should occur very few times, and others should occur more frequently. In other words, a distribution of the weight of a criterion will appear. If the experts’ assessments are at the extremes of this distribution, it can be stated that the experts made an extreme value assessment; therefore, concerns about the health of the assessment may arise. Conversely, if the experts’ evaluations are close to the mean in this distribution, it may be concluded that their evaluation is reasonable.

• Evaluations made by experts and simulation results with random weights can be compared. According to Xiao et al. (2023), the growing complexity of discrete event dynamic systems has increased the usage of simulation for their evaluation. In this way, the number of evaluations made by the experts in the interquartile range in random evaluations can be calculated, and the consistency of the evaluations can be revealed. If most of the alternatives are in the interquartile range according to the experts’ evaluations, it is possible to ensure that their evaluations are not outliers.

• A table can be created regarding how many sets of weights an alternative is ranked first, second, third, and so on. Such a table reveals that the alternative is ranked higher not only in one weight set but also in more than one weight set. In other words, even if the weights are determined randomly, it will be possible to determine that an alternative is a superior (or inferior) alternative.

• With random evaluations, the statistical superiority of the alternatives over each other can be revealed; thus, for example, we can determine the number of total sets of weights between differently ranked alternatives. These numbers can be determined for each pairwise comparison.

• Since the calculation is made with a large number of weight sets, it can be statistically determined whether there is a difference between the group means with the help of ANOVA analysis. As a continuation, post hoc analysis can be performed to determine which groups have a statistical difference between their means; thus, it will be possible to compare distribution means and perform statistical analysis instead of comparing with a single ranking.

Figure 1 shows the four methodological steps of the research as follows:

Fig. 1

Flowchart of the research

Step 1 Creating the decision matrix.

First, a decision matrix was created based on performance indicators. Next, nine indicators were used to determine financial performance.

Step 2 Determination of criterion weights.

The criteria weights were determined using the FUCOM method. This method was preferred because it integrates valuable expert judgments in decision-making. Therefore, the weights of all the criteria were determined according to this method. Four expert evaluations were collected at this stage, and the average of the weight sets was employed as the final weight set.

Step 3 Calculating and ranking the scores of the alternatives

Nine MCDM techniques were selected as ranking methods, and calculation steps were performed on the MATLAB platform; each method may produce different ranking results when more than one MCDM method is used. In such cases, integration methods are used to integrate different results and obtain a result. This step aims to integrate the different results suggested by each MCDM method using the Copeland method.

Step 4 Testing the validity of the results

In this step, a weight simulation was performed. Expert opinions were simulated with the help of random evaluations; 100,000 random evaluations were carried out, and as a result, 100,000 weight sets were created. Then, nine techniques were run for each weight set, and finally, the results were combined with the Copeland technique. As a result, 100,000 rankings emerged, allowing statistical evaluation of the results.

The purpose of weight simulation is to examine the system behavior in the case of many random weight sets. In the weight simulation, instead of only converging a single ranking, the superiority of the alternatives over each other is statistically examined; thus, for example, instead of concluding that "the first alternative is superior to the second alternative," it is possible to comment that "90% of the 10,000 weights are ranked in a higher position."

Financial performance indicators

The study used MCDM methods to reveal companies’ financial performances; several indicators were also used to achieve this aim. These indicators are current ratio, acid-test ratio, debt ratio, asset turnover, stock turnover, Earnings before Interest Taxes, Depreciation, and Amortization (EBITDA), net profit margin, return on equity (ROE), and stock return. Additionally, the debt ratio is a cost variable, whereas other variables are benefit variables. The justification for selecting these performance indicators can be explained as follows. ROE is the most essential and well-known ratio used in financial performance evaluation. Stock return is a significant financial indicator for research in which financial performance is determined together with MCDM methods. Furthermore, the current and acid-test ratios are the most popular ratios used to determine the future risks of the firm as well as its financial performance (Baydas and Pamučar 2022). The current ratio is a vital liquidity ratio commonly used by financial analysts and investors (Ghosh and Bhattacharya 2022). Moreover, Bhadu et al. stated that the current ratio is a crucial measure of the financial performance of firms (Bhadu et al. 2021). EBITDA is vital in determining shareholder returns for the relevant period (Buračas et al. 2015). In this respect, this ratio allows an understanding of the relationship between financial performance and the relevant stakeholders (Puška et al. 2023). The net profit margin is the most critical indicator of the firm’s financial and operational performance (Estiasih and Putra 2021). Aytekin stated that the current ratio, acid-test ratio, net profit margin, ROE, debt ratio, asset turnover, and stock turnover are the most used ratios in the literature to determine financial performance with MCDM methods (Aytekin 2019). The literature maintains two fundamental views on evaluating financial performance: traditional and modern. According to the traditional view, the stock return and the debt ratio are essential determinants of financial performance (Tavana et al. 2015), while stock turnover is an actual indicator of production performance (Bhadu et al. 2021). Explanatory information regarding these indicators is given below.

Current ratio

The current ratio is an essential indicator of short-term financial stability. The ratio allows the firm to compare its current assets with its current liabilities; therefore, the rate is expected to be high. A high current ratio guarantees that creditors meet short-term obligations (Bhadu et al. 2021).

Acid-test ratio

The acid-test ratio is calculated by dividing the current liabilities by the value resulting from deducting the stocks from the total current assets; stocks are not considered as their liquidity ratios are low (Akyüz and Bilgiç 2016).

Debt (leverage) ratio

The debt ratio is the ratio of total liabilities to total assets and is mainly used for information about long-term debts (Shaverdi et al. 2016). Furthermore, this ratio reveals the ratio of assets acquired by the firm using debt (Abdel-Basset et al. 2020).

Asset turnover

This ratio reveals the efficiency of the total resources the firm uses to make sales (Ertuǧrul and Karakaşoǧlu 2009). It also refers to the ability of the firm’s assets to be used to sell or generate profits (Abdel-Basset et al. 2020).

Stock turnover

The inventory turnover rate expresses the efficient and effective use of company stocks, and this rate can be measured monthly. A low inventory turnover rate indicates that the firm is overstocked or has excessive previous inventory (Roy and Shaw 2021).

Earnings before interest taxes, depreciation, and amortization (EBITDA)

This ratio is calculated by accounting for the operating profit, ignoring the interest, tax, and depreciation amounts, which indicates the firm’s ability to generate cash (Açıkgöz 2020).

Net profit margin

This ratio shows the firm’s amount in stock after all expenditures, including legal payments. In addition, this profitability ratio provides users with information about the company’s commercial activities (Roy and Shaw 2021).

Return on equity (ROE)

ROE shows the actual expenditure costs incurred against the expenditures made. This ratio is affected by the amount of debt businesses use to finance their assets. A high ratio indicates that the use of equity is efficient, and investors can obtain higher returns (Shaverdi et al. 2016). Table 3 shows the performance indicators, the formulas of the indicators, and references.

Table 3 Performance indicators, formulas, and references

MCDM tools

MCDM methods facilitate decision-makers’ work; they can be used in many areas requiring important decisions (Abdelli et al. 2020). These practical decision-making tools are used to evaluate and rank the alternatives for the decision. Each method has different basic features, advantages, and disadvantages (Chowdhury and Paul 2020; Badi et al. 2022); therefore, more than one method can solve the same problem and make more accurate decisions (Lee and Chang 2018). Although the number of MCDM methods has significantly increased in recent years, it is difficult to determine which methods are more appropriate and correct for any decision problem (Peng et al. 2011; Kiptum et al. 2022).

This study used CoCoSo, GRA, MABAC, MAIRCA, MOOSRA, OCRA, TOPSIS, TODIM, and VIKOR techniques to rank the alternatives. The calculation steps of the techniques are similar. For example, each technique requires a decision matrix, criteria of benefit or cost, and weight values data (Some techniques also have additional parameters). The decision matrix is normalized according to the benefit or cost criteria, and then the calculations continue with the help of different formulas in each technique. These nine techniques were chosen because of this similarity, and the practical aspects of each technique (additional reasons for inclusion in the analysis) are presented below.

Fully consistency method (FUCOM)

FUCOM is a comparison-based MCDM method that accepts the deviation from maximum consistency and pairwise comparison principles as basic assumptions (Feizi et al. 2021). This method determines criteria weights by subjective judgments; decision-makers rank the criteria according to their preferences and make pairwise comparisons of the criteria they rank. The most crucial difference between other subjective methods is that FUCOM shows minor deviations from the optimal values in the criterion weights (Stević and Brković 2020). In this method, few comparisons are made, and constraints are defined while determining the optimal values of the criteria; thus, the method minimizes the possibility of error in comparisons. In particular, methods such as BWM and AHP determine criterion weights with high pairwise comparisons, increasing the possibility of error (Pamučar et al. 2018).

The following steps are applied to determine the criterion weights according to the FUCOM method (Pamučar et al. 2018; Stević and Brković 2020):

Step 1: The experts rank the criteria/sub-criteria—the importance level of the criteria considered in the ranking.

$${C}_{j(1)}>{C}_{j(1)}>\dots >{C}_{j(k)}$$

(1)

where k is the rank of the criteria. The equality sign is used for criteria of equal importance.

Step 2: The ranked criteria are compared, and their comparative priority is determined.

$$ {\Phi } = (\varphi_{1/2} ,\varphi_{2/3} , \ldots , \varphi_{{k/\left( {k + 1} \right)}} ) $$

(2)

where ${\varphi }_{k/k+1}$ represents the importance (priority) of ${C}_{j(k)}$ over ${C}_{j(k+1)}$.

Step 3: The final values of the weight coefficients of the criteria are determined, considering two conditions:

Condition 1: The ratio of the weighting coefficients of the criteria should be equal to the comparative significance between the criteria.

$$\frac{{w}_{k}}{{w}_{k+1}}={\varphi }_{k/(k+1)}$$

(3)

Condition 2: The values of the weight coefficients have a mathematical transitivity condition.

$$ \frac{{w_{k} }}{{w_{k + 2} }} = \varphi_{{k/\left( {k + 1} \right) }} \otimes \varphi_{{\left( {k + 1} \right)/\left( {k + 2} \right)}} $$

(4)

Step 4: The model is defined to calculate the final values of the weighting coefficients of the criteria.

$$ \begin{aligned} & {\text{min}}\;\;x \\ & {\text{s}}.{\text{t}}. \\ & \left| {\frac{{w_{j\left( k \right)} }}{{w_{{j\left( {k + 1} \right)}} }} - \varphi_{{k/\left( {k + 1} \right) }} } \right| \le x, \;\forall j \\ & \left| {\frac{{w_{j\left( k \right)} }}{{w_{{j\left( {k + 2} \right)}} }} - \varphi_{{k/\left( {k + 1} \right) }} \otimes \varphi_{{\left( {k + 1} \right)/\left( {k + 2} \right) }} } \right| \le x, \;\forall j \\ & \mathop \sum \limits_{j = 1}^{n} w_{j} = 1 \\ & w_{j} \ge 0, \;\forall j \\ \end{aligned} $$

(5)

Step 5: The final values of the evaluation criteria/sub-criteria ${({w}_{1},{w}_{2},\dots ,{w}_{n})}^{T}$ are calculated.

Combined compromise solution (CoCoSo)

Three collection strategies, SAW, weighted aggregated sum product assessment (WASPAS), and exponentially weighted product (EWP), are integrated to obtain reliable and stable results in the CoCoSo method. This integration distinguishes CoCoSo from other MCDM methods (Ecer 2021). This method ranks alternatives according to their collective performance score and envisages the integration of the weighted-sum model and weighed-product model methods to determine the sum and power of the weighted comparability sequence (Kumar et al. 2022). The essence of this method is the combination of compromise perspectives, which distinguishes it from other MCDM techniques; it also includes the estimation of the final solution consensus, albeit with conflicting criteria (Ulutaş et al. 2021). As Ecer (2021) indicated, the $\lambda$ parameter in the method is fixed at 0.5. Furthermore, in this study, the $\lambda$ parameter is fixed at 0.5 in all calculations.

Grey relational analysis (GRA)

GRA is an integral part of the body of knowledge of the grey system theory proposed in 1982 by Deng Julong, followed by the development of its first GRA model in 1984. Deng’s GRA is a technique for absolute measurement (or normative evaluation). It estimates a degree called grey relational grade, which is ideally a weighted average of grey relational coefficients and is essentially a positive correlation metric (Javed et al. 2022). One of the main advantages of the gray systems theory is that it provides satisfactory results from small quantities of data and many factors of variables (Malek et al. 2017).

Multi-attributive border approximation area comparison (MABAC)

The MABAC technique was developed by Pamučar and Ćirović (2015). The technique developers applied a sensitivity analysis consisting of three stages and reported that MABAC showed stability (consistency) of its solution in all cases. The basis of the MABAC method is seen in the definition of the distance of the criterion function of each alternative from the border approximation area (Pamučar and Ćirović 2015), comprising regions, upper, lower, and border approximation areas. The upper (lower) approximation area contains the ideal (anti-ideal) alternative. The MABAC approach needs simple mathematical operations, integrates the gains and losses easily, allows combining with other methodologies, and creates functional outcomes (Pamučar and Ćirović 2015; Sun et al. 2018; Aydin et al. 2022).

Multi-attribute ideal real comparative analysis (MAIRCA)

The most important advantage of the MAIRCA method is the different linear normalization approach, which contributes to obtaining more effective results (Ecer 2021). Like TOPSIS, this method focuses on the positive and negative ideal solutions (Gul and Ak 2020). Furthermore, the method considers the gap between the ideal and empirical ratings; each criterion sums this gap, and as a result, the total gap for each alternative is formed. Finally, an alternative with the lowest gap value was selected (Gul and Ak 2020).

Multi-objective optimization based on simple ratio analysis (MOOSRA)

The MOOSRA technique was developed by Das et al. (2012). This technique calculates the simple ratio of the beneficial and cost criteria. Negative values do not appear during the calculation process, and results are less sensitive to variation in the rational values of the criteria (Narayanamoorthy et al. 2020). This method also requires less computational time, is more simplistic and more stable, and requires minimal mathematical calculations (Sarkar et al. 2015).

Operational competitiveness ratings (OCRA)

The OCRA technique was developed by Parkan in 1994 (Parkan 1994). It aims to evaluate the operational competitiveness of the production units. The OCRA method adopts an intuitive approach for capturing the experts’ inputs and can also consider the dependence of the criteria weights on the alternatives. The OCRA methodology has been used as a robust MCDM tool for sequencing problems (Thakur 2022).

Technique for order of preference by similarity to ideal solution (TOPSIS)

In the TOPSIS method, the deviation of the best alternative from the perfect positive solution should be minimum, and the geometric separation from the ideal-negative solution should be maximum. Therefore, this method includes determining each criterion ‘s weights, normalization, geometric distance, and ideal solutions (Chodha et al. 2022). In this method, the most suitable alternative is the one closest to the positive ideal solution and the farthest from the negative ideal solution (Khan and Maity 2017). TOPSIS can be considered one of the most well-known MCDM techniques.

The Portuguese acronym for interactive and multi-criteria decision-making (TODIM)

TODIM was developed by Gomes and Lima (1992). TODIM has some advantages, such as simple and easy application, readily comprehensible for practitioners. TODIM relies on prospect theory, which explains how individuals make decisions when facing risk. In this theory, individuals respond asymmetrically to gains and losses; that is, losses with the same level of gains have a higher absolute value. This response-level difference can be quantitatively embedded in TODIM with an attenuation factor (Alali and Tolga 2019). This study’s attenuation factor is fixed at 0.5.

Vise Kriterijumska Optimizacija I Kompromisno Resenje (VIKOR)

The VIKOR method was developed for the multi-criteria optimization of complex systems in 1998 by Opricovic (Opricovic 1998). It determines the compromise-ranking list, the compromise solution, and the weight stability intervals for the preference stability of the compromise solution obtained with the initial (given) weights. This method focuses on ranking and selecting from a set of alternatives in the presence of conflicting criteria. It introduces the multi-criteria ranking index based on the particular measure of “closeness” to the “ideal” solution (Opricovic and Tzeng 2004).

Copeland

Copeland considers the number of wins and losses for each determining criterion and ranks the options. The winner is the one who compares with all alternatives and provides an advantage (Naderi et al. 2013). Copeland ranked options favorably on all MCDM assessments (Ecer 2021). Different MCDM methods can give different results. For example, while the best alternative for the X method is A, the most suitable alternative for the Y method might be B. In such cases, it is unclear which method’s results are reliable or which alternative to choose. The Copeland method is used to solve this critical problem and obtain a generally accepted ranking, considering the different results (Beheshtinia and Omidi 2017).

Analysis results

Data set

This study uses MCDM methods to evaluate the financial performances of 22 companies (banks excluded) in the sustainable index every year between 2019 and 2021. The sustainability index in Turkey was first published in November 2014. Nine different performance indicators were used as decision criteria. The financial performances of the companies were calculated separately for each year. These data were for 2019 2020, and 2021. The financial statement data were retrieved from finnet.com, and stock market data were retrieved from finance.yahoo.com; for each firm in the dataset, the yearly return was calculated with the equation presented in the last row of Table 4. Moreover, Table 4 shows the initial decision matrix with non-normalized data for 2021.

Table 4 Decision matrix (2021)

Weight determination with the FUCOM method

The authors prepared a survey and invited four experts to fill out the forms: expert 1 was an academic working in finance; expert 2 was an academic and was competent in accounting finance; expert 3 was an academic working in finance and insurance; and expert 4 was an academic working in management and finance. All experts were familiar with the working principles of the FUCOM method and were competent in their fields. Furthermore, experts were confirmed to know all performance indicators and their opinions were taken. The first step of the FUCOM technique was to rank the criteria according to their significance, presented in Table 5.

Table 5 Expert evaluation permutations

Next, the listed criteria were compared, and the comparative importance of the evaluation criteria was determined. The comparative importance of the evaluation criteria was obtained with the help of experts’ opinions, as presented in Table 6.

Table 6 Comparative significance of criteria

In the next step, the final values of the weighting coefficients of the evaluation criteria were performed using the model (5). Applying Eqs. (3) and (4) and the data in Table 3 allowed us to create a unique model for determining the weighting coefficients of the criteria for each expert.

For expert 1, the mathematical model can be expressed as follows:

$$ \begin{aligned} & \min \chi \\ & subject\,to\left\{ \begin{gathered} \left| {\frac{{w_{6} }}{{w_{8} }} - 1} \right| = \chi , \left| {\frac{{w_{8} }}{{w_{1} }} - 2} \right| = \chi , \left| {\frac{{w_{1} }}{{w_{3} }} - 1.5} \right| = \chi \hfill \\ \left| {\frac{{w_{3} }}{{w_{7} }} - 1} \right| = \chi , \left| {\frac{{w_{7} }}{{w_{4} }} - 1.33} \right| = \chi , \left| {\frac{{w_{4} }}{{w_{9} }} - 1} \right| = \chi \hfill \\ \left| {\frac{{w_{9} }}{{w_{2} }} - 1} \right|, \left| {\frac{{w_{2} }}{{w_{5} }} - 1.25} \right| = \chi \hfill \\ \left| {\frac{{w_{6} }}{{w_{1} }} - 2} \right| = \chi ,\left| {\frac{{w_{8} }}{{w_{3} }} - 3} \right| = \chi ,\left| {\frac{w1}{{w_{7} }} - 1.5} \right| = \chi \hfill \\ \left| {\frac{{w_{3} }}{{w_{4} }} - 1.33} \right| = \chi ,\left| {\frac{{w_{7} }}{{w_{9} }} - 1.33} \right| = \chi ,\left| {\frac{{w_{4} }}{{w_{2} }} - 1} \right| = \chi \hfill \\ \left| {\frac{{w_{9} }}{{w_{5} }} - 1.25} \right| = \chi \hfill \\ \mathop \sum \limits_{i = 1}^{9} w_{i} = 1,w_{i} \ge 0,\forall_{i} \hfill \\ \end{gathered} \right. \\ \end{aligned} $$

(6)

Similar models were created for each expert evaluation. The model mentioned above was solved with MATLAB, and each expert’s results (weights of the criteria) are presented in Table 7.

Table 7 Weight of criteria

Table 7 indicates that FUCOM provides entirely consistent values of weighting coefficients, as DFC = 0 for each of the four expert assessments. The final weight coefficient values were reached by taking the arithmetic average of the four expert evaluation weights. As a result of the consensus obtained with the arithmetic mean, the criteria weights can be summarized as follows. The highest weight (0.1728) belongs to the first criterion. The following criteria are the sixth (0.1409) and seventh (0.1387), and the weights of these two criteria are very close. The three lowest weights belong to the second (0.08), fourth (0.0713), and fifth (0.0588) criteria.

Nine different MCDMs were applied to the decision matrix; Table 8 presents the results, indicating that the prioritization of the alternatives based on different methods varies. For example, alternative 16 is the best alternative according to CoCoSo and VIKOR. It is the second-best alternative per GRA, MABAC, OCRA, and TOPSIS. It is the third best alternative according to the MOOSRA technique. Finally, it is ranked 21st in MAIRCA and VIKOR techniques. Managing these different rankings was a difficult task. The Copeland technique achieved a consensus between the different rankings. Based on the Copeland rankings, the rankings of the alternative are as follows $A16\succ A8\succ A13\succ A7\succ A9\succ A4\succ A14\succ A18\succ A2\succ A19\succ A3\succ A12\succ A1\succ A5\succ A6\succ A22\succ A17\succ A15\succ A10\succ A11\succ A20\succ A21$. The best alternative is the 16th, and the worst is the 21st.

Table 8 Overall scores and ranking results of alternatives

Weight simulation

Changing the weights used as input in the MCDM analysis can change the ordering, which is the output of the analysis. For this reason, it is necessary to determine the effect different weight sets have on the research results. This study created different weight sets consisting of random values, and the robustness of the results obtained by expert evaluations was verified.

The expert in the FUCOM analysis performed two types of evaluation. First, the criteria weights were ranked from most to least important. Our study had nine criteria, so $9!=\mathrm{362,880}$ different possible rankings. Second, importance degrees were assigned to the criteria (number of criteria = k), which were ordered from the most important to the least important. Generally, integers between 1 and 9 were used in these assignments (n), which were made with the help of pairwise comparisons. It was assumed that the experts made comparisons with integers; if decimal numbers could be used, there would be many more possibilities. In such a case, the total number of evaluations was calculated as $\left(\genfrac{}{}{0pt}{}{n+k-1}{n}\right)=\left(\genfrac{}{}{0pt}{}{9+9-1}{9}\right)=\mathrm{24,310}$; however, since the first criterion always had 1 degree of importance (${\varphi }_{1}=1$) and a total number of evaluations $\left(\genfrac{}{}{0pt}{}{9+8-1}{8}\right)$ = 12,870, different evaluations are possible.

In the case of trying each possibility one by one, there were 362,880 × 12,870 = 4,670,265,600 different weight sets; however, it is impossible to evaluate such a high computation volume with today’s computing technology in a reasonable time. Considering the computational capacity of the hardware used in the analysis, 100 randomly selected weight permutations and 1,000 randomly selected importance levels for each weight ranking were tried, resulting in a total of 100,000 different weight sets.

Table 9 presents descriptive statistics of the weights; the first row includes the average of the expert evaluations. The minimum weight of a single criterion was around 0.01, and the maximum was around 0.5. The mean was around 0.1, and the median was about 0.06. Skewness and kurtosis values indicate a non-normal distribution shape.

Table 9 Descriptive statistics of the weights

Figure 2 presents the histogram distribution of the weights across the criteria, with the average of the expert evaluations indicated by a red mark. Considering the distributions in the figure, the evaluations made by the experts cannot be considered as an extreme value with a very low probability of occurrence. The distributions have a right-skewed form, indicating that the criteria cannot have high weights. This result may potentially occur because the study was conducted with nine criteria.

Fig. 2

Weight distributions of the criteria

The analysis was re-run for each weight set in the simulation data (100,000 weight sets). Nine MCDM technique evaluations were performed for each weight set, and a single ranking was obtained with the Copeland technique. Figure 3 presents the Copeland ranking histograms for each alternative. The figures also include the expert evaluation rankings with a red mark.

Fig. 3

Weight simulation results

Table 10 presents the descriptive statistics of the weight simulation. The first column indicates the alternatives, the second column includes the ranking result of expert evaluation, and the remaining columns indicate the descriptive statistics of the weight simulation process.

Table 10 Descriptive statistics of Copeland rankings in weight simulation

The number of observations in the interquartile was used to measure the performance. If the interval weight was more significant than 25% and less than 75% in the weight simulation, that observation was called an observation in the interquartile. Accordingly, 19 of 22 alternatives were in the interquartile; 86.36% (= 19/22) of expert evaluations and weight simulation results were compatible.

Table 11 presents the values of how many times each alternative was ranked in each position. Rows indicate the alternatives, and columns indicate the ranks (positions). For example, alternative 1 is never ranked in the first or second rank during the simulation and is ranked third in only 13 simulations (0.00013 = 13/100,000). Similarly, alternative 16 is ranked first in 58,967 simulations (0.58967 = 58,967/100,000). Different alternatives can take the first place in different weight sets; however, the 16th alternative was chosen as the best alternative in the largest number of weight sets during the weight simulation process. This finding indicates the superiority of the 16th alternative over the other alternatives. Similarly, the 21st alternative ranked in the last position in most weight sets (0.77987 = 77,987/100,000), indicating the alternative’s inferiority.

Table 11 Number of times each alternative is ranked in each position

The simulation numbers show that each alternative is in a higher position than the alternatives presented in Table 12. The values in the table are in the form of pairwise comparisons. For example, alternative 1 took a higher position than alternative 2 in 13,742 weight sets. However, in 86,258 (= 100,000–13,742) weight sets, the A2 alternative was in a better position than the A1 alternative. The high values in row A16 are another indicator of the superiority of this alternative over other alternatives; similarly, the low values in row A21 indicate this alternative’s inferiority.

Table 12 Number of times each alternative is ranked higher than others

A one-way variance analysis was performed to analyze the results statistically. The dataset utilized in ANOVA analysis is a 100,000 × 22 matrix where each row indicates the rank of the alternative for each simulated weight set, and each column indicates the alternatives. Figure 4 presents the box plot of the dataset.

Fig. 4

Box plot of the rank dataset

The figure shows that the rank value of the 16th alternative is always in the upper ranks (ranking 1), and the 21st alternative is also in the lower ranks (ranking 22); Table 13 presents the ANOVA test results.

Table 13 ANOVA results

A low p-value in the table indicates that the null hypothesis, which states no difference among group means, is rejected. In other words, the differences among the alternatives’ ranks differ statistically, and there are 231 $=\left(\genfrac{}{}{0pt}{}{22}{2}\right)$ multiple comparisons. Instead of listing all the results, Fig. 5 only indicates the mean of the ranks to save space. Multiple compared tests were performed, and all comparisons were significant at the 1% level. This difference is also significant if one alternative is ranked higher than the others.

Fig. 5

Multiple comparison summaries

Repeating analysis with different periods

The analysis was repeated separately for the data set in 2020 and 2019. Table 14 presents the decision matrix for 2020, MCDM scores and rankings are presented in Table 15, and Fig. 6 presents rankings for weight simulation. Descriptive statistics of Copeland rankings are presented in Table 16 in detail, and Table 17 presents the number of times each alternative is ranked in each position. Table 18 indicates the number of times each alternative is ranked higher than another alternative for 2020. Box plots of the Copeland rankings are presented in Fig. 7, ANOVA results are presented in Table 19, and finally, post hoc results are presented in Fig. 8. The ranking result for the year 2020 with the weights determined by the experts is $A22\succ A8\succ A16\succ A6\succ A4\succ A12\succ A9\succ A14\succ A19\succ A3\succ A2\succ A13\succ A7\succ A5\succ A1\succ A18\succ A21\succ A10\succ A20\succ A17\succ A11\succ A15$. As a result of the weight simulation, the values of 19 of the 22 companies are in the interquartile range (0.86 = 19/22).

Table 20 presents the decision matrix for 2019, while the MCDM scores and rankings are presented in Table 21. Rankings for weight simulation are presented in Fig. 9, and Table 22 shows the descriptive statistics of Copeland rankings. Table 23 presents how often each alternative is ranked in each position, while Table 24 indicates how often each alternative is ranked higher than another alternative for 2020. Box plots of the Copeland rankings are presented in Fig. 10, ANOVA results in Table 25, and post hoc results in Fig. 11. The ranking result for 2019 with the weights determined by the experts is $A8\succ A15\succ A16\succ A19\succ A12\succ A22\succ A4\succ A9\succ A6\succ A18\succ A2\succ A13\succ A14\succ A1\succ A3\succ A10\succ A7\succ A17\succ A5\succ A21\succ A20\succ A11$. As a result of the weight simulation, the values of 20 of the 22 companies are in the interquartile range (0.91 = 20/22).

It is understood that the rankings for 2021, 2020, and 2019 differ. A company’s financial statements and stock market performances do not remain the same every year, and this difference causes the rankings to change.

Discussion

This research determined the long-term performance of 22 companies included in the sustainability index in Turkey using MCDM methods. One of the critical research results is related to the weights of financial performance indicators. This study determined criterion weights using the FUCOM method, revealing that the current ratio was the criterion that affected financial performance the most. This finding differs from the findings of similar studies in the literature. For example, Abdel-Basset et al. (2020) found that the financial ratios that impact manufacturing firms’ financial performance were the quick and debt-to-equity ratios, respectively. Furthermore, among 20 performance indicators, asset turnover ranked 13th and debt ratio third regarding criterion weights. In their research on SMEs, Roy and Shaw (2021) calculated the criteria weights for each firm separately. They determined that the return on total capital-employed ratio was the criterion with the highest weight for five of the six firms. Shen et al. (2017) found that the criterion with the highest weight was the research and development expense ratio while examining the effect of research and development on financial performance. Visalakshmi et al. (2015) examined the financial performance of GREENEX companies. They determined that the criterion with the highest effect on performance was the current ratio and quick ratio, and the criterion with the most negligible effect was ROA. Ghadikolaei et al. (2014) found that the criterion with the highest weight was cash value, and the criterion with the lowest weight was ROE. Their study also determined the financial performance of companies operating in Iran. Similarly, Erdoğan et al. (2016) found that the ratio with the highest impact on the financial performance of the food companies in BIST was the leverage ratio. The different results from these studies reveal that the most important criteria affecting financial performance and those with the most negligible impact differ regarding the period, sector, company, and financial ratios examined together.

This study’s second most important finding is that the companies with the highest financial performance differ by MCDM methods. That is, the companies with the highest performance for all four methods regarding the examined periods are not the same in any period. One of the most important reasons for this situation is that each method’s methodological flow and calculation methods in ranking the companies differ; therefore, it would be incorrect to make inferences about which of the four methods should be used to make a more accurate performance ranking based on the results of the current study. In this context, the study results are similar to those of the extant literature. For example, Ecer (2021) determined that the best alternative is the same for all methods (SECA, ARAS, COPRAS, MAIRCA, and MARCOS) except for the CoCoSo method; however, these results differ from the findings of similar studies in the literature. For example, Baležentis et al. (2012) found that the best alternative selected by fuzzy VIKOR, Fuzzy TOPSIS, and Fuzzy ARAS methods are the same. Similarly, Ghadikolaei et al. (2014) found that the two best alternatives in performance ranking were the same for all three methods (fuzzy VIKOR, ARAS-F, and fuzzy COPRAS).

Another result of the current research is the integration of the rankings suggested by the MCDM methods with the Copeland method. In addition, a weight simulation was performed, where expert evaluations were simulated with random evaluations. The results indicated that the Copeland method ensured a consensus among different methods. This result complied with the results of similar studies in the literature. For example, Ecer (2021) consolidated the results of the six MCDM methods (SECA, MARCOS, MAIRCA, COCOSO, ARAS, and COPRAS) with Copeland and Borda methods and tested the robustness of the ranking results by performing sensitivity analysis. In conclusion, it was revealed that the best alternative of the six MCDM methods and the alternatives of Copeland and Borda methods were the same, confirmed by sensitivity analysis. Beheshtinia and Omidi (2017) integrated the different sequencing results obtained by the four MCDM methods (MDL-FVIKOR, MDL-FTOPSIS, AHP-FVIKOR, and AHP-FTOPSIS) with the Copeland method and created a final ranking. Furthermore, they found that the best alternative was the same in all methods, including the Copeland method; only the other alternatives differed. Kiani et al. (2022) ranked the alternatives using three MCDM methods (fuzzy SAW, fuzzy TOPSIS, and fuzzy VIKOR) and used the Borda and Copeland methods to obtain final results since each method produced different ranking results. As a result, the Borda and Copeland methods gave the same results for all alternatives.

Managerial implications

The most crucial managerial result of this research concerns the findings related to the determinants of financial performance and the weights of these determinants. In the research, 22 companies were ranked by their financial performance regarding the 9 performance indicators; the criteria were weighted FUCOM methods. Furthermore, validation of the results was performed with a weight simulation; therefore, the results were sufficiently robust and reliable that managerial inferences could be made. The current ratio has been the most critical determinant of performance in the current research. The subsequent vital ratios were EBITDA and net profit margin (NPM); as the NPM increases, the efficiency of the business also increases. A constantly rising NPM indicates that a company can generate more profit with less equity over time; therefore, managers must develop strategies to increase net income. Covering expenses and making a net profit starts with the correct pricing. To improve profitability, managers must define a target gross profit margin to cover operating expenses, make competitive pricing, and monitor the gross profit margin monthly. With a solid profit analysis (financial modeling), it is possible to determine when the prices will increase, thus increasing the gross profit margin and making a profit. Break-even analysis is vital for pricing and profitability; managers must comprehensively evaluate what needs to change for strategic goals and pricing to be more accurate, looking at the break-even point. First, it is crucial to determine the fixed and variable costs correctly. In addition, one of the first things to be done to increase business profitability is to reduce fixed expenses. The business should try to convert many of its expenses into variable expenses. Costs can be reduced by performing a break-even analysis. For example, it may be beneficial to reduce financing costs or to reduce costs and offer the same quality at a more affordable price. Conversely, the hours of working machines can be reduced by decreasing the labor hours.

Conclusions

Measuring financial performance is one of the oldest known methods of comparing companies competing in the same industry. This research ranked companies included in the sustainability index in Turkey in 2019, 2020, and 2021 according to their financial performance. Twenty-two firms were ranked according to nine criteria, including eight financial ratios and one stock market indicator. The model proposed in the research included determining the criteria weights with FUCOM and performance ranking with CoCoSo, GRA, MABAC, MAIRCA, MOOSRA, OCRA, TOPSIS, TODIM, and VIKOR. According to the FUCOM method results, the current ratio is the criterion with the highest weight; in other words, it was the most influential on the financial performance ranking. Each MCDM method gave a different ranking, so these results were consolidated using the Copeland method and Borda rule. According to the results, the A16 (TOASO) alternative is the best. Afterward, a weight simulation was performed to test the robustness of Copeland’s results. Expert evaluations were simulated with random evaluations, and 100,000 weight sets were created; the analysis was re-run for each weight set. The results indicate that the ranks of the expert evaluations and the mean of the weight evaluations are similar, indicating the robustness of the results.

Limitations of the research

The research examined companies (excluding banks) traded in the BIST sustainability index included in the sustainability index in 2019, 2020, and 2021; data were limited to these years. The decision matrix includes eight financial ratios and one stock market indicator. FUCOM determined criterion weights, and nine MCDM methods were used for performance ranking. Another limitation of this study was using a subjective weight determination technique. Objective weight determination techniques, such as MAIRCA, SECA, and SAW, can be used in other studies. In this study, parameters of the MCDM techniques (such as attenuation factor in TODIM or β coefficient in CoCoSo) were fixed as the default values proposed by the developers of the techniques. Optimizing these parameters is a valuable future research direction.

In the weight simulation process, random assessments were used. Random evaluations allow for a more objective evaluation of the results; however, it leads to results spread over an extensive range.

Future work directions

Future studies can include banks in the alternatives, and indexes containing only banks can also be used. Researchers can also analyze the performance of firms over even longer terms (like the last ten years or the last 20 years) and implement more financial ratios (especially cost-oriented) and stock market indicators (such as the risk of stock return). Future studies can also determine the criterion weights with a different method (entropy, equal weighting, and BWM) and use different MCDM methods (such as VIKOR, SAW, DEMATEL, and fuzzy AHP).

Expert opinions can guide weight simulations, and random numbers can be generated based on expert evaluations; in this case, a narrower distribution of the results can be achieved.

The order of alternatives also changed in different periods. Future studies can create a new decision matrix by averaging the decision matrices of different periods; the ranking can then be made over the decision matrix containing these average values.

Appendices

Appendix A: Results for the year 2020

See Tables 14, 15, 16, 17, 18, 19 and Figs. 6, 7, 8.

Table 14 Decision matrix (2020)

Table 15 Overall scores and ranking results of alternatives

Fig. 6

Weight simulation results

Table 16 Descriptive statistics of Copeland Rankings in Weight Simulation

Table 17 How many times each alternative is ranked in each position

Table 18 How many times each alternative is ranked higher than other alternative

Fig. 7

Box plot of the rank dataset

Table 19 ANOVA results

Fig. 8

Multiple comparison summary

Appendix B: Results for the year 2019

See Tables 20, 21, 22, 23, 25 and Figs. 9, 10, 11.

Table 20 Decision matrix (2019)

Table 21 Overall scores and ranking results of alternatives

Fig. 9

Weight simulation results

Table 22 Descriptive statistics of Copeland Rankings in Weight Simulation

Table 23 How many times each alternative is ranked in each position

Table 24 How many times each alternative is ranked higher than other alternative

Fig. 10

Box plot of the rank dataset

Table 25 ANOVA results

Fig. 11

Multiple comparison summary