1 Introduction

The fundamental work by John Maynard Smith and George Price [1] revealed the central role that Evolutionary Game Theory can play in the description of interactions and conflicts in nature. Recently, such studies have been done in modeling cancer to improve its treatment [2,3,4]. So far, both Classic and Evolutionary Game Theory are considered powerful tools for modeling and analyzing problems in economy, governments’ policies [5, 6], supplies’ chain [7,8,9] etc. Moreover, also several interactions among plants, including competition for resources, can be viewed as an evolutionary game [10]. For example, the interactions among plants with different level of allelochemical production as well as the interactions among mutualists and non-mutualists plants have been studied recently through an Evolutionary Game Theory point of view in [11, 12], respectively.

In the present work, we focus on a universal phenomenon of plant species, so-called autotoxicity or, more accurately, Plant-Soil Negative Feedback (PSNF) [13]. PSNF is defined as the development of negative conditions for plant vegetative and reproductive performances of individual belonging to the same species induced into the soil by the plants themselves. The mechanisms underlying PSNF are different as the build-up of soil-borne pathogen inoculum [14], the shift in composition of soil microbiome, and the release of autotoxic compounds during litter decomposition [15] including degraded self DNA [13]. Plant life forms can face or escape PSNF by different strategies. For example plants unable of vegetative propagation can avoid the negative impact of the “home” soil [16] by long-range seed dispersion, producing the well known Janzen-Connell distribution [17, 18]. Alternatively, clonal plants like grasses and sedge could escape the localized PSNF raised in the “home” soil by spreading in a radial fashion away through vegetative growth [19].

So far, mathematical tools as Ordinary Differential Equations systems [20] and Agent-Based Modelling [21] have been used to describe the above procedure, as well as the interaction between competitive plants (in terms of fitness) with different autotoxicity sensitivities. The competition occurs due to fitness’ maximum capacity. The capacity may derive, for example, from resources’ limitations. In the present work, we describe this competition as an Evolutionary Game. Our main goal is to indicate how each distribution of species in the population affects the total fitness derived by the game and whether this total fitness is being optimized. Particularly, we develop a \(2\times 2\) matrix game to model the interaction among plants with different levels of sensitivity to autotoxicity. We also take into account the fact that the autotoxicity compounds affect not only the plant they come from but the other plant as well, either positively or negatively [22]. The game-theoretical framework allows us to express and analyze directly this interaction. Now, through the \(2\times 2\) matrix, we describe the result for any plant as a function of the other’s plant relative abundance. This approach reveals the frequency-dependence nature of this phenomenon [23], whereas the current literature focuses only on the density-dependence one. As population dynamics induced by the game we use the replicator equations [24] and we make a stability classification of the Nash Equilibrium Strategies (NES) in relation to the equilibrium points of the Replicator Dynamics. We then indicate under which conditions an heterogeneous population, consists of plants with different sensitivity to autotoxicity, shares the available space and under which a unique species prevails over the others and survives alone. Using this analysis, we explicitly express the total fitness that derives by any possible population state as well as the one that derives by the stable equilibrium state. The form of the total fitness function varies based on the specific conditions of the competition, meaning that it differs for each distinct relationship between the elements of the \(2\times 2\) matrix. In biological terms, any such relation may refers to a different plants’ environment. On most occasions, a critical difference between the maximum value of the total fitness function and its value at the equilibrium state occurs. In particular, we find out that under conditions that favor the dominance of highly sensitive to autotoxicity plants, the corresponding maximum total fitness cannot be reached. On the contrary, coexistence of the two species might achieves to reach it’s own maximum as well as the dominance of low sensitive plants, always under some extra conditions.

2 Model description

We introduce a \(2\times 2\) matrix game of interactions between representants of a population consisting of species with low (L) and high (H) autotoxicity. The payoffs of the game correspond to the proportion of fitness (B) the plants get in each competition case. The species pay a cost for preventing autotoxicity according to how sensitive they are. Particularly, when an L plant faces another L, each plant gets a proportion \(\theta\) of the fitness B and pays as a cost a proportion \(s_l\) which defines the sensitivity to autotoxicity of species L. Accordingly, when both plants have high sensitivity, each plant gets a proportion \(\lambda\) and pays a cost \(s_h\) as the sensitivity of species H. Finally, in a mixed competition of H and L plants, the H plant gets a proportion \(\phi\) and pays the cost \(s_h\) and the L plant gets a proportion \(\alpha\) and pays the cost \(s_l\).

In monospecific competitions, we likely observe a fitness loss besides the autotoxicity effect, that is \(0\le \theta , \lambda \; \le 0.5\). This loss is less likely observed in mixed competitions since the presence of different species guarantees that (again, besides the autotoxicity effect) one species occupy all the resources the other left [25]. Therefore, we will study this game under the following assumption:

Assumption

In a mixed population of plants with low (L) and high (H) sensitivity to autotoxicity and in the absence of the cost for preventing autotoxicity, the two species share the total fitness. That is, \(\alpha +\phi = 1 \Leftrightarrow \alpha =1-\phi\), \(\alpha , \phi \in [0,1]\).

We define \(\Delta s:=s_h-s_l\) as the difference between the two autotoxicities’ sensitivity. Since \(s_h,s_l \in [0,1]\) and \(s_h \ne s_l\) (otherwise, there is no distinction between L and H phenotypes), we get \(\Delta s\in (0,1]\). For sake of simplicity we assume that \(B=1\).

By the abovementioned, one can derive the \(2\times 2\) payoff matrix \({\textbf{A}}\) which describes the payoffs for the species in every interaction among them. Each element of matrix \({\textbf{A}}\) consists of a pair where the first and the second coordinates refer to the payoffs of the row-player and the column-player, respectively:

(1)

3 Symmetric \(2\times 2\) games and replicator dynamics

We first give some propositions and known theoretical results that will be used throughout the paper.

3.1 Nash equilibria on symmetric \(2\times 2\) games and total payoff

A \(2\times 2\) matrix game \({\textbf{A}}=\Big ((a_{ij},b_{ij})\Big )_{i,j=1,2}\) is called symmetric if the players have identical sets of strategies and \(a_{ij}=b_{ji}\) for all \(i,j \in \{1,2\}\). The game described by (1) is symmetric. Such games can be fully characterized by the first coordinates (i.e. \(a_{ij}\)) of the payoff matrix \({\textbf{A}}\).

Proposition 1

[26] Symmetric \(2\times 2\) games have at least one symmetric Nash Equilibrium (NE), i.e., there is some \(x \in [0,1]\) such that the point \(\big ((x,1-x),(x,1-x)\big )\) is a NE.

Proposition 2

[27] On symmetric \(2\times 2\) games, we have the following three cases for the NE according to the values of \(a_{ij}\):

  1. 1.

    (\(a_{11}>a_{21}\) and \(a_{12}>a_{22}\)) or (\(a_{11}<a_{21}\) and \(a_{12}<a_{22}\)): We have just one pureFootnote 1 equilibrium since there is a dominant strategy. The point \(\big ((1,0),(1,0)\big )\) is the NE if the first relation holds, the point \(\big ((0,1),(0,1)\big )\) otherwise. We will call this game Domination Game.

  2. 2.

    \(a_{11}>a_{21}\) and \(a_{12}<a_{22}\): We have three NE. The points \(\big ((1,0),(1,0)\big )\) and \(\big ((0,1),(0,1)\big )\) are the pure ones and the point \(\big ((x^*,1-x^*),(x^*,1-x^*)\big )\) is the mixed one. We will call this game Priority Effect.

  3. 3.

    \(a_{11}<a_{21}\) and \(a_{12}>a_{22}\): We have three NE. The points \(\big ((1,0),(0,1)\big )\) and \(\big ((0,1),(1,0)\big )\) are the pure ones and the point \(\big ((x^*,1-x^*),(x^*,1-x^*)\big )\) is the mixed one. We will call this game Hawk-DoveFootnote 2.

where \(x^*=\frac{a_{22}-a_{12}}{a_{11}-a_{12} -a_{21}+a_{22}}, \;\; x^* \in (0,1)\).

Proposition 3

[26] On a \(2\times 2\) matrix game \({\textbf{A}}=\Big ((a_{ij},b_{ij})\Big )_{i,j=1,2}\), the expected payoff for the row-player under the pair of strategies \(\big (X,Y\big )=\big ((x,1-x),(y,1-y)\big )\), where \(x,y \in [0,1]\), is given by the function

$$\begin{aligned} v^r(X,Y)=X\, (a_{ij})_{i,j=1,2}\, Y^T=x\, \big (y\, a_{11} + (1-y)\, a_{12}\big ) + (1-x)\, \big (y\, a_{21}+(1-y)\, a_{22}\big ). \end{aligned}$$

Accordingly, the expected payoff for the column-player is

\(v^c(X,Y)=X\, (b_{ij})_{i,j=1,2}\, Y^T\).

We denote the expected total payoff of a \(2\times 2\) matrix game \({\textbf{A}}\) under the pair of strategies \(\big (X,Y\big )\) as \(v(X,Y):=v^r(X,Y)+v^c(X,Y)\).

Proposition 4

[26] On a symmetric \(2\times 2\) game \({\textbf{A}}=(a_{ij})_{i,j=1,2}\), the payoff functions of the two players are symmetric, i.e. \(v^r(X,Y)=v^c(Y,X)\).

Remark 1

On a symmetric \(2\times 2\) game \({\textbf{A}}=(a_{ij})_{i,j=1,2}\) and under a symmetric pair of strategies \(\big (X,X\big )\), Proposition 4 gives that \(v^r(X,X)=v^r(x)=v^c(x)\). According to Proposition 3, the expected total payoff is given by the function:

$$\begin{aligned} v(x)=2\,\big ( x^2\, (a_{11}-a_{12}-a_{21}+a_{22}) + x\, (a_{12}+a_{21}-2\, a_{22}) + a_{22}\big ) \end{aligned}$$
(2)

If the value

$$\begin{aligned} x_0=\frac{2\, a_{22}-a_{12}-a_{21}}{2\, (a_{11}-a_{12}-a_{21}+a_{22})} \end{aligned}$$
(3)

belongs to the domain of function (2) (that is, \(x_0 \in [0,1]\)), then (2) gets its maximum or minimum value on \(x_0\) and this value is

$$\begin{aligned} v(x_0)=- \frac{(a_{12}+a_{21}-2\, a_{22})^2}{2\, (a_{11}-a_{12}-a_{21}+a_{22})} +a_{22} \end{aligned}$$
(4)

Remark 2

After standard calculations, we observe that if \(a_{11}-a_{12}-a_{21}+a_{22}<0\), then \(v(x_0)\) is the maximum value of (2) and if \(a_{11}-a_{12}-a_{21}+a_{22}>0\), \(v(x_0)\) is the minimum.

3.2 Replicator dynamics and nash equilibrium strategies of the game

So far, the matrix \({\textbf{A}}\) describes the specific matching of the plants and the relative fitness. We now move to its consequences in terms of population dynamics using Replicator Dynamics [24] to describe the dynamical process for the densities of the strategies in the game (1). The state of the population can be denoted by the probability matrix \(p=(x_L\;\;\;x_H)^T\), \(x_L+x_H=1\). Therefore, \(x_L\) and \(x_H\) are the proportions of L and H strategy in the population, respectively. The rate of increase \(\frac{dx_L}{dt}/x_L\) of L strategy is a measure of its evolutionary success. The Replicator equation expresses this success as the difference between the fitness \(v^r\Big (e_1^T,p^T\Big )=e_1^T A\, p\) of \(x_L\) and the average fitness \(v^r\Big (p^T,p^T\Big )=p^T A\, p\) of the population, where \(e_1^T=(1 \;\;0)\). Accordingly, the rate of increase \(dx_H/x_H\) of H strategy is given by the difference \(v^r\Big (e_2^T,p^T\Big ) - v^r\Big (p^T,p^T\Big )=e_2^T A\, p - p^T A\, p\), where \(e_2^T=(0 \;\;1)\). Thus, we obtain:

$$\begin{aligned} {\left\{ \begin{array}{ll} \frac{dx_L}{dt}=x_L\cdot (e_1^T A p - p^T A p)\\ \\ \frac{dx_H}{dt}=x_H\cdot (e_2^T A p - p^T A p) \end{array}\right. } \end{aligned}$$
(5)

Regarding the equilibrium points of (5), we firstly observe that the origin (0,0) is not an admissible equilibrium point since \(x_L+x_H=1\). According to the stability analysis of (5) for symmetric games as it is provided in [29], the equilibria of (5) can either be stable or saddles. Moreover, we have that the only possible equilibria are the three NESs (1, 0), (0, 1) and \((x^*,1-x^*)\). If the matrix \({\textbf{A}}\) defines a Domination game, then the mixed equilibrium \((x^*,1-x^*)\) is not admissible (\(x^*<0\) or \(x^*>1\)) and the unique equilibrium point of the system is either (1, 0) or (0, 1) and it is stable (therefore, globally stable). Moreover, the pure NES defines the Evolutionary Stable StrategyFootnote 3 of the evolutionary game. In the case of the Priority Effect game, the pure NESs are the locally stable equilibria.Footnote 4and the mixed one is the unstable. In the case of the Hawk-Dove game, the mixed NES is the unique (therefore, globally) stable equilibrium point of the system. Summarizing, we get the table:

Table 1 Relation between the NESs of the games defined in Proposition 2 and the stable equilibrium points of the system (5)

4 Results

4.1 Stability conditions for the nash equilibrium strategies

Implementing Proposition 2 and according to Table 1, we have the following results regarding the NESs and the stable states of system (5) for the game (1):

  1. 1.

    The case \(\phi -\theta<\Delta s<\lambda -(1-\phi )\), i.e. the case of Priority Effect game, is never possible since \(\lambda ,\theta \, \le 0.5\), therefore \(\lambda -(1-\phi )\le \phi -\theta\).

  2. 2.

    If \(\Delta s>\phi -\theta\): We have the Domination game and L is the stable NES.

  3. 3.

    If \(\Delta s<\lambda -(1-\phi )\): We have the Domination game game and H is the stable NES.

  4. 4.

    If \(\lambda -(1-\phi )<\Delta s<\phi -\theta\): We have the Hawk-Dove game and the coexistence mixed strategy \((x^*,1-x^*)\) is the stable NES, where

    $$\begin{aligned} x^*=\frac{\Delta s\, -\phi \, + 1-\lambda }{1-\theta -\lambda } \end{aligned}$$
    (6)

    Recall that \(x^*\) defines the share of L strategy in the population and, consequently, \(1-x^*\) defines the share of H. Regarding how changes in the parameters change the mixture of strategies at this equilibrium point, we can see the followings, taking the partial derivatives of (6) with respect to each parameter:

    • \(\frac{\partial x^*}{\partial \Delta s}=\frac{1}{1-\theta -\lambda }>0\)

    • \(\frac{\partial x^*}{\partial \phi }=-\frac{1}{1-\theta -\lambda }<0\)

    • \(\frac{\partial x^*}{\partial \theta }=\frac{\Delta s\, -\phi \, + 1-\lambda }{(1-\theta -\lambda )^2}>0\)

    • \(\frac{\partial x^*}{\partial \lambda }=\frac{\Delta s\, -(\phi \, - \theta )}{(1-\theta -\lambda )^2}<0\)

The possible stable NESs for \(\theta , \lambda \in \{0.1,0.3,0.5\}\) in positive \(\phi\) and \(\Delta s\) phase space are presented in Fig. 1:

Fig. 1
figure 1

Lines in \(\phi\) and \(\Delta s\) phase space depending on the value of \(\lambda\) (rows) and \(\theta\) (columns). These lines define the different areas of the phase space regarding which NES corresponds to the stable state of the dynamical system (5). The solid line represents the line \(\Delta s=\phi -\theta\) and the dashed one the line \(\Delta s=\lambda -(1-\phi )\). White space is the area where L strategy is the stable NES. Dark grey is where H is the stable NES. The space in between (light grey) is where the mixed NES (i.e. coexistence of both species) is the stable state of 5

Remark 3

The space where H strategy dominates the competition (dark grey) defines an isosceles right triangle with perpendicular sides equal to \(\lambda\). The space where we observe coexistence of L and H plants (light grey) defines an isosceles trapezium with parallel sides defined by the lines \(\Delta s=\phi -\theta\) and \(\Delta s=\lambda -(1-\phi )\) with length equal to \((1-\theta )\sqrt{2}\) and \(\lambda \sqrt{2}\), respectively. The non parallel sides are of length \(1-\theta -\lambda\).

Therefore, we can explicitly calculate, in terms of \(\theta\) and \(\lambda\), the area for each different space:

Corollary 1

The areas of the three different spaces are:

  • \(E_{\textrm{coex}}=\frac{1}{2}(\theta -1)^2 - \frac{1}{2}\lambda ^2\);

  • \(E_{\textrm{high}}=\lambda ^2/2\);

  • \(E_{\textrm{low}}=1 - (E_{\textrm{coex}}+E_{\textrm{high}})=1-\frac{1}{2}(\theta -1)^2\).

Since \(\theta , \lambda \in [0,0.5]\), \(E_{\textrm{coex}}\) is a decreasing function on both \(\theta\) and \(\lambda\), \(E_{\textrm{high}}\) is an increasing function on \(\lambda\) and \(E_{\textrm{low}}\) is an increasing one on \(\theta\). We also observe that \(E_{\textrm{low}}\ge 0.5\).

4.2 Total fitness maximization

Now, we will investigate under which conditions the actual total fitness that derives by a stable equilibrium strategy coincides with the maximum total fitness the two species can get. This analysis relates to the well known notion, in algorithmic game theory, of price of anarchy (PoA) [4] which measures the efficiency of a game by comparing the social welfare of an equilibrium state with the best social welfare, by taking the ratio of these two values. In this work, we take the difference between the maximum and the actual total fitness instead of using a ratio, as it offers a more intuitive understanding of the results. We consider the total payoffs only on symmetric pairs of strategies \((X,X)=\Big ((x,1-x),(x,1-x)\Big )\) since there is no ecological interpretation for the non symmetric pairs. Therefore, implementing Remark 1, the total fitness of the game (1) is given by the function \(v(x):\, [0,1]\longrightarrow [0,1]\):

$$\begin{aligned} v(x)=2\, \big (x^2\, (\theta +\lambda -1) +x\, (1-2\, \lambda +\Delta s) +\lambda -s_h) \big ) \end{aligned}$$
(7)

Since \(\theta +\lambda -1<0\), from Remark 2 the function \(v(x)\) is concave. Morevover, since \(\lambda \le 0.5\) and \(\Delta s > 0\), we have that \(\Delta s > 2\lambda -1\) which is equivalent to \(x_0>0\) as one can verify in the Appendix A. The latter condition, along with the continuity of function (7), proves that the dominance of the H strategy cannot produce the maximum possible total fitness. For the maximum value of function (7) (denoted by \(V_{\textrm{max}}\)) we have the followings (the analytical passages can be found in the Appendices A,B):

  • If \(\Delta s<1-2\, \theta\): \(v(x)\) gets its maximum value on \(x_0\in (0,1)\) where

    $$\begin{aligned} x_0=\frac{1+\Delta s-2\, \lambda }{2\, (1-\theta -\lambda )} \end{aligned}$$
    (8)

    and

    $$\begin{aligned} V_{\textrm{max}}=v(x_0)=\frac{(1+\Delta s-2\, \lambda )^2}{2\, (1-\theta -\lambda )} + 2\, (\lambda -s_h) \end{aligned}$$
    (9)

    Therefore, \(v(x)\) is an increasing function for \(x\in [0,x_0]\) and a decreasing one for \(x\in (x_0,1]\). Regardless of the game form, the maximum total fitness is given on the mixed state \(\big ((x_0,1-x_0)\big )\) and it is equal to \(v(x_0)\). This state is a stable equilibrium iff the game has the Hawk-Dove form and additionally \(x_0=x^*\). That is, when \(\lambda -(1-\phi )<\Delta s<\phi -\theta\) and \(\Delta s=2\, \phi -1\).

    • If the game is Domination with L strategy as the dominant, then the actual total fitness is:

      $$\begin{aligned} V_{\textrm{low}}=v(1)=2\, (\theta -s_l)<v(x_0) \end{aligned}$$

      and the fitness loss at the equilibrium is given by the equation

      $$\begin{aligned} V_{\textrm{max}} - V_{\textrm{low}} = \frac{(1+\Delta s-2\, \lambda )^2}{2\, (1-\theta -\lambda )} + 2\, (\lambda - \theta - \Delta s) \end{aligned}$$
      (10)
    • If the game is Domination with H strategy as the dominant, then the actual total fitness is:

      $$\begin{aligned} V_{\textrm{high}}=v(0)=2\, (\lambda -s_h)<v(x_0) \end{aligned}$$

      and the fitness loss at the equilibrium is given by the equation

      $$\begin{aligned} V_{\textrm{max}} - V_{\textrm{high}} = \frac{(1+\Delta s-2\, \lambda )^2}{2\, (1-\theta -\lambda )} \end{aligned}$$
      (11)
    • If the game is Hawk-Dove and \(\Delta s\ne 2\, \phi -1\) then the actual total fitness is:

      $$\begin{aligned} V_{\textrm{coex}}=v(x^*)<v(x_0) \end{aligned}$$

      and the fitness loss at the equilibrium is given by the equation

      $$\begin{aligned} V_{\textrm{max}} - V_{\textrm{coex}} = \frac{(1+\Delta s-2\, \lambda )^2 - 4 \, (\Delta s - \phi +1 - \lambda )(\phi -\lambda )}{2\, (1-\theta -\lambda )} \end{aligned}$$
      (12)
  • If \(\Delta s\ge 1-2\, \theta\): The function \(v(x)\) is increasing in [0,1]. Hence, its maximum value is:

    $$\begin{aligned} V_{\textrm{max}}=v(1)=2\, (\theta -s_l) \end{aligned}$$

    Therefore, the maximum total fitness is given on the state (1,0). If the game is Domination with L strategy as the dominant, the maximum total fitness coincides with the actual one (\(V_{\textrm{low}}=v(1)\)). In any other case, the actual total fitness is less than the maximum one and the fitness loss is given by the equations:

    • for the Domination game with H strategy as the dominant:

      $$\begin{aligned} V_{\textrm{low}} - V_{\textrm{high}} = 2\, (\theta -\lambda +\Delta s) \end{aligned}$$
      (13)
    • for the Hawk-Dove game:

      $$\begin{aligned} V_{\textrm{low}} - V_{\textrm{coex}} = 2\, (\theta - \lambda + \Delta s) - 2\, \frac{(1+\Delta s - \phi - \lambda )(\phi -\lambda )}{1-\theta -\lambda } \end{aligned}$$
      (14)

The above results can be illustrated in Table 2 where we present a generic form of the concave function (7). The numerical values of \(V_{\textrm{max}}, V_{\textrm{low}}, V_{\textrm{high}}, V_{\textrm{coex}}, x_0\) and \(x^*\) vary based on the assigned values to the parameters:

Table 2 Fucntion \(v(x)\) of total fitness as a function of the share of L strategy in the population in each possible case. Red dots denote the position of the maximum total fitness, while green ones denote the actual total fitness that derives by the stable equilibrium state

4.3 Examples

In order to make our results more intuitive, we assign the following values to our parameters: \(\theta =0.3, \lambda =0.5, \phi =0.6, s_l=0.05\). We will take different values for \(s_h\), increasing each time the sensitivities’ difference (\(\Delta s\)). Therefore, we are in the case of panel b in Fig. 1.

Example 1

If \(s_h=0.25\), then \(\Delta s=0.2\). We are in the light grey area of panel b (Hawk-Dove game) and since \(\Delta s=2\, \phi -1\), the total fitness function can be illustrated by the graph (6) in Table 2. Hence, in that case the actual total fitness coincides with the maximum one.

Example 2

If \(s_h=0.4\), then \(\Delta s=0.35\). Increasing the sensitivities’ difference above the threshold \(\phi -\theta =0.3\), we move to the white area of panel b (L strategy dominates the competition). Since \(\Delta s < 1- 2\, \theta\), the total fitness function’s graph is as in (1) in Table 2. That is, there is an unstable state \((x_0,1-x_0)\) of a mixed population which creates higher total fitness than the one in which the system finally ends up in its stable equilibrium state (1, 0).

Example 3

If \(s_h=0.45\), then \(\Delta s=0.4\). Now, the sensitivities’ difference not only overcomes the threshold \(\phi -\theta\) which makes L strategy to be dominant, but, moreover, reaches also the threshold \(1-2\, \theta\). Therefore, the actual total fitness is indeed the maximum one as shown in graph (2) in Table 2.

In Fig. 2 below, we can see how system (5) evolves over time in each of the examples above. In the first and the 3rd plot (Examples 1 and 3) the system ends up to the distribution of the two strategies that maximizes the total fitness. On the other hand, in the second plot (Example 2) the dynamics pass from the unstable state which maximizes the total fitness to end up to a state of a reduced one.

Fig. 2
figure 2

Evolution in time of the system (5) for the values of Examples 1–3. The grey solid line corresponds to \(x_L\) (proportion of L strategy) and the black one to \(x_H\) (proportion of H strategy). The dashed line denotes the distribution that provides the maximum total fitness (\(x_0\)), as it is given by the equation (8). We have that \(x_0=0.5\), \(x_0=0.875\) and \(x_0=1\) for the Examples 1, 2 and 3, respectively. When \(x_0\) coincides with the stable state, the dashed line is green, otherwise is red. The chosen initial conditions are [0.2, 0.8]

For \(\theta =0.3\) and \(\lambda =0.5\) (as we did for the Examples 1–3), we can reproduce the panel b of Fig. 1 dividing it into regions according to which graph of Table 2 describes the total fitness function. This result is shown in Fig. 3.

Fig. 3
figure 3

Reproduction of panel b of Fig. 1. Recall that \(\theta =0.3,\, \lambda =0.5\), the solid black line represents the line \(\Delta s=\phi -\theta\), the dashed black one the line \(\Delta s=\lambda - (1-\phi )\) and the region of stable states are as follows: For \(\Delta s>\phi -\theta\), L is the stable strategy, for \(\Delta s<\lambda -(1-\phi )\) is H and for the intermediate values, the coexistence mixed strategy \((x^*,1-x^*)\) is the stable one. The different colours correspond to the seven different graphs as they are numbered in Table 2. The case of graph (6) is possible only along the green line denoting the function \(\Delta s=2\phi -1\) in the coexistence region. We also indicate the specific position on the panel for the chosen values of Examples 1, 2 and 3 with a times symbol, a filled triangle and a filled circle symbol, respectively

5 Discussion

We first need to clarify the biological meaning of the classification results presented in Sect. 4.1. We notice that the L plants start with an a priori advantage because of the less sensitivity to autotoxicity. In other words, if we consider just the sensitivities, the L plants will definitely dominate the game. This can be seen also in Fig. 1 where the area of L strategy domination (white area) is the biggest in any case of our model (i.e. \(E_{\textrm{low}}\ge 0.5\)). Hence, the H strategy can only appear when it is highly competitive with respect to the sensitivities difference. The competitiveness of H strategy can be defined as the difference between the benefits of H and L strategies in any possible competition. For a careful and accurate discussion of our results, we need to add some extra notation, defining as L-competitiveness and H-competitiveness the competitiveness of H in a competition against an L and an H plant, respectively. Hence, L-competitiveness equals to \(\phi - \theta\) since in the competition against an L plant, the benefit of H is \(\phi\) and of L is \(\theta\). Analogously, H-competitiveness equals to \(\lambda - (1-\phi )\). Having clarified these terms, we can say that the dominance of a monospecific population, or the coexistence of both species, depends on the relation between the sensitivities’ difference (i.e., \(\Delta s\)) and the competitiveness of highly sensitive plants in each possible competition.

When the sensitivities’ difference is bigger than the L-competitiveness of H strategy (i.e \(\Delta s > \phi - \theta\)), the L strategy dominates the competition (white areas in Fig. 1). In that case, the total fitness the two species gain at stable state, can be maximum only under the additional condition \(\Delta s \ge 1-2\, \theta\). This inequality can be rewritten as \(\theta - s_l \ge 1-\theta -s_h\). Thus, a monospecific population consisting of L plants achieves maximum total fitness when each L plant has a payoff greater than or equal to the difference between the remaining fitness left by each plant (\(1-\theta\)) and the sensitivity of H plants.

In contrast, when the sensitivities’ difference is less than the H-competitiveness of H strategy (i.e \(\Delta s <\lambda -(1- \phi )\)), the cost for preventing autotoxicity does not outweigh the advantages of being a highly sensitive plant. Therefore, the H strategy dominates the competition (dark grey areas in Fig. 1). Then, the actual total fitness is always less than the maximum one, namely there exists an unstable population state that creates higher total fitness than the one that is been created by the stable population state. This result derives directly by the mathematical analysis of the total fitness’ function as it is provided in Sect. 4.2.

Finally, coexistence occurs if the sensitivities’ difference lies in intermediate values, namely, no strategy is strong enough to dominate. Now, the total fitness gets its maximum value only under the additional condition \(\Delta s=2\, \phi -1\). To reveal the biological meaning of this condition, we rewrite it as \(1-\phi -s_l = \phi - s_h\). That is, the payoffs of the two species in the heterogeneous competition coincide. Hence, in a stable heterogeneous population, any small advantage for one of the two species creates a loss to the total fitness. This effect is similar to well known environmental results where different species have to share available resources equally in order to optimize their cumulative fitness and maximize biomass production, a phenomenon called over-yielding [30].

When coexistence occurs, a well-expected result can be easily seen from equation (6): the percentage of L strategy in a mixed stable population is an increasing function on the sensitivities’ difference and a decreasing one on the benefit of H strategy in mixed competition.

Regarding the results on the structure of the stable regions of the game, we take a deeper look at the Corollary 1. As the benefit in monospecific competitions increases (\(\theta\) and \(\lambda\) for L and H species, respectively), the corresponding area of dominance increases too. That is, the number of the cases in which a monospecific population dominates the competition increases. At the same time, the shape of the opponent dominance area does not change (since \(E_{\textrm{low}}\) and \(E_{\textrm{high}}\) are independent of \(\lambda\) and \(\theta\), respectively). On the contrary, the coexistence area decreases with the benefits in monospecific competition (\(E_{\textrm{coex}}\) is decreasing on both \(\theta\) and \(\lambda\)); that is, the number of the cases in which a mixed population survives decreases as the benefit in monospecific competitions increases. In particular, the biggest coexistence area occurs when both benefits in monospecific competitions are at their minimum value (as in panel g in Fig. 1).

For the special case \(\theta =\lambda =0.5\) (panel c in Fig. 1), we have that \(\lambda -(1-\phi )=\phi -\theta\) and therefore coexistence is not possible. In that case, the benefits \(\theta\) and \(\lambda\) get the maximum value they can; therefore, both strategies can potentially dominate the competition (the dominant strategy derives according to the values of \(\Delta s\) and \(\phi\)).

6 Conclusions

In this article, we developed and analyzed an evolutionary game to describe and extract conclusions for the interactions among plants with different sensitivity to autotoxicity. Evolutionary Game Theory studies how strategies perform, in relation to fitness, against other interacting strategies. As a result, we delve directly into the essence of these interactions. Our model has a simple structure, trying to include into a few parameters (\(\theta ,\lambda ,\phi ,s_l,s_h\)) all the information about the physical process. This simple structure was necessary for an interpretable illustration and the utilization of the results by different scientific fields (mathematics, plant ecology).

By the analysis of the model, we concluded under which conditions the two different species (of low and of high autotoxicity) dominate the competition and which conditions favor the coexistence of the species. We also described the level of the coexistence as the percentage of the two species in the population, by an explicit mathematical formula. Using our analysis, we can describe the borders between the different cases of the model as linear functions, and therefore properly define the space of each different case. Based on these results, we investigated which distribution of species in the population cause a maximization of the total fitness (i.e. the sum of the payoffs of the two players) in stable equilibrium states. We highlighted the result of total fitness maximization only under either coexistence conditions with equal share of the available resources or a strengthened dominance of the low sensitive plant. We also extracted explicit results about the relation between the total payoff in Nash Equilibrium Strategies and the maximum total payoff of the game. The outcomes smoothly follow our intuition.

Autotoxicity is a specific case of Plant-Soil feedbacks (PSFs). In PSFs, the plants alter soil conditions, either abiotic (like self-DNA decomposition) or biotic (like myccorhiza’s populations), in ways that modify the growth of a plant or community subsequently growing in the same soil. Hence, the competition between the plants can also be described implicitly through their interaction with soil for any special model of PSFs. In such a case, an asymmetric game between a population of different plants’ species and their corresponding soil conditions might be a truthful way to describe the model. Note that in the case of \(2\times 2\) asymmetric games equipped with the Replicator Dynamics, we can observe evolutionary oscillations rather than convergence to a stable point [31].

Furthermore, when it is possible to quantify accurately and on the continuoum the phenotypes (i.e, the strategies), either in symmetric or asymmetric games, it becomes worthwhile to extend the matrix (bimatrix for the asymmetric case) game to the context of games with continuous sets of strategies. In such a game we can also consider spatial effects that might affect the evolution of the competition.

In future research, we intend to merge a game theoretical framework with the traditional Ordinary Differential Equations (ODEs) [20] incorporating logistic growth. This integration will involve describing the interaction term between biomass and toxicity as a \(2\times 2\) bimatrix game, where each biomass considers its own toxic compound as well as that of the other biomass. This approach will enable us to model the dynamics of species growth while considering the game-theoretic aspects of the system.

In the present work, we made the significant assumption that the benefits for the two species in a mixed competition are complementary, i.e. add to 1. It is of interest for future work to provide results relaxing this assumption, setting \(\alpha \le 1-\phi\), as we assumed for the benefits in monospecific competitions.