Introduction

Factors affecting the persistence of food webs are a fundamental concern in ecology, particularly as human activities continue to modify community composition and ecosystem function (Scheffer et al. 2001; Simberloff et al. 2013). Anthropogenic species introductions continue to rise (Levine and D’Antonio 2003) with the result that novel predator–prey interactions have become increasingly frequent across terrestrial, marine, and freshwater ecosystems (Ricciardi 2007). A key feature of predator introductions is a lack of shared evolutionary history between predator and prey (Cox and Lima 2006) which can result in strong effects if prey fail to recognize or effectively respond to novel predator cues (Sih et al. 2010). As exotic predators have been implicated in numerous species extinctions (Blackburn et al. 2004), it is of considerable interest to illuminate how predator introductions influence the persistence of native communities and to clarify the mechanisms by which their direct and indirect interactions can lead to native species loss.

Past approaches to understanding the effects of invasion on recipient communities have fallen into two broad categories. The first generates large food webs consisting of species with random variation in trait values and uses simulations to assess the effects of introducing additional species (Lurgi et al. 2014). These primarily numerical approaches have shed important light on the roles that species traits and web topology play in determining invasion success (Romanuk et al. 2009) and in mitigating the effects of invasion on the loss of native species (Lurgi et al. 2014). The complementary approach, which we take here, focuses instead on the analysis of small community modules that reflect the commonly observed interaction structures of just 2–4 species for analytical tractability. Such analyses have proven powerful in isolating the key processes driving dynamics in more complex communities (Holt and Hochberg 2001). Furthermore, by allowing the specification of particular phenomena known to promote native species coexistence, they can provide detail on how novel species traits can alter the function of important stabilizing mechanisms, with implications for broader food web persistence (Kondoh 2008; Stouffer and Bascompte 2010).

One mechanism generally thought to promote coexistence is phenotypic plasticity, whereby flexible responses to environmental conditions — including other species — can confer stability to population dynamics (DeWitt and Scheiner 2004; Miner et al. 2005). A particular form of plasticity, adaptive antipredator defense, allows prey to allocate defensive effort dynamically in response to a changing trophic landscape (Abrams 2000). A well-studied empirical example involves larval tadpoles of the genus Rana that express different morphological adaptations when exposed to dragonfly larvae—increased fin depth to facilitate escape—than when exposed to gape-limited salamanders or fishes—“bulgy” heads that inhibit consumption (Van Buskirk and McCollum 1999). Importantly, each defensive response is predator-specific, allowing the alternate predator to maintain high consumption rates when predator species and the prey’s defensive phenotype are mismatched (Kishida and Nishimura 2005). Such dynamic prey responses can promote predator coexistence by creating a stabilizing tradeoff in the allocation of predator-specific defense effort (Kondoh 2007). Predator-induced morphological or behavioral defenses promote coexistence because prey defense reduces the target predator’s feeding rate when predator abundance is high, resulting in a stabilizing negative feedback (Matsuda et al. 1996). Thus, adaptive responses to predator densities can represent a stabilizing coexistence mechanism (Chesson 2000) that reduces the potential for competitive exclusion by introducing a positive interaction among predators that is mediated through prey traits (Werner and Peacor 2003).

Nevertheless, not all plasticity is adaptive (Padilla and Adolph 1996) and maladaptive plasticity may be particularly common in novel situations, including the presence of novel, non-native species. This is because effective antipredator defense relies on an accurate perception of predation risk and an appropriate antipredator response. Predator novelty can affect either of these components of the prey defense. Lack of eco-evolutionary experience with novel predators can result in a reduced or missing antipredator response (hereafter, naïveté) or can result in a defense that is elicited but ineffective in reducing predation rates (hereafter, suboptimal defense) (Saul and Jeschke 2015). A naïve (lack of) antipredator response is predicted when prey have no evolutionary experience with predators of a similar archetype (Cox and Lima 2006), resulting in a mismatch between predator cues and the prey’s recognition template (Carthey and Banks 2014). In contrast, an introduced predator that is similar to native predators may induce a prey antipredator response due to cue similarity, yet retain high predation rates due to key differences in predatory behavior (Lohrer and Whitlatch 2002). In the latter case, the consumptive effects (CEs) of predation are compounded by non-consumptive effects (NCEs) of costly defensive responses on prey fitness (Sih et al. 2010). Thus, novel predators may represent a significant source of maladaptation in prey, with antipredator defenses being not only energetically costly but also ineffective.

One important module for understanding the effects of adaptive (or maladaptive) defense on species coexistence is the intraguild predation (IGP) module, in which a top predator (hereafter, omnivore) both preys upon and competes with an intermediate predator (consumer) for a shared prey (resource) (Polis et al. 1989). IGP systems have been studied extensively because they comprise a variety of ecological interactions (e.g., tri-trophic chain, apparent competition), display a rich range of dynamical behaviors, and feature prominently in empirical food webs (e.g., Arim and Marquet 2004; Stouffer et al. 2007; Borrelli 2015). Early work using a Lotka-Volterra model with dynamically fixed interactions showed that three-species coexistence was difficult, particularly in productive environments (Holt and Polis, 1997; Diehl and Feißel 2000). With support from microcosm experiments (Morin 1999; Diehl and Feissel 2001), these models thereby spawned intense interest in factors that could increase the coexistence of IGP species, with recent theory and empirical studies highlighting the potential importance of dynamically variable interaction strengths and structures (including additional species) in particular (e.g., (Holt and Huxel 2007; Novak 2013; Pahl et al. 2020). The study of adaptive defense in prey as a cause of variable interaction strengths has featured prominently in these efforts and to date has considered not only generalized (Kimbrell et al. 2007) and predator-specific (Nakazawa et al. 2010) defense types but also the joint use of both types of defense (Ikegawa et al. 2015). However, these studies have assumed prey to have perfect perception of predation risk and effective antipredator defenses. Given the complex tradeoffs inherent to the IGP module, it therefore remains unclear how predator novelty will affect coexistence in contexts of non-native species introductions.

Here, we investigate the effects of predator novelty on the coexistence and stability of the IGP food web module. We consider two invasion scenarios: (i) suboptimal defense, whereby an invader that is similar to the native predator (in terms of predator cues) elicits ineffective antipredator response; and (ii) naïveté toward an unfamiliar invader, for which prey fail to accurately perceive predation risk. Because of the role of enrichment (resource productivity) in shifting the relative importance of competition and predation and determining coexistence in IGP settings (Diehl and Feißel 2000), we examine naïveté and suboptimal defense across a basal enrichment gradient. Further, because the two invasion scenarios (naïveté and suboptimal defense) differ in the presence of NCEs—the magnitude of which depends greatly on the costliness of defense (Peacor et al. 2013)—we consider the effects of predator novelty over a range of defense cost levels. We consider both the situation in which the novel predator is the omnivore and the situation where the novel predator is the intermediate consumer, examining the conditions that lead to the exclusion of the native predator in each. We demonstrate that (i) predator introduction can cause the exclusion of native predators and induce unstable dynamics in previously stable regions of parameter space by weakening the stabilizing feedback provided by adaptive defense; (ii) an introduced omnivore will exclude a native consumer over a wider parameter region than if predator origin is reversed; (iii) three-species coexistence is more sensitive to the effects of suboptimal defense than to naïveté and differentially leads to exclusion of native predators in highly productive environments and when defense costs are low; (iv) moderate novelty of the omnivore—but not the consumer—can increase resource density by strengthening the tri-trophic chain of the IGP model; and (v) consumer novelty can facilitate three-species coexistence by providing a subsidy to an otherwise excluded native omnivore.

Methods

We take as our starting point the IGP model of Nakazawa et al. (2010) which assumes adaptive, predator-specific defense. This model describes the interactions of the resource (R), consumer (N), and omnivore (P) by

$$\frac{dR}{dt}=R\left(r c -\frac{R}{k }-N {a}_{NR}{ d}_{N}-P {a}_{PR}{ d}_{P}\right)$$
(1)
$$\frac{dN}{dt}=N\left(R {a}_{NR}{b}_{NR}{d}_{N}-P{ a}_{PN}- {m}_{N}\right)$$
(2)
$$\frac{dP}{dt}=P\left(R{ a}_{PR}{ b}_{PR}{ d}_{P}+N{ a}_{PN }{b}_{PN}- {m}_{P}\right).$$
(3)

Parameter r is the resource’s intrinsic growth rate and parameter k controls its density dependence, the latter of which we assume to reflect the system’s basal productivity. Parameter aij is the attack rate of predator i on prey j and parameter bij represents the efficiency with which consumed j are converted to i. Parameter c denotes the resource’s total cost of allocating energy from growth toward defensive efforts and parameter di represents the effectiveness of the resource’s defense in reducing predator i’s attack rate (0 ≤ di ≤ 1). The realized attack rate owing to the defense effect is thus di times aij.

The resource’s total cost of defense is comprised of the sum of its defensive efforts toward each predator,

$$c=1- \sum_{i=N,P}{c}_{0}{e}_{i},$$
(4)

where ei represents the level of predator-specific defense effort that is allocated toward predator i. Parameter c0 is a coefficient of cost (common to both predators) that affects a reduction in the resource’s population growth rate as a result of defensive effort (0 ≤ c0 ≤ 1). The magnitude of c0 controls how costly the employment of adaptive defense is to resource’s population growth and thus facilitates comparisons of invasion outcomes along a range of defense costliness from inexpensive (e.g., modestly reduced foraging with c0 near zero) to very costly (e.g., morphological changes as c0 approaches one). It is assumed that any effort allocated toward defense is thereby not allocated to growth and reproduction (0 ≤ eN + eP ≤ 1). Defense effectiveness is similarly assumed to be linearly proportional to predator-specific defense effort, ei, and a defense efficiency parameter, f, such that

$${d}_{i}=1- {e}_{i }f.$$
(5)

The model assumes that the resource species allocates defense effort in such a way that its fitness, w, defined as its per capita growth rate, w = 1/R dR/dt, is maximized. When the resource benefits from perfect perception of the trophic environment, its fitness is given by

$$w=\left(\frac{1}{R}\right)\left(\frac{dR}{dt}\right)=r c- \frac{R}{k}-N{a}_{NR} {d}_{N }- P{a}_{PR} {d}_{P}.$$
(6)

Defensive efforts toward each predator, ei, are specified as dynamic variables that respond to the trade-off that the resource experience between predation risk and fitness gains, adaptively optimized in response to the trophic environment. The dynamics of effort allocation are described by the replicator equations,

$$\frac{d{e}_{i}}{dt}=v{e}_{i}\left\{\frac{\partial w}{\partial {e}_{i}}- \left({\sum }_{x=N,P}{e}_{x}\frac{\partial w}{\partial {e}_{x}}\right)\right\},$$
(7)

such that effort toward the predator i will increase when the gain in fitness of changing the effort, \(\frac{\partial w}{\partial {e}_{i}}\), is greater than \({\sum }_{x=N,P}{e}_{x}\frac{\partial w}{\partial {e}_{x}}\) (Matsuda et al. 1996). The overall rate with which defensive efforts respond to a given fitness gradient is controlled by the adaptive rate, v.

We depart from earlier efforts by considering separately two invasion scenarios. First, we model an introduced predator that elicits the resource’s defensive efforts as a result of similarity with native predators, but to which antipredator defense yields reduced effectiveness (suboptimal defense). We do so by modifying the defense effectiveness equations to include a new parameter, φ, which controls the efficiency of defense against the introduced predator relative to its effectiveness against the native predator (0 ≤ φ ≤ 1):

$${d}_{\mathrm{introduced}}=1- {e}_{\mathrm{introduced} }{f}_{\mathrm{native}}\varphi .$$
(8)

Thus, φ = 1 reflects maximum defense effectiveness toward a novel predator while φ = 0 renders the resource’s defense completely ineffective toward the invader.

Second, to consider an invader that fails to elicit defensive effort commensurate with its predation threat (naïveté), we incorporate a condition in which the resource incorrectly optimizes defensive allocation by maximizing “perceived fitness,” wp, whose difference from w, its true fitness, is determined by the resource’s recognition level, ρ, of the invading predator (0 ≤ ρ ≤ 1). Thus, when the invader is the intermediate consumer, we specify

$${w}_{p}=r c- \frac{R}{k}-{N a}_{NR} {d}_{N } \rho - {P a}_{PR} {d}_{P}$$
(9)

and when the invader is the omnivore, we specify

$${w}_{p}=r c- \frac{R}{k}-{N a}_{NR} {d}_{N }- {P a}_{PR} {d}_{P}\rho .$$
(10)

Naïveté modifies the perceived predation threat of the introduced predator in the fitness equation and hence alters the resource’s adaptive response to changing predator density; ρ = 1 reflects perfect perception (i.e., equivalent to a native-only system), while ρ = 0 reflects complete naïveté toward an introduced predator. Naïveté is propagated through the replicator equations (via perceived fitness) to reduce the level of defense effort allocated toward an unrecognized invader.

We use invasibility analysis to determine the boundaries for three-species coexistence. The invasibility criterion for coexistence requires that each species can increase from low density in the presence of the remaining predator–prey community, a criterion that has been justified for a variety of models (Chesson 2000) and which provides a link between models and empirical tests of coexistence theory (HilleRisLambers et al. 2012). Note that our use of invasibility analysis techniques does not correspond only to an assessment of the conditions under which a non-native predator can invade the native predator and resource predator–prey system, but rather permits the assessment of the boundary conditions between the three-species coexistence and the exclusion of either the native or the non-native predator. Therefore, to avoid confusion with predator origin we henceforth employ the term “coexistence boundary” instead of the more commonly used term “invasion boundary.”

We first solve for the equilibria of the five-dimensional system of equations (three species plus two dynamical defense effort variables) by setting Eqs. 1a–c and 5 to zero. We then evaluate the system’s Jacobian matrix at each equilibrium and use the real part of its maximum eigenvalue (λ) to determine the parameter regions over which each equilibrium exhibits asymptotical stability (Re(λ) < 0). Because defensive effort toward a predator will vary dynamically only when that predator is present, each prey-effort-single predator system is three-dimensional, allowing us to determine the stable regions of these equilibria analytically.

We then determine the coexistence boundaries that describe the conditions under which each predator could invade an existing single-predator system. We rearrange coexistence boundaries in terms of basal productivity (k) because of the role of enrichment in determining the relative importance of interaction types (competition, apparent competition, and predation). Similarly, we examine coexistence boundaries in terms of defense cost (c0) because of its importance in determining the relative magnitude of non-consumptive predator effects. Following previous efforts (Kimbrell et al. 2007; Nakazawa et al. 2010), we examine coexistence along these important gradients with respect to the strength of intraguild predation (aPN, the omnivore’s attack rate on the intermediate consumer) because this parameter controls the degree to which the three-species system reflects a system of exploitative competition versus a trophic chain and thereby provides insight into the shifting dominance between omnivore and consumer.

We determine how the resultant coexistence boundaries are altered by the consideration of suboptimal defense (decreasing φ) and naïveté (decreasing ρ) in order to generate predictions for the types of empirical systems that are most sensitive to each type of invading predator. We use numerical methods to determine regions of parameter space that correspond to locally unstable dynamics (i.e., Re(λ) ≥ 0) reflective of limit cycle or chaotic dynamics. In the five-dimensional case where equilibria are not analytically accessible, we simulate over time to determine the equilibrium densities of each species and levels of defense for each invasion scenario (suboptimal defense and naïveté).

Note that throughout the manuscript, we use novel and novelty as general terms reflecting lack of eco-evolutionary experience with the invading predator that encompasses either component of the predator–prey interaction. A novel predator can therefore have unfamiliar traits that inhibit either recognition or effective defense. Thus, we employ the term novelty parameters to denote the parameters φ and ρ collectively. When only one invasion scenario is being considered, we employ the more specific terms naïveté or suboptimal defense. Similarly, the term predator is employed in the general case that applies to either the omnivore (IG predator) or the consumer (IG prey). When a case applies to only one predator species individually, we specify using the terms omnivore or consumer.

Results

Defense allocation and resource abundance in each single-predator case

For each combination of the resource and a single predator (either the omnivore or the consumer), there are exactly three feasible equilibria—one in which the resource’s antipredator defense is zero, one in which antipredator defense is maximized at 1, and one in which antipredator defense remains at an intermediate value determined by parameter values and the consequent abundance of the predator. These three equilibria do not represent alternative stable states but rather feasibly exist and are stable along different parameter regions, including both productivity (as examined in Nakazawa et al. (2010) and defense cost gradients (Fig. 1). When defense cost is zero, effort is maximized at 1 (Fig. 1c), resulting in the highest possible equilibrium resource abundance (Fig. 1a) and a reduced abundance of the predator (Fig. 1b). Effort remains maximized at 1 as costs increase to a threshold, causing predator abundances to decrease. Then, above a first threshold cost-level separating maximized- and intermediate-defense equilibria (Fig. 1a–c, gray vertical lines), the resource monotonically reduces defense allocation in response to the costs to growth associated with defense. The location of this threshold is determined by specified parameters, including the attack rate of the focal predator on the resource, aiR, and its defense efficiency, fi. Specifically, a higher attack rate or defense efficiency value increases the cost threshold at which the resource reduces defense effort below the maximum. The predator’s density increases linearly with the reduced allocation of defense (Fig. 1b, positive slope). Beyond a second threshold cost level, the resource abandons defense allocation and neither resource nor predator densities vary with further increasing costs. The location of this second threshold is again determined by a combination of parameters that alter the balance between predation risk and defense cost. As intuition would suggest, in the absence of an alternative predator, the resource decreases defense effort unidirectionally along a gradient of increasing defense cost.

Fig. 1
figure 1

Equilibrium values for the resource’s abundance (a), the abundance of a single predator species (b), and the resource’s defense effort toward that predator (c) in the absence of the alternative predator along a gradient of defense costliness. Solid colored lines indicate asymptotically stable equilibria while dashed lines indicate locally unstable equilibria. Vertical gray lines indicate transitions between equilibrium states. Between the ordinate and the left-most vertical gray line, defense effort remains maximized at one and increasing defense costliness results in decreasing predator abundance (negative slope of blue line in panel b). Between the thresholds indicated by vertical lines, defense effort is intermediate and decreases adaptively with increasing defense cost (monotonic decline of purple curve in panel c). In this same region, predator abundance increases and resource abundance decreases because of declining defense effort. To the right of the second threshold, defense effort is adaptively abandoned due to high cost; beyond this cost threshold, predator and resource abundances do not vary with defense cost

Conditions for coexistence of introduced and native predators

With adaptive defense, the coexistence boundary for an introduced predator (the boundary between exclusion and coexistence) is not itself affected by novelty, regardless of whether the predator is the omnivore or the consumer. This somewhat counterintuitive result is explained by the following: in the vicinity of the coexistence boundary, the density of the introduced predator is too low to elicit a defensive response from the resource. With no effort allocated toward defense against a novel predator there is no avenue for defense to be circumvented by novelty. Correspondingly, neither the defense efficiency parameter, φ, nor the naïveté parameter, ρ, is present in the coexistence boundary conditions of either introduced predator (Tables S1, S2).

In contrast, novelty can decrease (or in special cases, increase) the parameter region wherein a native predator coexists with the novel predator and shared resource. Correspondingly, both novelty parameters occur in coexistence boundary conditions for native predators (Tables S1, S2). The model therefore predicts that the parameter space of novel predator coexistence does not depend on traits that induce naïveté or suboptimal defense. Rather, such invader traits alter the coexistence boundary and equilibrium density of the native predator through interactions mediated by both the resource density and its defense allocation, as described below.

Introduced omnivore

Native consumer coexistence along productivity gradient

With adaptive defense, increasing productivity does not simply decrease the IGP strength, aPN, at which the native consumer can persist (Fig. 2a). As observed by Nakazawa et al. (2010), enrichment first reduces the parameter range of consumer coexistence (i.e., lowers the magnitude of aPN at which the consumer can persist) by increasing omnivore abundance, then facilitates greater coexistence via increased defense effort, and finally reduces the range of consumer persistence when defense toward the omnivore is maximized (parameter region of three species coexistence labeled “RNP” in Figs. 2, 3, 56). At maximum defense effort, enrichment causes the exclusion of the native consumer at a lower attack rate aPN because resource defense can no longer compensate for increasing omnivore abundance.

Fig. 2
figure 2

Coexistence conditions across gradients of productivity (k, x-axes) and IGP strength (aPN, y-axes) in the case of an introduced omnivore (P). For all panels, red regions labeled RNP indicate regions of three-species coexistence. Blue regions labeled RN indicate coexistence of resource (R) and consumer (N), but exclusion of the omnivore. Gold regions labeled RP indicate exclusion of the intermediate consumer. White regions indicate regions of locally unstable dynamics within areas of three-species coexistence (oscillatory dynamics confirmed using numerical simulation). Within like-colored regions, transitions between equilibria with different abundance or defensive effort states are indicated by darker boundary lines. In panel (a), both novelty parameters are set to 1 (no naïveté or suboptimal defense) and the resulting stable coexistence region encompasses a large portion of the parameter space shown. Panels (b) and (c) display a reduced region of coexistence with increasingly suboptimal response to the omnivore (φ = 0.75 and 0.5, respectively). Panels (d) and (e) show the effect of increasing naïveté toward the omnivore (ρ = 0.75, and 0.5, respectively)

Fig. 3
figure 3

Coexistence conditions across a range of defense costs (c0, x-axes) and IGP strength (aPN, y-axes) in the case of an introduced omnivore (P). Color, shading, and labeling conventions are as in Fig. 2 with the addition of bistable regions (alternative stable states) labeled RN/RP and RP/RNP. In (a), both novelty parameters are set to 1 (no naïveté or suboptimal defense) and the resulting three-species coexistence region encompasses a large portion of the parameter space. Panels (b) and (c) display a reduced region of three-species coexistence with increasingly suboptimal response to the omnivore (φ = 0.75 and 0.5, respectively). Panels (d) and (e) show the effect of increasing naïveté toward the omnivore (ρ = 0.75, and 0.5, respectively)

The effect of suboptimal response toward an introduced omnivore is to reduce the level of IGP strength, aPN, at which the native consumer is excluded (Fig. 2a–c). Because suboptimal defense results in a reduced efficiency anytime defense is employed, suboptimal defense promotes the exclusion of the native consumer at both intermediate and high levels of enrichment and reduces native consumer persistence to a narrow parameter region of low productivity. Suboptimal defense also reduces the ranges of parameter space that are asymptotically stable in its absence (Fig. 2, white regions).

In contrast to suboptimal defense, naïveté causes the exclusion of the native consumer at intermediate, but not high, productivity (Fig. 2d–e). Further, the overall effect of naïveté on the parameter region of coexistence is low relative to the effect of suboptimal defense (compare Fig. 2d–e to 2b–c). At intermediate productivity when defense effort is intermediate at levels determined by perceived omnivore density, naïveté causes reduced defense effort which in turn causes the native consumer to be excluded. At highest productivity, however, when effort toward introduced omnivore is maximized, defense continues to effectively reduce the abundance of the omnivore. The boundary of native consumer coexistence at high productivity (defense effort maximized) therefore remains unchanged by naïveté. The overall effect of naïveté on stability is similarly proportionally weaker relative to the effect of suboptimal defense, being most pronounced at low IGP strength where the relative strength of the omnivore-resource interaction is highest.

Native consumer coexistence across range of defense cost

Similar to the effect of enrichment, increasing cost does not monotonically decrease the level of IGP strength, aPN, at which the native consumer can persist, but rather also increases the three-species coexistence range over a portion of the cost gradient (Fig. 3b–c, positive slope of RNP-RP boundary toward left side of panels). However, unlike productivity, this increase does not occur when defense is intermediate but rather at the lowest cost range where defense against the omnivore is maximized. This positive relationship between resource defense cost and consumer persistence at maximized defense is mediated through resource density: increasing costs reduce the resource abundance available for the omnivore, thereby depressing the omnivore’s ability to exclude the consumer via predation and exploitative competition. As cost increases further, the resource switches to intermediate levels of defense. The resulting effect of increasing costs is reduced defense effort, which enhances the omnivore’s abundance and promotes the exclusion of the consumer via predation (Fig. 3, negative slope in central region). Finally, at the highest defense costs, the resource abandons defense altogether and coexistence depends on IGP strength, aPN, and initial conditions (Fig. 3, right side of panels). That is, in the absence of defense (here, at high cost) a region of bistability can exist wherein either predator can persist in the absence of the other (as observed by Ikegawa et al. (2015)).

Next, we examine the effects of suboptimal defense across a cost gradient. When effort is inexpensive and maximally employed, making defense less efficient leads to the higher abundance of the omnivore and promotes the exclusion of the native consumer (Fig. 3b–c, reduced area of RNP region). Less efficient defense also leads to the exclusion of the native consumer at intermediate levels of cost by causing the resource to reduce defense effort against the omnivore. Likewise, regions of locally unstable dynamics expand with suboptimal defense, particularly at low cost where defense is maximally employed.

The effect of naïveté on coexistence across a gradient of defense costliness is modest compared to the effect of suboptimal defense (Fig. 3d–e compared to Fig. 3b–c). Further, naïveté toward an introduced omnivore causes the exclusion of the native consumer at intermediate defense cost, but not at low cost. Despite naïveté, defense effort toward the introduced omnivore remains fixed at one when defense is inexpensive. This effectively inhibits the omnivore’s abundance and allows for consumer persistence at low cost. Similar to the effects of suboptimal defense, the effects of naïveté in reducing stability are strongest when defense is too costly to be employed, regardless of any inaccurate perception of predation risk.

Equilibrium densities across a gradient of omnivore novelty

Both suboptimal defense and naïveté of the resource toward the omnivore increase the equilibrium density of the resource (Fig. 4, green curve) compared to the case where the resource exhibits perfect recognition and response (Fig. 4, at the ordinate). This novelty-mediated increase in density is caused by a strengthening of the tri-trophic chain with the IGP model. The increase in resource density is limited to moderate levels of novelty as suboptimal defense and naïveté eventually lead to the abandonment of defense effort toward to invader (Fig. 4, dashed red curves) and a correspondingly lower resource density. Further, the trajectory of defense effort directed toward each predator with low but increasing levels of novelty differs between the two invasion scenarios: suboptimal defense (Fig. 4a) results in an initial increase in defense effort directed toward an introduced omnivore (to compensate for reduced efficiency) and steadily decreasing effort toward the native consumer. In contrast, naïveté (Fig. 4b) results in decreasing effort toward the unrecognized omnivore, causing the resource to shift allocation toward defense directed toward the native consumer.

Fig. 4
figure 4

Equilibrium densities and defense effort levels across the full potential range of suboptimal defense (a) and naïveté (b) toward an introduced omnivore (P). Solid green, blue, and red lines indicate equilibrium densities of the resource, consumer, and omnivore, respectively. Dashed lines indicate levels of defense effort directed toward each predator species (colors match predator identity). Suboptimal defense toward the omnivore (a) results in increased but ineffective defense effort toward the omnivore, eventually resulting in consumer exclusion. In (b), naïveté causes decreased effort toward an unrecognized omnivore. Defense effort is shifted toward the consumer, which is again excluded at higher levels of omnivore novelty. In both cases, the maximum resource abundance does not occur at perfect recognition and optimal defense (φ = ρ = 1) but rather at intermediate levels of omnivore novelty due to a novelty-mediated release from consumer predation. Note: Parameter values differ from those used in other figures in order to more clearly represent the effects of defense effort allocation in each invasion scenario (see Table S3)

Introduced consumer

Due to the asymmetrical nature of IGP, novelty of an introduced consumer affects only the coexistence boundaries of the IGP system under restricted conditions: where defense against the consumer is employed (see coexistence criteria in Table S2) and where reduction in effectiveness of defense (via novelty) can result in the consumer outcompeting the native omnivore for resources.

Native omnivore coexistence along productivity gradient

The effect of enrichment on the coexistence boundary for the native omnivore (Fig. 5, coexistence boundary between RNP and RN-only system) is simpler than for the native intermediate consumer, facilitating coexistence of the native omnivore over a greater range of IGP strengths, aPN. However, novelty of an introduced consumer modifies the boundary of omnivore coexistence in complex ways because the consumer represents both its competitor and resource. Whether suboptimal response to a consumer causes omnivore exclusion or facilitates consumer persistence depends on the relative contribution of energy to the native omnivore by each of its prey, which in turn depends on relative efficiencies of the indirect versus direct pathways. At lowest productivity, defense toward the consumer is minimized at zero and the coexistence boundary remains unchanged by novelty (Fig. 5b–c and see coexistence criteria in Table S2). When adaptive defense is intermediate at a magnitude set by predator abundance (at intermediate productivity), decreasing defense efficiency results in an increase in consumer abundance (which promotes omnivore coexistence) but a decrease in the abundance of the resource (which promotes omnivore exclusion). Similarly, when effort is fixed at one (high productivity), a suboptimal response results in a reduced resource abundance but an increased consumer abundance. Whether this leads to the exclusion of the omnivore depends on the efficiency of pathways. As the majority of energy flux is through the indirect route at high productivity, suboptimal defense results in the exclusion of the native omnivore at a higher level of aPN (Fig. 5a–c). With regards to stability, in the region of parameter space where adaptive defense comes into play, suboptimal defense toward the consumer serves to strengthen the consumer-resource interaction, resulting in a transition from unstable three-species coexistence to a stable consumer-resource system.

Fig. 5
figure 5

Coexistence conditions across gradients of productivity (k, x-axes) and IGP strength (aPN, y-axes) in the case of an introduced consumer (N). Color, shading, and labeling conventions as in Fig. 2. In (a), both novelty parameters are set to 1 (no naïveté or suboptimal defense toward the consumer). Panels (b) and (c) display a reduced region of coexistence with increasingly suboptimal response to the consumer (φ = 0.75 and 0.5, respectively). Panels (d) and (e) show the effect of increasing naïveté toward the consumer (ρ = 0.75, and 0.5, respectively)

The effect of naïveté toward an introduced consumer on omnivore coexistence is modest compared to the effects of suboptimal defense (Fig. 5d–e). Moderate naïveté (ρ = 0.75) results in little change in coexistence boundaries or stability (Fig. 5d). However, there is a sharp threshold of naïveté above which the resource abandons defensive effort toward the consumer, resulting in the omnivore’s exclusion across a broad range of productivity (Fig. 5e). The location of this threshold is determined primarily by the attack rate of the consumer on the resource, aNR, and the efficiency of resource defense toward the consumer, fN (and see Table S2). A narrow region of stable, three-species coexistence is surrounded by omnivore exclusion on all sides, indicating a region along the productivity gradient where adaptive defense toward the consumer facilitates omnivore coexistence by reducing direct competition.

Native omnivore coexistence across a range of defense cost

Increasing cost of defense decreases the parameter region of native omnivore persistence in the absence of novelty (Fig. 6a). When the indirect pathway is dominant and the omnivore primarily feeds on the intermediate consumer (at low productivity and high IGP strength), increasing costs of defense has little effect on omnivore persistence. In contrast, when the direct pathway is dominant (low IGP strength), suboptimal defense reduces the parameter region of omnivore coexistence by reducing the availability of the resource at both low and intermediate cost (Fig. 6a–c). There is no effect of suboptimal defense at high cost because defense is not employed. Notably, the region where adaptive defense is intermediate (between maximum and absent) is severely truncated by highly suboptimal defense.

Fig. 6
figure 6

Coexistence conditions across a range of defense costs (c0, x-axes) and IGP strength (aPN, y-axes) in the case of an introduced consumer (N). Color, shading, and labeling conventions as in Fig. 2 with the addition of bistable regions (alternative stable states) labeled RN/RN and RP/RNP. In (a), both novelty parameters are set to 1 (no naïveté or suboptimal defense). Panels (b) and (c) display a reduced region of coexistence with increasingly suboptimal response to the omnivore (φ = 0.75 and 0.5, respectively). Panels (d) and (e) show the effects of increased naïveté toward the omnivore (ρ = 0.75, and 0.5, respectively)

In contrast to suboptimal defense, naïveté does not have any effect on coexistence when cost is low and defense is maximally employed (Fig. 6d–e). Provided that defense remains maximally employed, it still effectively reduces consumer predation of the resource and precludes competitive exclusion of the native omnivore. Naïveté does reduce the allocation of defense toward the consumer at intermediate cost, thereby increasing the magnitude of IGP strength at which the omnivore is excluded. Further, naïveté results in the emergence of both alternative stable states (bistability) and unstable dynamics. This is because the equilibrium threshold cost-level at which defense is abandoned is reduced below the threshold cost-level separating maximal and intermediate defense. This means that there are two alternative consumer-resource states possible (labeled RN-RN in Fig. 6d), one with defense maximally employed and one with defense absent. Similarly, we observe a region of alternative states where either consumer-resource and no defense or three-species coexistence with adaptive defense toward the consumer occur. This presence of alternative stable states in the case of naïveté but not suboptimal defense deserves further exploration in future efforts.

Equilibrium densities across a gradient of consumer novelty

Increasing novelty of an introduced consumer results in a decreasing abundance of the resource (Fig. 7; solid green lines). In contrast to the increased resource abundance observed with the novelty of an introduced omnivore (via the tri-trophic chain), neither suboptimal defense nor naïveté toward the introduced consumer increases resource abundance. In fact, since the consumer is also prey for the native omnivore, increasing novelty of the consumer can subsidize the omnivore and result in further reduced abundances of the resource. The abundance of the introduced consumer itself (Fig. 7; solid blue curves) increases until naïveté results in the abandonment of antipredator defense by the resource. Increasing novelty can result in exclusion of native omnivore (Fig. 7; solid red curves) when omnivore persistence depends on the increased resource abundance provided by defense (at low productivity and low IGP strength). Conversely, consumer novelty can also facilitate persistence of a native omnivore (Fig. S1) that would be otherwise excluded (in the absence of consumer novelty) by subsidizing the omnivore via the indirect route energy pathway. As resource defense toward the consumer becomes less efficient, the omnivore can invade the system subsidized by increasing abundance of the basal resource (Fig. S1, red curve).

Fig. 7
figure 7

Equilibrium densities and defense effort levels across the full potential range of suboptimal defense (a) and naïveté (b) toward an introduced consumer (N). Solid green, blue, and red lines indicate equilibrium densities of the resource, consumer, and omnivore, respectively. Dashed lines indicate levels of defense effort directed toward each predator species (colors match predator identity). Suboptimal defense toward the consumer (a) results in increased but ineffective defense effort toward the consumer, resulting in reduced resource densities and omnivore exclusion. In (b), naïveté results in decreased effort toward an unrecognized consumer. Defense effort is shifted toward the omnivore, which increases with increasing naïveté toward the consumer. In both cases, the maximum resource abundance occurs at perfect recognition and optimal defense (φ = ρ = 1) because there is no mechanism by which novelty of a consumer can indirectly facilitate resource abundance

Discussion

In this study, we investigate the effects of an introduced predator on IGP community coexistence. Previous work has identified adaptive defense as an important phenomenon that can contribute to food web stability, yet to our knowledge, ours is the first study to examine the effects of predator novelty on species coexistence in an IGP context. We find that a novel predator reduces the parameter region of three-species coexistence by excluding the native predator from a parameter space in which it could otherwise persist in the absence of novelty. This exclusion occurs when the stabilizing positive interaction between predators, mediated by predator-specific defense allocation in shared resources, is eroded by either reduced or ineffective defense toward the invader. Thus, while our model corroborates previous theory regarding the positive effect of adaptive defense on community coexistence (Matsuda et al. 1996; Kondoh 2007; Nakazawa et al. 2010), we extend these findings by showing that coexistence is sensitive to the assumptions of perfect risk perception and effective defense that may not be justified for novel predators.

While both naïveté and suboptimal defense reduced the parameter region of coexistence, we found strong differences in the magnitude of changes caused by each invasion scenario. Specifically, holding other parameters constant, naïveté had far more modest effects on native predator persistence than suboptimal defense. The mechanism primarily driving differential effects on coexistence boundaries was not the presence of non-consumptive effects (NCEs) in suboptimal defense, as expected. Indeed, compared to naïveté, suboptimal defense eroded coexistence to a greater extent at low defense cost, not high cost as would be predicted if differences were driven by NCEs. Rather, changes in the region of coexistence resulted from the relationship between defense effort and defense effectiveness in the two invasion scenarios. Specifically, naïveté reduced the level of defensive effort that prey allocated to a novel predator, but allowed defense to remain maximally effective provided that it was employed. In contrast, suboptimal defense caused prey to increase effort in response to reduced defense efficiency, but the effectiveness of the defense toward the invader was eroded even at high levels of defense effort. This difference can be seen by examining which equilibria are sensitive to novelty: while naïveté alters only the equilibria corresponding to intermediate levels of defense effort (when effort varies based on perceived predation threat), suboptimal defense changes the equilibria and hence the coexistence boundaries associated with both intermediate and maximized defensive effort (Table S1; Figs. 2, 3, 5, 6).

Effects of introduced predators across productivity and defense cost gradients

Because the clearest differences between invasion scenarios arise when defense is maximally employed, and because defense investment changes along productivity and defense cost gradients, our model makes specific predictions for where along these gradients species exclusion will occur in each scenario. First, our model predicts that highly productive systems will be more sensitive to an introduced predator that provokes suboptimal responses than to an unfamiliar predator to which prey are naïve. Productivity has long been recognized as one of the primary factors determining coexistence in IGP systems, shifting the importance of exploitative competition and predation and precipitating the exclusion of the omnivore and the consumer at low and high productivity, respectively (Holt and Polis 1997; Diehl and Feißel 2000). While adaptive defense can widen the productivity region at which three-species coexistence occurs, we show that this effect is sensitive to predator novelty, and that an introduced predator will alter coexistence at different regions of productivity depending on whether it elicits suboptimal defense or reduced recognition. In the case of an introduced omnivore, defense is maximally employed at the highest productivity region. Suboptimal defense therefore lowers the strength of the omnivore’s predator effect at which the native consumer is excluded across a broad range of intermediate to high productivity. In contrast, because naïveté does not alter the effectiveness of defense provided that it is employed, native consumers can persist at the highest levels of productivity in this invasion scenario. It has been previously demonstrated that, at low defense efficiency, adaptive antipredator behavior can reinforce the paradox of enrichment (Rosenzweig 1971) by lowering the productivity levels at which oscillatory dynamics replace stable IGP coexistence (Urbani and Ramos-Jiliberto 2010). This is in line with our results regarding the effect of suboptimal response toward an introduced omnivore: coexistence at high levels of productivity was dramatically reduced by lowered defense efficiency (suboptimal response) but not by reduced recognition (naïveté).

In contrast to productivity, the effects of defense cost on communities with adaptive defense have been relatively understudied. Abrams and Fung (2010) compared IGP models with cost-free versus costly defense in terms of their responses to top-down and bottom-up effects. However, their treatment did not explore how varying levels of costliness alter the employment of defense, and hence to changes in coexistence across a defense cost gradient. Using a graphical model of adaptive trait change, Peacor et al. (2013) clarified the effects of higher or lower costs of adaptive trait change on fitness to identify when large NCEs should be expected. Specifically, they predicted that large NCEs will occur when defense is costly, but the benefits outweigh these costs because predation in the absence of defense is high. Therefore, exploring the effects of novelty across a cost gradient, we expected the greatest differences between suboptimal defenses versus naïveté would occur in high-cost situations because of the increased relative strength of NCEs. In contrast, we observed that the greatest differences occurred at low cost, when defense was maximally allocated. Specifically, neither suboptimal defense nor naïveté results in altered coexistence in the high-cost region because defense is simply too expensive to employ. Further, both components of novelty caused exclusion when costs are intermediate—in essence suboptimal response and naïveté always oppose the positive effect on coexistence provided by adaptive defense. The discrepancy between invasion scenarios appears at the lowest range of defense cost because when defense is essentially cost-free it will be employed regardless of reduced effectiveness or predation recognition. This employment of defense causes an increased parameter range of exclusion in the case suboptimal defense but not naïveté. Thus, systems with low-cost defense are predicted to be more sensitive to the effects of an introduced predator that matches a prey’s recognition template than to the effects of an unfamiliar predator.

Nonetheless, we also observed that the relationship between defense cost and native predator coexistence depends on the identity of the introduced predator and the relative importance of the direct versus indirect energy pathway from resource to omnivore. As shown by Ikegawa et al. (2015), predictions about IGP coexistence across parameter gradients are strongly dependent on the relative strength of the direct versus indirect energy pathways. In the case of an introduced consumer, which competes with—but also provides a food resource for—the omnivore, the effects of increasing defense costs can either promote or preclude native omnivore coexistence. Our study therefore highlights the importance of quantifying interaction strengths in IGP systems as their response to defense cost will differ between communities that approximate a food chain compared to those that are more similar to exploitative competition scenarios (Stier et al. 2016).

Effects of novelty on resource abundance and native predator facilitation

Many of the predictions from our model align with and are explainable by the characteristic asymmetry between predators that is inherent to IGP systems. However, several counterintuitive predictions also emerge. For example, a resource that inaccurately assesses the predation threat posed by an introduced omnivore shows higher equilibrium density relative to a resource with perfect perception of the trophic landscape. This result occurs because, in maximizing its own per capita growth rate through defense allocation, the resource indirectly benefits the intermediate consumer by reducing the growth rate of the omnivore. Hence, perfect perception results in reduced resource density. Naïveté reduces the recognition of the predation threat posed by, and therefore the optimal level of effort toward, the omnivore, thereby increasing predation on the consumer and enhancing resource abundance. Naïveté can thereby increase the strength of the trophic cascade. The evolution of predator-specific adaptive defenses which are insensitive to the indirect effects of multi-predator systems may therefore not result in increased abundance. This observation raises questions regarding the degree to which prey adaptations for predation risk assessment and defense allocation reflect the effects of multiple predators and the indirect effects among them in nature.

The highest resource density is found at an intermediate novelty of the omnivore due to the strengthening of the trophic cascade via either suboptimal defense or naïveté. In contrast, increasing novelty of an intermediate consumer results only in a reduced or unchanged resource abundance as there is no indirect effect through which reduced defense toward the consumer can benefit the resource. Therefore, our analyses predict that the introduction of trophic omnivores can either increase or decrease resource abundance, but that the introduction of intermediate consumers can only reduce the abundance of the resource. In fact, since consumers also represent a second resource for the omnivore, novelty of a non-native consumer can facilitate the persistence of an omnivore that would otherwise be excluded. While this observation runs counter to simple intuition, increasing novelty weakens the effect of resource defense, making consumers more abundant prey for the omnivore. That is, both suboptimal defense and naïveté toward an introduced consumer can promote increased consumer abundance in such a way that it acts as an energy subsidy for a native omnivore that would otherwise be unable to persist at the same parameter region in the absence of novelty. This prediction is supported by empirical studies showing that introduced prey can in fact benefit natives of higher trophic levels when native predators also have access to native prey (Pintor and Byers 2015).

Predator novelty and invasion success

Another prediction of the model is that predator novelty does not alter the coexistence boundary for the introduced predator itself. Rather, because the coexistence boundaries for the non-native predator necessarily occur in parameter regions where that predator represents a low predation threat, the resource does not allocate defensive effort toward the invader and defense can therefore not be circumvented by novelty. However, this result should not be interpreted to imply that novel traits cannot alter initial invasion success (as defined by the establishment of a persistent self-sustaining population) for two reasons. First, we examined equilibrium coexistence conditions, and the behavior of the system at an invasion (coexistence) boundary should not be equated with the transient dynamics likely to occur during the initial phases of an empirical invasion. Second, in our model, defense is allocated in such a way that it maximizes resource per capita growth in response to changes in predator population density; therefore, prey do not allocate defense effort toward predator species at low equilibrium abundance. Yet, in nature, prey individuals may indeed display a defensive response to predators that pose an immediate predation risk, regardless of the predator population size. Further, local predator density may be uneven, leading to a patchy landscape for both predation risk and predator cues. For phenotypic and behavioral defenses, individual prey experience only the local trophic environment, which may differ from the aggregate mean predation risk at the population level. An individual invader could therefore experience a fitness gain if novel traits facilitated high predation rates by rendering prey defense ineffective, regardless of predator population density. We therefore conclude that predator traits may very well influence initial invasion success (Lurgi et al. 2014), but that the invader’s coexistence boundaries at equilibrium will not vary based on its ability to circumvent prey defense.

Key assumptions of the model

Alternative modeling choices may alter the inferences of model and should be explored in future work. First, in order to preserve tractability of the model, we limited our analysis to a strict IGP system without embedding the module in a wider food web. While the dynamic effects that increasing species and interaction richness may have are often difficult to predict (Novak 2013), models that include additional resource species do often show increased IGP coexistence (Holt and Huxel 2007; Daugherty et al. 2007). It is possible, therefore, that the corrosive effects of predator novelty on coexistence would be mitigated in more complex systems.

Second, we considered predator-specific defenses only and it would be informative to explore the effects of imperfect predator recognition or response in the context of generalized or joint-use defenses. While adaptive predator-specific defenses promote coexistence of multi-predator systems by allowing the resource to allocate effort in response to dominant predators (Lima 1992; Matsuda et al. 1996; Kondoh 2008; Nakazawa et al. 2010), this stabilizing mechanism is not present with generalized defense except in the presence of further model restrictions (Matsuda et al. 1993; Kimbrell et al. 2007). Further, Ikegawa et al. (2015) showed that the joint use of predator-specific and generalized antipredator defense promotes three-species coexistence in IGP systems even at high productivity. It is therefore entirely possible for resources to be protected from attack by a novel predator—even one that is unrecognized as a predation threat—by a generalized defense that is elicited by recognition of a native predator alone.

Third, in order to facilitate analytical tractability, we assumed linear functional responses for all predator–prey interactions. In contrast, saturating functional responses are more commonly inferred to occur in empirical studies (Jeschke et al. 2004) and are predicted to alter the stability properties of IGP at high productivity levels in particular (Diehl and Feißel 2000; Mylius et al. 2001; Křivan and Diehl 2005). Therefore, our predictions may be most applicable at low prey abundance, where trophic control is strong, and the predator consumption rate is adequately approximated by a linear function. We also assumed a linear relationship between defense effort and effectiveness at reducing attacks rates by the target predator. A non-linear functional form may be more biological reasonable in many cases (e.g., a decelerating function that reflects diminishing returns of increased defense effort). Nevertheless, Peacor et al. (2013) note that their qualitative predictions (regarding fitness optima) were identical when comparing linear and non-linear relationships between defense effort and reduced predation. We anticipate decelerating functions would lower the optimum defense effort compared to the linear case, but this model variation too merits future consideration.

We considered suboptimal defense and naïveté separately in order to compare the cases of similar predators bearing cue similarity to natives, and predators representing a novel predator archetype to which resources are naïve. In nature, however, these two components of novelty are not mutually exclusive (Sih et al. 2010) and prey may respond to a single introduced predator with both reduced effort and a lowered effectiveness of antipredator defense (Carthey and Banks 2014). The total effects of such a predator (combining strong CEs and weak NCEs) are expected to be intermediate between that of a totally novel predator (CEs only) and that of a similar predator that elicits suboptimal defense (strong CEs and NCEs) (Sih et al. 2010). Based on our work, we anticipate that a novel predator to which resources display both naïveté and suboptimal response will cause the exclusion of native predators over a large parameter range, primarily owing to the negative effects of suboptimal defense on native persistence over the entire parameter region in which it is employed.

Finally, we motivated our work with a well-known empirical example of induced antipredator defense and assumed that phenotypic plasticity provides the adaptive stabilizing mechanism promoting coexistence. Behavioral or evolutionary trait change each represents alternative stabilizing mechanisms (at shorter and longer timescales, respectively, compared to induced defenses) that could be explored with similar models. The adaptive rate of trait change (in our model, represented by the parameter v) provides a means to “tune” the rate of adaptation to the appropriate timescale relative to population dynamics (Matsuda et al. 1996), but we did not consider variation in this parameter in our analyses. Adaptation rates in response to the introduced predator that are slower than the rate we assumed may provide an alternative means for representing partial naïveté in models that should be explored. That said, over evolutionary scales, prey response to novel predators itself is not static, so our model may be most applicable to plasticity or behavioral dynamics. Future modeling scenarios that allow prey recognition and defense efficiency to evolve in response to selection would improve our understanding of the role of novelty in shaping invasion outcomes.

Conclusions

The effects of predator introductions on natural communities span the entire continuum from a failure to establish viable populations to precipitating native food web collapse (Mack et al. 2000). Understanding the interaction between predator traits and native community characteristics that determine which scenario plays out for given introduction remains a major goal of conservation ecology (Kolar and Lodge 2001; Mata et al. 2013). Here we demonstrate theoretically that predator novelty can lead to native species exclusion in an IGP module. In a broader food web context, such biodiversity loss may lead to secondary extinctions (Lundberg et al. 2000; Dunne et al. 2002). Moreover, and even in the absence of initial native predator exclusion, introduced predator novelty could alter population dynamics by influencing interaction strengths throughout the food web. Indeed, we observed the emergence of locally unstable (limit cycle) dynamics in parameter regions where adaptive defense had—in the absence of novelty—dampened the focal predator–prey interaction strength below the threshold between stable and unstable dynamics. Invader traits that result in ineffective antipredator defense could even facilitate subsequent invasions (Simberloff and Von Holle 1999), especially if they lead to increased resource availability, as was demonstrated by our model in the case of an introduced omnivore. These and the many other ways that predator novelty can influence species coexistence suggest that the predictions of models that do not consider these attributes of non-native predators may be limited in their empirical application.