What can cognitive linguistics tell us about language-image relations? A multidimensional approach to intersemiotic convergence in multimodal texts

4.1 Schematisation

Arguably the most fundamental dimension of conceptualisation lies in image-schematic representations of event-structure (Langacker 2008; Talmy 2000). Image schemas stand as archetypal conceptions representing basic patterns of experience (Langacker 2008: 355). Their role in structuring knowledge within semantic domains like action, force, space and motion, where they serve as folk theories of the way the world works, has been widely studied in Cognitive Linguistics (Hampe 2005; Johnson 1987; Oakley 2010). A key claim of Cognitive Linguistics is that such schemas “work their way up in to our system of meaning” (Johnson 1987: 42) to constitute the semantic basis of linguistic – lexical and grammatical – forms. One pervasive pattern of experience, for example, is an interaction involving the transference of energy, through forceful physical contact, from one participant to another. This type of interaction is represented by an action chain schema which, Langacker (2002, 2008 argues, forms the conceptual basis of the prototypical transitive clause. Talmy (2000) has similarly argued that image schemas representing various types of force and motion event are encoded in the meanings of both open and closed class linguistic elements. In discourse, the linguistic form selected to describe an event is not determined by any objective properties of the referential situation but rather invites a particular conceptualisation of it. Image-schematic structuring through language is thus a matter of construal. The same or ostensibly the same type of material situation may be schematised differently through alternate linguistic formulations. For example, in the context of media discourse on political protests, Hart (2013) has shown that violent interactions between police and protesters can be schematised as action, force or motion events depending on the ideological perspective of the text-producer and that within these domains alternate schemas are available to further construe the same situation in different, ideologically vested ways. Crucially, in terms of L-I relations and intersemiotic convergence, the same image-schematic patterning may underpin co-text images. Take, as a first example, the image-caption combination in (2). In this usage event, language and image converge in instantiating the three-participant action chain represented in Figure 2. Different types of action and event are associated with different archetypal roles (Langacker 2008: 356). A three-participant action chain represents a transfer of energy from an agent to a patient via an instrument. In (1), this schema is instantiated in the ditransitive construction where the agent is encoded as subject, the instrument as direct object and the patient as indirect object. The same event-structure is seen in the image where all three participants are represented and the interaction between them is suggested by the dynamicity of the image. For Kress and van Leeuwen (2006), this dynamicity is what gives the image its ‘narrative’ structure and is a product of vectors formed by visually depicted elements. In (2), such a vector is formed by the outstretched limb of the refugee. The continued trajectory of this ‘effector’ implies the bottle’s direction of travel toward the police and thus the sequence of energy flow between participants. The vector formed in the image may therefore correspond with the arrows representing energy transference in Figure 2. From a perspective more akin with Simulation Semantics, it has been shown that static photographs of human actions where there is implied motion activate motor areas of the brain (Kim and Blake 2007; Kourtzi and Kanwisher 2000; Proverbiaet al. 2009). This suggests that in understanding images viewers ‘complete the picture’ by performing dynamic simulations which unfold along the lines laid down by corresponding image schemas.

Figure 2:

Three-participant action chain.

(2)

A refugee throws a bottle toward Hungarian police at the “Horgos 2” border crossing into Hungary, near Horgos, Serbia.

Telegraph.co.uk, 22 September 2015

In (2), the archetypal conception instantiated in both language and image is a one-tailed action schema in which the transfer of energy flows unidirectionally from an agent as the energy source to a patient as the energy sink. However, another archetypal conception represents a bi-directional exchange of energy between two participants who are both agentive and thus simultaneously act as energy source and energy sink in the interaction. This two-tailed action schema, represented in Figure 3, forms the meaningful basis of reciprocal verbs. In discourse, it serves to construe a physical interaction as two-sided and thus assigns mutual blame and responsibility for the event (Hart 2018). In example (3), this schema is instantiated linguistically by the reciprocal verb ‘clash’ but also visually by the co-text image. In contrast to the image in (2), in which only one participant is depicted in an ‘action shot’ while the second participant is more passive, in (3) both sets of participants are shown engaged in the mutual transfer of force.

Figure 3:

Two-tailed action schema.

There is also a clear contrast in viewpoint between examples (2) and (3) with the viewpoint in (2), realised both verbally and visually, being from that of the agent of the action and the viewpoint in (3), again realised both verbally and visually, being from that of an observer (see Section 4.2).

(3)

Police and protesters clash in Oxford Circus.

Telegraph.co.uk, 27 March 2011

In the domain of motion, Talmy (2000) identifies several different types of motion event. However, a basic distinction that can be made is between motion events which are force-dynamically neutral and those that involve a force-dynamic component. In a force-dynamically neutral event, the impetus for motion begins with the agent and their ability to move is not hindered in any way. Motion events that involve a force-dynamic component include caused motion and impeded motion. In impeded motion, the agent’s ability to move freely is constrained by the presence of some ‘barrier’ which they are able to circumvent or penetrate in order to complete the intended translocation.^[3] These two types of impeded motion event are represented by image schemas such as modelled in Figure 4.^[4] The arrows in Figure 4a and 4b represent a path of motion rather than a transfer of energy and the stepped arrow in Figure 4b represents the change in state to the barrier brought about in the course of realising the event. Many linguistic expressions, both open and closed class, include a force-dynamic conceptualisation as part of their meaning (Talmy 2000). This includes the try + infinitive construction which focuses on an effort to overcome an obstacle (without making known the outcome of this effort) (Talmy 2000: 436–437).^[5]

Figure 4:

Impeded motion schemas.

In news texts reporting immigration, we find L-I combinations such as (4) where an impeded motion schema is instantiated linguistically by forms like ‘trying to reach’ and ‘attempting to get to’ but also visually by the image of a border fence, which immigrants are shown climbing through or over.

(4)

Migrants in Calais attempting to get to Britain.

Express.co.uk 15 August 2015

These L-I combinations may thus be said to converge in schematising immigration in force-dynamic terms. They are candidates for a discourse-level construction where, in the visual mode, the barrier element of the impeded motion schema is conventionally instantiated by images of fences – a recurrent visual trope in immigration discourse (see Dancygier and Vandelonotte 2017 for discussion of visual instantiations of the barrier schema). However, there is a difference between the two modes in levels of specificity. While in the language, the form or nature of the impediment is not specified and the manner by which it is overcome (e.g., circumvented vs. penetrated) is not expressed, this information is contained within the co-text image. Such L-I combinations may therefore be described as exhibiting a hyponymic relation whereby the image instantiates a more specific type of impeded motion schema. In this sense, while convergent at a basic level of schematisation, the image in (4) may also be said to supplement information expressed in its verbal co-text. The schema instantiated by the image in (4) is the one represented in Figure 4b. Of course, intersemiotic convergence is not limited to images and their captions but may also be found between images and conceptualisations evoked by other regional and functional parts of the text (Bednarek and Caple 2012: 96–100). In the case of (4), for example, the headline of the article instantiated the same schema as the image: “Migrants trying to ‘break into’ Britain”.

Dancygier and Vandelanotte (2017) argue that image schemas like the barrier schema include as part of their meaning viewpoint affordances whose realisation in usage events yields different experiential results with clear emotional consequences. The viewpoint in (4), instantiated multimodally, construes the barrier from the perspective of ‘us’ on the other side of the barrier to the migrants.

Another aspect of schematisation concerns not the event-structure itself but the participants within it. Talmy (2000) suggests that distinctions within the linguistic category of number, i.e., singular versus plural (as well as aspectual distinctions like semelfactive versus iterative), are accounted for conceptually in terms of plexity of structure (see Figure 5).

Figure 5:

Multiplex versus uniplex.

For Talmy (2000: 48), plexity is “a quantity’s state of articulation into equivalent elements”. While singular nouns specify a uniplex referent, plural nouns specify a multiplex referent. The construal evoked by examples like (3) and (4) is therefore one in which the referents are treated as multiplex. However, certain nominal forms, including collective nouns and noun phrases, construe multiplex referents as uniplex. In opposite relation to what Talmy describes as multiplexing, this may be said to represent a cognitive operation of uniplexing. It is found in example (5) where the collective NP ‘column of migrants’ construes the group of immigrants being referred to as a uniplex structure of a specific oblong shape. In images, plexity is realised in the dispersion of, or degree of agglomeration of, visually depicted elements. The higher the degree of agglomeration (and therefore lower dispersion), the more uniplex the structure. In (5), language and image are therefore intersemiotically convergent in the dimensions of plexity and shape where the co-text image displays a high degree of agglomeration such that the individual migrants come to form a uniplex structure that is similarly oblong in shape. Rhetorically, this does several things which are worth commenting on. In both modes, immigrants are aggregated or de-individuated so that their own personal stories are not recognised and they can all be viewed and treated in the same way. It is also worth noting that the image in (5) is unbounded – other than by the frame of the photo. A structure that is unbounded is conceived as “continuing on indefinitely with no necessary characteristic of finiteness intrinsic to it” (Talmy 2000: 50). In this context, the unboundedness of the image functions rhetorically to suggest a ‘column’ of significant magnitude.

(5)

The huge column of migrants passes through fields in Rigonce, Slovenia, after having been held at the Croatia border for several days.

MailOnline, 25 October 2015

Plexity is related to viewpoint where the further away one is from a given scene, the more uniplex that scene becomes (see Hart 2015). Hence, the camera angle presented by (5) is that of an aerial shot. We turn to viewpoint in the proceeding section.

4.2 Viewpoint

If schematisation represents one dimension of construal, in so far as it involves the apprehension of particular conceptual content in order to conceptualise the structural properties of the referential situation, further dimensions of construal concern the way in which that conceptual content is viewed. As Langacker (2008: 55) states:

An expression’s meaning is not just the conceptual content it evokes – equally important is how that content is construed. As part of its conventionalised semantic value, every symbolic structure construes its content in a certain fashion … In viewing a scene, what we actually see depends on how closely we examine it, what we choose to look at, which elements we pay most attention to, and were we view it from.

Other dimensions of construal, then, are based in cognitive systems of perspective and attention (Croft and Cruse 2004; Langacker 2008; Talmy 2000). In terms of perspective, a key claim of Cognitive Grammar is that “many expressions undeniably invoke a vantage point as part of their meaning” (Langacker 2008: 75). Zwaan (2004) argues that in comprehending an utterance, hearers are immersed experiencers who simulate the scene described from a particular linguistically cued perspective. On the basis of these claims, Hart (2015) proposes an embodied ‘grammar’ of viewpoint modelled in three dimensions – anchor, angle and distance – and argues that the meanings of alternate linguistic expressions can be characterised, in part, as a shift in one or other of these aspects and that linguistic forms which include a viewpoint specification as part of their meanings therefore have analogues in images, which, by the hypothesis, share the same perspective. According to this model, collective nouns such as found in (5) combine vertical angle with maximal distance to engender a construal analogous to the aerial shot found in the image of (5). On the anchor plane, where a viewpoint shift equates to panning, transitive versus reciprocal verb constructions are analysed as not only indexing alternative image schemas but also as including contrasting viewpoints as part of their semantic values. On this hypothesis, transitive constructions encode a viewpoint which construes the event sagittally from the perspective of the agent or the patient depending on voice while reciprocal constructions encode a viewpoint which construes the event transversally from a perspective that is orthogonal to that of participants within it and which is equidistant between them (Hart 2015). The contrasting viewing arrangements (Langacker 2008) that result are represented in Figure 6.

Figure 6:

Viewing arrangements in the anchor plane.

This hypothesis is borne out experimentally. In a sentence-image matching task, Hart (2019) gave subjects transitive and reciprocal verb constructions and asked them to indicate which schematic image, presented in four orientations, best represented the situation described in each of the sentence types. The results showed a significant level of agreement between subjects where, for transitive constructions, subjects converged on sagittal images while, for reciprocal constructions, they converged on transversal images. The study therefore demonstrates that viewpoint is indeed a meaningful aspect of at least these linguistic expressions. Accordingly, viewpoint constitutes a potential site of intersemiotic convergence in multimodal texts which, in the context of news texts, we should expect to be realised. This is the case, for example, in (2) and (3). In (2), the ditransitive construction occurs with a co-text image in which the vector representing the action unfolds along the sagittal axis observed from the perspective of the agent. By contrast, in (3), the reciprocal construction occurs with a co-text image whose viewpoint locates the vectors representing the action along the transversal axis. Thus, in (2), language and image converge in instantiating the viewing arrangement in Figure 6a while in (3) they converge in instantiating the viewing arrangement in Figure 6c or 6d.^[6]

A further finding of Hart (2019) was that information structure in reciprocal constructions is associated iconically with contrasting left-right arrangements on the transversal axis and thus with opposite viewpoints. Thus, if we assign police to Agent 1 and protesters to Agent 2, the viewing arrangement instantiated by the L-I combination in (3) is that of Figure 6c. By contrast, in (6) below, the L-I combination presents a multimodal instantiation of the viewing arrangement in Figure 6d.^[7] On this analysis, then, the L-I combinations presented in (2), (3) and (6) converge not only in their basic schematic patterning but also in viewpoint.

(6)

Protesters waving placards clash with police at a pro-migration demonstration in central London.

MailOnline, 19 March 2016

The motivations and functions of different viewing arrangements are discussed in detail by Hart (2015). But further support for the claim that viewpoint is a meaningful feature of these linguistic expressions comes from the consistent way that language usages instantiating these constructions and images which, by the hypothesis, share a common viewpoint behave in eliciting textual effects. Hart (2018, 2019 tested the effects of transitive versus reciprocal verbs and images where the only relevant variable is viewpoint on blame assignment and perception of aggression. Reciprocal verbs and transversal images both invite more equal distribution of blame than their opposites in transitive verbs and sagittal images. Within reciprocal constructions and transversal images, participants are judged as more aggressive when they appear left in the organisation of the clause and the image. This is consistent with Casasanto’s finding that, for the majority of people (i.e., right-handers), leftward space is associated with negative valence. It may also be explained by an association between leftward space and agency in both language and image that is, in turn, based in the left-right conceptualisation of the timeline along which events unfold.

The area of language most recognised for encoding viewpoint is, of course, deixis. The viewpoint encoded by deictic expressions represents the situatedness of interlocutors in space and time. Thus, in spatial deixis, coming is used to describe movement toward a destination where at least one interlocutor is located while going is used to describe movement toward a destination where neither interlocutor is located. In the context of migration discourse, where destination countries are construed as containers (Charteris-Black 2006; Chilton 2004), deictic expressions are typically of the ‘coming’ variety and may thus be described as encoding a perspective from inside of the container. Although more background knowledge and pragmatic inferencing is required to determine the viewpoint, this vantage point interior perspective may also be encoded in co-text images. Example (7) is an interesting example in which the conceptualisation evoked by the lead paragraph is instantiated visually across two consecutive images in the text (each with their own captions). When read narratively as a visual ‘subject’ + ‘predicate’, the two images fulfil the dynamic script-like structure of the conceptualisation evoked by the linguistic expression in the lead paragraph, consisting of a motion schema + viewpoint interior perspective as represented in Figure 7. The first image depicts the agent or figure (Talmy 2000) undergoing the motion (‘children as young as six years old’). The rest of the script is fulfilled in the second image. Motion itself and the manner of motion are instantiated in the image of a train, which implies mode of transport. And although the train’s direction of travel and thus path is not made explicit, the image depicts the ground (‘the channel tunnel’) from a perspectival location inside of the UK, which coincides with the presumed deictic point of reference encoded by ‘coming’ in the linguistic co-text. When read in conjunction with the linguistic co-text, the train’s simulated direction of travel is therefore likely to be emerging out of the tunnel and into the UK. In this sense, while intersemiotically convergent in some aspects of meaning, language and image also exist in a supplementary relation.

Figure 7:

Motion schema exterior to interior + viewpoint interior perspective.

(7)

Children as young as six years old are coming through the Channel Tunnel without parents or guardians and claiming asylum in Britain, it was claimed today.

MailOnline, 13 April 2016

4.3 Windowing of attention

Viewpoint is linked to attention in so far as it defines which aspects of a scene are in the foreground and which are in the background (Chilton 2014; Talmy 2000). L-I combinations therefore often converge in both viewpoint and attentional configuration. One aspect of attentional configuration resides in what Talmy (2000) calls windowing which has its reflex in gapping.^[8] According to Talmy (2000: 257), language places a portion of a wider image schema or event-frame, representing a coherent referential situation, into the foreground of attention by virtue of explicit mention. Windowing of attention has clear analogues in visual forms of representation where images, defined by the scope of their viewing frame, necessarily only capture a part of the events they document, representing a particular snapshot in time and space. Intersemiotic convergence in the dimension of attention may therefore occur where co-text images capture the same portion of an event that gets windowed in the language usages they accompany. To illustrate this, let us consider the example in (8). The viewpoint in both the language and the image is from the perspective of the Palestinian protesters. The event they are ‘witnessing’ is an instance of what Talmy (2000: 265) calls an open path event. An open path event is an event involving an object physically in motion in the course of a period of time that is “conceptualised as an entire unity thus having a beginning and an end, and whose beginning point and ending point are at different locations in space” (Talmy 2000: 265). Path windowing occurs when language directs attention over different facets of the conceptually complete path. Talmy identifies three forms of windowing that may be imposed on different regions of the path: initial, medial and final. This is represented in Figure 8. In (8), the object in motion is tear gas canisters. Such an event-type involves a launch site and a landing site. The beginning and end points of the path, however, do not receive linguistic representation. Instead, the nominalised form ‘falling tear gas canisters’ is an example of medial path windowing which leaves the beginning and end points of the event-frame attentionally backgrounded. This is represented in Figure 7b. Likewise, in the image we see only the tear gas canisters as they are moving through the air and do not see from where they emanated or where they end up. The multimodal representation is thus convergent in distribution of attention as well as perpsective. A contrasting image, involving final path windowing, would be one capturing the moment of impact. Of course, as Talmy (2000: 266) points out, given sufficient context, we can mentally trace the whole path to reconstruct or complete it, but it is only the medial path portion that is foregrounded for attention in both the language and the image. Thus, in (8), language and image converge in instantiating the conceptualisation represented in Figure 8b.

Figure 8:

Path windowing.

(8)

Palestinian protesters look up at falling tear gas canisters near the border with Israel in the southern Gaza Strip on Tuesday.

Wall Street Journal, 14 May 2018

We also find intersemiotic convergence in path windowing in the context of immigration discourse. Immigration, similarly, is an instance of an open path event in which migrants depart a country of origin and end up at a destination country. L-I combinations can converge in directing attention over particular aspects of this process. Compare examples (9) and (10). In (9), the linguistic expression (‘crossing the Channel’) and the co-text image both involve medial path windowing. In the image we are not shown the beginning or the endpoint of the journey (though the language suggests the intended destination). By contrast, in (10), both the language and the image involve final path windowing. Final path windowing is an inherent semantic feature of the verb arrive. In the image, final path windowing occurs where migrants are shown exiting a boat at a shoreline representing the terminus of their journey. The L-I combination in (10) is thus intersemiotically convergent in instantiating the conceptualisation represented in Figure 8c. Of course, in the image, the shoreline could be any shoreline but it is specified in the co-text as being that of the UK. Conversely, while there is nothing to indicate viewpoint in the linguistic expression, the image specifies a viewpoint interior perspective, from the land side of the shoreline, which, when interpreted together, lends the linguistic expression a deictic quality. In this sense, while intersemiotically convergent in windowing of attention, language and image in (10) are supplementary in respect of viewpoint.

(9)

A group of migrants crossing The Channel in a small boat headed in the direction of Dover, Kent, on 10 August.

Independent.co.uk, 10 August 2020

(10)

The number of child migrants arriving to the UK to claim asylum has rocketed.

Immigrationnews.co.uk, 23 June 2020

4.4 Metaphor

The final dimension of construal we will address is metaphor. Metaphor is a central topic in Cognitive Linguistics and is the construal operation that has been most widely investigated in multimodal research. Metaphor, as a conceptual process that is instantiated in and evoked by metaphorical expressions, involves the apprehension of a source frame to provide a template for sense-making inside a target frame (Lakoff and Johnson 1980). Based on Lakoff and Johnson’s (1980: 153) claim that metaphor is “primarily a matter of thought and action and only derivatively a matter of language”, Forceville (2006: 381) argues that metaphors should be expected to “occur non-verbally and multimodally as well as verbally”. And indeed, many metaphors that find linguistic expression in a given discourse are also found to receive visual forms of representation elsewhere within the same discourse (e.g., El Refaie 2003; Fridolfsson 2008; Koller 2005, 2009). In multimodal metaphor theory, the focus of attention has been on articulations of metaphors where source and target frames are represented separately in different semiotic modes which work together in realising the metaphor (Forceville 2006, 2008). Comparatively less attention has been given to the co-articulation of metaphors across semiotic modes within multimodal texts. That is, to metaphor as a potential site of intersemiotic convergence (cf. O’Halloran 1999). As gesture research has shown, however, metaphors may be expressed simultaneously in more than one mode (Cienki and Müller 2008). Of course, given the affordances of different modes, even when expressing the same metaphor, there are likely to be differences between them. For example, in degree of specificity or in the extent to which the metaphorical interpretation is forced. In relation to the latter, we can distinguish at least two types of intersemiotic convergence in metaphor. In the first type, verbal and visual modes are fully convergent in so far as the basic underlying metaphor is recoverable from each mode independently of the other. In the second type, images are consistent with the metaphorical framing presented verbally but their potential metaphoric reading is unlikely to be realised in the absence of a verbal co-text metaphor. For example, images of migrants contained by fences such as found in (1) have a potential metaphorical reading in which migrants are construed as caged animals. However, such a reading is unlikely without a verbal instantiation of the metaphor being co-present. Language may therefore serve to highlight or downplay the potential metaphoricity of images. In other words, language may have a metaphor anchoring effect (cf. Barthes 1977). Where images have a potential metaphoric reading that is consistent with a metaphoric framing presented verbally, we refer to them as frame-consistent images.^[9] It is important to note that these categories are not absolute or discrete and that the experience of images as metaphorical or not will depend on individual subjectivities. Let us consider some examples to illustrate.

(11)

The Sunday Times, 2 December 2018

In the multimodal text given as example (11), reporting on the gilets juanes (yellow vests) protests in Paris, the metaphor protest is war is strung throughout verbal portions of the text, realised directly by repeated descriptions of Paris as a ‘war zone’ as well as indirectly by the description of police ‘battling’ protesters. The same metaphor is also expressed in the image but with a greater degree of specificity. While the war frame evoked in the verbal portions of the text is a generic war frame, the frame evoked by the image represents a specific historic event, namely the Second French Revolution of July 1830. The metaphoricity of the image is achieved via the intertextual reference it makes to Eugène Delacroix’s famous painting Liberty Leading the People which, produced to commemorate the events of July 1830, shows Marianne, a national symbol of the French Republic, personifying the Goddess of Liberty. The intertextually referenced image provides an access point to the frame it instantiates which is then brought to bear in the interpretation of the current image. Of course, where intertextuality is a vehicle for metaphor, the metaphorical interpretation of the image depends on the reader having the requisite background knowledge to recover the intertextual reference (Werner 2004).

In (11), the metaphoric reading of the image is possible based on the image alone. As an instance where a potential metaphoric construal is dependent on linguistic co-text, reconsider example (5). Since ‘column of X’ is the designation for a group of soldiers and, by metaphorical extension, a group of ants, the caption in (5) may be analysed as expressing one of two metaphors: immigrants are soldiers or immigrants are insects. Both are well documented metaphors in media discourses of immigration (e.g., Hart 2010; Santa Ana 1999). The image in (5) is potentially consistent with the imagery of both of these metaphors. While in (11) it is the content of the image that resembles another specific image, in (5) the image bears a structural resemblance ^[10] to typical images of invading armies and insects which may thus be interdiscursively rather than intertextually brought to bear in interpreting the current image. The metaphoric construal of both language and image in (5) is therefore likely to be determined by metaphorical expressions in other prominent regions of the text which perform a frame-setting function. Here we find that the headline of the text in which (5) is embedded contains a militarising metaphor:

Thus, the L-I combinations in both (11) and (5) present instances of intersemiotic convergence in the dimension of metaphor with war providing the source frame in each example. The images in each case, though, do not fit neatly within Forceville’s (2008) classification of pictorial metaphor as contextual or hybrid. Rather, both instances represent a third type of pictorial metaphor (holistic) where it is the image as a whole that is reminiscent of another iconic image or type of image that belongs to a different context (Hart 2017).