Human communication is by default polysemiotic: it involves the spontaneous combination of two or more semiotic systems, the most important ones being language, gesture, and depiction. We formulate an original cognitive-semiotic framework for the analysis of polysemiosis, contrasting this with more familiar systems based on the ambiguous term “multimodality.” To be fully explicit, we developed a coding system for the analysis of polysemiotic utterances containing speech, gesture, and drawing, and implemented this in the ELAN video annotation software. We used this to analyze 23 video-recordings of sand drawing performances on Paama, Vanuatu and 20 sand stories of the Pitjantjatjara culture in Central Australia. Methodologically we used the conceptual-empirical loop of cognitive semiotics: our theoretical framework guided general considerations, such as distinguishing between the “tiers” of gesture and depiction, and the three kinds of semiotic grounds (iconic, indexical, symbolic), but the precise decisions on how to operationalize these were made only after extensive work with the material. We describe the coding system in detail and provide illustrative examples from the Paamese and Pitjantjatjara data, remarking on both similarities and differences in the polysemiosis of the two cultural practices. We conclude by summarizing the contributions of the study and point to some directions for future research.

中文翻译：

---

分析多义症：两种文化实践中的语言、手势和沙画描绘

人际交往是默认的多符号学：它涉及两个或多个的自发组合符号系统，最重要的是语言,手势， 和描写。我们为多义现象的分析制定了一个原始的认知符号学框架，并将其与基于模糊术语“多模态”的更熟悉的系统进行对比。为了完全明确，我们开发了一个编码系统，用于分析包含语音、手势和绘图的多符号话语，并在 ELAN 视频注释软件中实现了该系统。我们用它来分析瓦努阿图帕马的 23 个沙画表演视频和澳大利亚中部 Pitjantjatjara 文化的 20 个沙故事。在方法论上我们使用了概念-经验循环认知符号学：我们的理论框架指导了一般性考虑，例如区分手势和描述的“层次”，以及三种符号学基础（象征性、索引性、象征性），但如何操作这些的精确决定是做出的只有在对材料进行大量研究之后。我们详细描述了编码系统，并提供了 Paamese 和 Pitjantjatjara 数据中的说明性示例，评论了两种文化实践的多义现象的相似点和差异。最后，我们总结了这项研究的贡献，并指出了未来研究的一些方向。