Tactile sensing plays an irreplaceable role in robotic material recognition. It enables robots to distinguish material properties such as their local geometry and textures, especially for materials like textiles. However, most tactile recognition methods can only classify known materials that have been touched and trained with tactile data, yet cannot classify unknown materials that are not trained with tactile data. To solve this problem, we propose a tactile Zero-Shot Learning framework to recognise materials when they are touched for the first time, using their visual and semantic information, without requiring tactile training samples. The biggest challenge in tactile Zero-Shot Learning is to recognise disjoint classes between training and test materials, i.e., the test materials that are not among the training ones. To bridge this gap, the visual modality, providing tactile cues from sight, and semantic attributes, giving high-level characteristics, are combined together and act as a link to expose the model to these disjoint classes. Specifically, a generative model is learnt to synthesise tactile features according to corresponding visual images and semantic embeddings, and then a classifier can be trained using the synthesised tactile features for zero-shot recognition. Extensive experiments demonstrate that our proposed multimodal generative model can achieve a high recognition accuracy of 83.06% in classifying materials that were not touched before. The robotic experiment demo and the FabricVST dataset are available at .

中文翻译：

---

用于触觉纹理识别的多模态零样本学习

触觉传感在机器人材料识别中发挥着不可替代的作用。它使机器人能够区分材料属性，例如局部几何形状和纹理，尤其是纺织品等材料。然而，大多数触觉识别方法只能对已被触摸并经过触觉数据训练的已知材料进行分类，而无法对未经触觉数据训练的未知材料进行分类。为了解决这个问题，我们提出了一种触觉零样本学习框架，可以在第一次触摸材料时使用其视觉和语义信息来识别材料，而不需要触觉训练样本。触觉零样本学习的最大挑战是识别训练材料和测试材料之间的不相交类，即不属于训练材料的测试材料。为了弥合这一差距，将视觉模态（提供视觉触觉线索）和语义属性（提供高级特征）组合在一起，并充当将模型暴露给这些不相交类的链接。具体来说，学习生成模型根据相应的视觉图像和语义嵌入来合成触觉特征，然后可以使用合成的触觉特征来训练分类器以进行零样本识别。大量实验表明，我们提出的多模态生成模型在对以前未接触过的材料进行分类时可以达到 83.06% 的高识别准确率。机器人实验演示和 FabricVST 数据集可在 获取。