Deep cross-modal hashing has promoted the field of multi-modal retrieval due to its excellent efficiency and storage, but its vulnerability to backdoor attacks is rarely studied. Notably, current deep cross-modal hashing methods inevitably require large-scale training data, resulting in poisoned samples with imperceptible triggers that can easily be camouflaged into the training data to bury backdoors in the victim model. Nevertheless, existing backdoor attacks focus on the uni-modal vision domain, while the multi-modal gap and hash quantization weaken their attack performance. In addressing the aforementioned challenges, we undertake an invisible black-box backdoor attack against deep cross-modal hashing retrieval in this paper. To the best of our knowledge, this is the first attempt in this research field. Specifically, we develop a flexible trigger generator to generate the attacker’s specified triggers, which learns the sample semantics of the non-poisoned modality to bridge the cross-modal attack gap. Then, we devise an input-aware injection network, which embeds the generated triggers into benign samples in the form of sample-specific stealth and realizes cross-modal semantic interaction between triggers and poisoned samples. Owing to the knowledge-agnostic of victim models, we enable any cross-modal hashing knockoff to facilitate the black-box backdoor attack and alleviate the attack weakening of hash quantization. Moreover, we propose a confusing perturbation and mask strategy to induce the high-performance victim models to focus on imperceptible triggers in poisoned samples. Extensive experiments on benchmark datasets demonstrate that our method has a state-of-the-art attack performance against deep cross-modal hashing retrieval. Besides, we investigate the influences of transferable attacks, few-shot poisoning, multi-modal poisoning, perceptibility, and potential defenses on backdoor attacks. Our codes and datasets are available at https://github.com/tswang0116/IB3A.

深度跨模态哈希因其优异的效率和存储能力推动了多模态检索领域的发展，但其易受后门攻击的脆弱性却很少被研究。值得注意的是，当前的深度跨模态哈希方法不可避免地需要大规模的训练数据，从而导致中毒样本具有难以察觉的触发因素，这些触发因素很容易伪装到训练数据中，从而在受害者模型中埋下后门。然而，现有的后门攻击主要集中在单模态视觉领域，而多模态间隙和哈希量化削弱了其攻击性能。为了解决上述挑战，我们在本文中针对深度跨模态哈希检索进行了隐形黑盒后门攻击。据我们所知，这是该研究领域的首次尝试。具体来说，我们开发了一个灵活的触发器生成器来生成攻击者指定的触发器，它学习非中毒模态的样本语义以弥合跨模态攻击差距。然后，我们设计了一个输入感知注入网络，它将生成的触发器以样本特定隐形的形式嵌入到良性样本中，并实现触发器和中毒样本之间的跨模态语义交互。由于受害者模型的知识不可知性，我们允许任何跨模态哈希仿冒，以促进黑盒后门攻击并减轻哈希量化的攻击削弱。此外，我们提出了一种令人困惑的扰动和屏蔽策略，以诱导高性能受害者模型专注于中毒样本中难以察觉的触发因素。对基准数据集的大量实验表明，我们的方法对于深度跨模态哈希检索具有最先进的攻击性能。此外，我们还研究了可转移攻击、少发中毒、多模式中毒、可感知性和潜在防御对后门攻击的影响。我们的代码和数据集可在 https://github.com/tswang0116/IB3A 获取。