The growing complexity of security threats and the pervasive prevalence of cyberattacks have become more apparent in the present era, and the advent of big data, characterized by its distinctive features, has introduced layers of complexity to security tasks. Intrusion Detection Systems (IDSs) constitute a crucial line of defense, but their adaptation to the realm of big data is imperative. While traditional Machine Learning (ML)-based IDSs have been pivotal in detecting malicious patterns, they are often incapable to keep pace with the demands of expansive big data networks. This paper proposes a novel decentralized Multi-Agent Reinforcement Learning (MARL)-based IDS designed to address the specific challenges posed by big data. Our solution employs decentralized cooperative MARL, securing communicative channels throughout the detection process and concurrent data preprocessing which significantly reduces the overall processing time. Furthermore, the integration of Cloud computing and Big Data streaming techniques further facilitates real-time intrusion detection as cloud’s resources allow rapid pre-process and analyse of massive data streams using powerful clusters. Likewise, Big Data streaming techniques ensure that potential intrusions are identified and addressed as they occur. Experimental results, conducted on the widely recognized NSLKDD benchmark dataset, demonstrate the superiority of our solution over other state-of-the-art approaches for big data networks, achieving an accuracy rate of 97.44%.

当今时代，安全威胁日趋复杂、网络攻击普遍存在，大数据的出现也以其鲜明的特点，给安全任务带来了层层复杂性。入侵检测系统 (IDS) 构成了至关重要的防线，但它们适应大数据领域势在必行。虽然传统的基于机器学习 (ML) 的 IDS 在检测恶意模式方面发挥着关键作用，但它们通常无法跟上广泛的大数据网络的需求。本文提出了一种新型的基于去中心化多智能体强化学习（MARL）的 IDS，旨在解决大数据带来的特定挑战。我们的解决方案采用分散式合作 MARL，在整个检测过程中保护通信通道并进行并发数据预处理，从而显着减少总体处理时间。此外，云计算和大数据流技术的集成进一步促进了实时入侵检测，因为云资源允许使用强大的集群快速预处理和分析海量数据流。同样，大数据流技术可确保潜在入侵在发生时得到识别和解决。在广泛认可的 NSLKDD 基准数据集上进行的实验结果证明了我们的解决方案相对于其他最先进的大数据网络方法的优越性，准确率达到 97.44%。