There are concerns that Artificial General Intelligence (AGI) could pose an existential threat to humanity; however, as AGI does not yet exist it is difficult to prospectively identify risks and develop requisite controls. We applied the Work Domain Analysis Broken Nodes (WDA-BN) and Event Analysis of Systemic Teamwork-Broken Links (EAST-BL) methods to identify potential risks in a future ‘envisioned world’ AGI-based uncrewed combat aerial vehicle system. The findings suggest five main categories of risk in this context: sub-optimal performance risks, goal alignment risks, super-intelligence risks, over-control risks, and enfeeblement risks. Two of these categories, goal alignment risks and super-intelligence risks, have not previously been encountered or dealt with in conventional safety management systems. Whereas most of the identified sub-optimal performance risks can be managed through existing defence design lifecycle processes, we propose that work is required to develop controls to manage the other risks identified. These include controls on AGI developers, controls within the AGI itself, and broader sociotechnical system controls.

中文翻译：

---

当明天到来时：对未来基于人工智能的无人作战飞行器系统的前瞻性风险评估

有人担心通用人工智能（AGI）可能对人类构成生存威胁；然而，由于 AGI 尚不存在，因此很难前瞻性地识别风险并制定必要的控制措施。我们应用工作域分析断裂节点（WDA-BN）和系统团队合作断裂链接事件分析（EAST-BL）方法来识别未来“设想的世界”基于 AGI 的无人作战飞行器系统中的潜在风险。研究结果表明，这种情况下存在五类主要风险：次优绩效风险、目标一致性风险、超智能风险、过度控制风险和衰弱风险。其中两个类别，即目标一致性风险和超智能风险，以前在传统安全管理系统中从未遇到或处理过。尽管大多数已识别的次优性能风险可以通过现有的国防设计生命周期流程进行管理，但我们建议需要开发控制措施来管理已识别的其他风险。其中包括对 AGI 开发人员的控制、AGI 本身内部的控制以及更广泛的社会技术系统控制。