To improve reliability and the understanding of AI systems, there is increasing interest in the use of formal methods, e.g. model checking. Model checking tools produce a counterexample when a model does not satisfy a property. Understanding these counterexamples is critical for efficient debugging, as it allows the developer to focus on the parts of the program that caused the issue. To this end, we present a new technique that ascribes a responsibility value to each state in a transition system that does not satisfy a given safety property. The value is higher if the non-deterministic choices in a state have more power to change the outcome, given the behaviour observed in the counterexample. For this, we employ a concept from cooperative game theory -- namely general power indices, such as the Shapley value -- to compute the responsibility of the states. We present an optimistic and pessimistic version of responsibility that differ in how they treat the states that do not lie on the counterexample. We give a characterisation of optimistic responsibility that leads to an efficient algorithm for it and show computational hardness of the pessimistic version. We also present a tool to compute responsibility and show how a stochastic algorithm can be used to approximate responsibility in larger models. These methods can be deployed in the design phase, at runtime and at inspection time to gain insights on causal relations within the behavior of AI systems.

中文翻译：

---

使用通用功率指数的转型系统中的后向责任

为了提高人工智能系统的可靠性和理解，人们对使用形式化方法（例如模型检查）越来越感兴趣。当模型不满足某个属性时，模型检查工具会生成反例。了解这些反例对于高效调试至关重要，因为它使开发人员能够专注于导致问题的程序部分。为此，我们提出了一种新技术，为不满足给定安全属性的转换系统中的每个状态赋予责任值。考虑到反例中观察到的行为，如果某个状态中的非确定性选择更有能力改变结果，则该值更高。为此，我们采用合作博弈论中的一个概念——即一般实力指数，例如沙普利值——来计算国家的责任。我们提出了乐观和悲观的责任版本，其不同之处在于它们如何对待不依赖于反例的国家。我们给出了乐观责任的特征，从而产生了一种有效的算法，并显示了悲观版本的计算难度。我们还提出了一种计算责任的工具，并展示了如何使用随机算法来近似大型模型中的责任。这些方法可以在设计阶段、运行时和检查时部署，以深入了解人工智能系统行为中的因果关系。