Facial action unit (AU) recognition and optical flow estimation are two highly correlated tasks, since optical flow can provide motion information of facial muscles to facilitate AU recognition. However, most existing AU recognition methods handle the two tasks independently by offline extracting optical flow as auxiliary information or directly ignoring the use of optical flow. In this paper, we propose a novel end-to-end joint framework of AU recognition and optical flow estimation, in which the two tasks contribute to each other. Moreover, due to the lack of optical flow annotations in AU datasets, we propose to estimate optical flow in a self-supervised manner. To regularize the self-supervised estimation of optical flow, we propose an identical mapping constraint for the optical flow guided image warping process, in which the estimated optical flow between two same images is required to not change the image during warping. Experiments demonstrate that our framework (i) outperforms most of the state-of-the-art AU recognition methods on the challenging BP4D and GFT benchmarks, and (ii) also achieves competitive self-supervised optical flow estimation performance.

中文翻译：

---

联合面部动作单元识别和自监督光流估计

面部动作单元（AU）识别和光流估计是两个高度相关的任务，因为光流可以提供面部肌肉的运动信息以促进AU识别。然而，大多数现有的AU识别方法通过离线提取光流作为辅助信息或直接忽略光流的使用来独立处理这两个任务。在本文中，我们提出了一种新颖的AU识别和光流估计的端到端联合框架，其中这两个任务相互促进。此外，由于 AU 数据集中缺乏光流注释，我们建议以自我监督的方式估计光流。为了规范光流的自监督估计，我们为光流引导图像扭曲过程提出了相同的映射约束，其中要求两个相同图像之间的估计光流在扭曲期间不改变图像。实验表明，我们的框架 (i) 在具有挑战性的 BP4D 和 GFT 基准上优于大多数最先进的 AU 识别方法，(ii) 还实现了有竞争力的自监督光流估计性能。