An important challenge in reinforcement learning is to solve multimodal problems, where agents have to act in qualitatively different ways depending on the circumstances. Because multimodal problems are often too difficult to solve directly, it is often helpful to define a curriculum, which is an ordered set of subtasks that can serve as the stepping stones for solving the overall problem. Unfortunately, choosing an effective ordering for these subtasks is difficult, and a poor ordering can reduce the performance of the learning process. Here, we provide a thorough introduction and investigation of the Combinatorial Multiobjective Evolutionary Algorithm (CMOEA), which allows all combinations of subtasks to be explored simultaneously. We compare CMOEA against three algorithms that can similarly optimize on multiple subtasks simultaneously: NSGA-II, NSGA-III, and $ε$-Lexicase Selection. The algorithms are tested on a function-optimization problem with two subtasks, a simulated multimodal robot locomotion problem with six subtasks, and a simulated robot maze-navigation problem where a hundred random mazes are treated as subtasks. On these problems, CMOEA either outperforms or is competitive with the controls. As a separate contribution, we show that adding a linear combination over all objectives can improve the ability of the control algorithms to solve these multimodal problems. Lastly, we show that CMOEA can leverage auxiliary objectives more effectively than the controls on the multimodal locomotion task. In general, our experiments suggest that CMOEA is a promising algorithm for solving multimodal problems.

强化学习的一个重要挑战是解决多模态问题，其中代理必须根据情况以不同的方式采取不同的行动。由于多模态问题通常很难直接解决，因此定义课程通常很有帮助，课程是一组有序的子任务，可以作为解决整体问题的垫脚石。不幸的是，为这些子任务选择一个有效的排序是很困难的，而一个糟糕的排序会降低学习过程的性能。在这里，我们对组合多目标进化算法 (CMOEA) 进行了全面的介绍和研究，该算法允许同时探索所有子任务的组合。我们将 CMOEA 与可以同时优化多个子任务的三种算法进行比较：NSGA-II、$ε$-词汇选择。这些算法在具有两个子任务的功能优化问题、具有六个子任务的模拟多模式机器人运动问题以及将一百个随机迷宫视为子任务的模拟机器人迷宫导航问题上进行了测试。在这些问题上，CMOEA 要么表现出色，要么与控制竞争。作为一个单独的贡献，我们表明在所有目标上添加线性组合可以提高控制算法解决这些多模态问题的能力。最后，我们表明 CMOEA 可以比多模式运动任务的控制更有效地利用辅助目标。总的来说，我们的实验表明 CMOEA 是解决多模态问题的有前途的算法。