We study a game between liquidity provider (LP) and liquidity taker agents interacting in an over-the-counter market, for which the typical example is foreign exchange. We show how a suitable design of parameterized families of reward functions coupled with shared policy learning constitutes an efficient solution to this problem. By playing against each other, our deep-reinforcement-learning-driven agents learn emergent behaviors relative to a wide spectrum of objectives encompassing profit-and-loss, optimal execution, and market share. In particular, we find that LPs naturally learn to balance hedging and skewing, where skewing refers to setting their buy and sell prices asymmetrically as a function of their inventory. We further introduce a novel RL-based calibration algorithm, which we found performed well at imposing constraints on the game equilibrium. On the theoretical side, we are able to show convergence rates for our multi-agent policy gradient algorithm under a transitivity assumption, closely related to generalized ordinal potential games.

中文翻译：

---

走向多智能体强化学习驱动的场外市场模拟

我们研究了场外市场中流动性提供者（LP）和流动性接受者之间的博弈，典型的例子就是外汇。我们展示了参数化奖励函数族的适当设计与共享策略学习如何构成该问题的有效解决方案。通过相互竞争，我们的深度强化学习驱动的代理可以学习与各种目标相关的紧急行为，包括损益、最佳执行和市场份额。特别是，我们发现有限合伙人自然会学会平衡对冲和倾斜，其中倾斜是指根据库存的函数不对称地设定买入和卖出价格。我们进一步引入了一种新颖的基于强化学习的校准算法，我们发现该算法在对游戏平衡施加约束方面表现良好。在理论方面，我们能够在传递性假设下显示多智能体策略梯度算法的收敛率，与广义序数潜在博弈密切相关。