Accurate throwing skills can expand the pick-and-place ability of a manipulator, which is significant but challenging in the field of robotics. Most existing robotic throwing methods neglect the mass of an object and air drag, not to mention the effect of a grasp on the subsequent throw, resulting in inaccurate throws. In this regard, we propose collaborative grasping and throwing learning (CGTL). It consists of a grasp agent with a grasping network (G-Net), a throw agent with a learning-based throw reference (LTR), and a multi-modal throw compensator network (MTC-Net). First, G-Net generates multi-channel grasp affordances for inferring grasps. Subsequently, LTR predicts a throw velocity reference by exploiting an air resistance estimation network (ARE-Net) and a projectile equation considering air drag. Meanwhile, MTC-Net uses multi-modal data to predict the compensation for the throwing velocity reference. Moreover, CGTL takes throwing performances into the reward of the grasp agent and the grasp affordances into the throw agent's observation to facilitate more accurate throwing. Finally, extensive experiments show that our CGTL outperforms its peers regarding throwing accuracy, especially when throwing different objects into new positions.

中文翻译：

---

用于协作机器人抓取投掷的自监督多模态学习

准确的投掷技能可以扩展机械臂的拾取和放置能力，这在机器人领域意义重大但具有挑战性。现有的机器人投掷方法大多忽略了物体的质量和空气阻力，更不用说抓握对后续投掷的影响，导致投掷不准确。在这方面，我们提出协作抓取和投掷学习（CGTL）。它由具有抓取网络（G-Net）的抓取代理、具有基于学习的投掷参考（LTR）的投掷代理和多模态投掷补偿器网络（MTC-Net）组成。首先，G-Net 生成用于推断抓取的多通道抓取可供性。随后，LTR 通过利用空气阻力估计网络 (ARE-Net) 和考虑空气阻力的弹丸方程来预测投掷速度参考。同时，MTC-Net使用多模态数据来预测投掷速度参考的补偿。此外，CGTL将投掷表现纳入抓握智能体的奖励中，将抓握可供性纳入投掷智能体的观察中，以促进更准确的投掷。最后，大量实验表明，我们的 CGTL 在投掷准确性方面优于同行，尤其是在将不同物体投掷到新位置时。