Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models,arXiv - CS - Operating Systems

当前位置： X-MOL 学术 › arXiv.cs.OS › 论文详情

Our official English website, www.x-mol.net, welcomes your feedback! (Note: you will need to create a separate account there.)

Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models
arXiv - CS - Operating Systems Pub Date : 2024-02-10 , DOI: arxiv-2402.07033
Keisuke Kamahori, Yile Gu, Kan Zhu, Baris Kasikci

Large Language Models (LLMs) based on Mixture-of-Experts (MoE) architecture are showing promising performance on various tasks. However, running them on resource-constrained settings, where GPU memory resources are not abundant, is challenging due to huge model sizes. Existing systems that offload model weights to CPU memory suffer from the significant overhead of frequently moving data between CPU and GPU. In this paper, we propose Fiddler, a resource-efficient inference engine with CPU-GPU orchestration for MoE models. The key idea of Fiddler is to use the computation ability of the CPU to minimize the data movement between the CPU and GPU. Our evaluation shows that Fiddler can run the uncompressed Mixtral-8x7B model, which exceeds 90GB in parameters, to generate over $3$ tokens per second on a single GPU with 24GB memory, showing an order of magnitude improvement over existing methods. The code of Fiddler is publicly available at \url{https://github.com/efeslab/fiddler}

更新日期：2024-02-13

点击分享查看原文

点击收藏

阅读更多本刊最新论文

全部期刊列表>>