In many grant review settings, proposals are selected for funding on the basis of summary statistics of review ratings. Challenges of this approach (including the presence of ties and unclear ordering of funding preference for proposals) could be mitigated if rankings such as top-k preferences or paired comparisons, which are local evaluations that enforce ordering across proposals, were also collected and incorporated in the analysis of review ratings. However, analyzing ratings and rankings simultaneously has not been done until recently. This paper describes a practical method for integrating rankings and scores and demonstrates its usefulness for making funding decisions in real-world applications. We first present the application of our existing joint model for rankings and ratings, the Mallows-Binomial, in obtaining an integrated score for each proposal and generating the induced preference ordering. We then apply this methodology to several theoretical “toy” examples of rating and ranking data, designed to demonstrate specific properties of the model. We then describe an innovative protocol for collecting rankings of the top-six proposals as an add-on to the typical peer review scoring procedures and provide a case study using actual peer review data to exemplify the output and how the model can appropriately resolve judges’ evaluations. For the theoretical examples, we show how the model can provide a preference order to equally rated proposals by incorporating rankings, to proposals using ratings and only partial rankings (and how they differ from a ratings-only approach) and to proposals where judges provide internally inconsistent ratings/rankings and outlier scoring. Finally, we discuss how, using real world panel data, this method can provide information about funding priority with a level of accuracy in a well-suited format for research funding decisions. A methodology is provided to collect and employ both rating and ranking data in peer review assessments of proposal submission quality, highlighting several advantages over methods relying on ratings alone. This method leverages information to most accurately distill reviewer opinion into a useful output to make an informed funding decision and is general enough to be applied to settings such as in the NIH panel review process.

中文翻译：

---

授予评审评估的新方法：评分，然后排名

在许多赠款审查环境中，根据审查评级的汇总统计来选择资助提案。如果还收集并纳入诸如 top-k 偏好或配对比较之类的排名（这些排名是强制跨提案排序的本地评估），则可以减轻这种方法的挑战（包括存在联系和提案资助偏好排序不明确）。评论评级分析。然而，直到最近才同时分析收视率和排名。本文描述了一种整合排名和分数的实用方法，并展示了其在实际应用中做出资助决策的有用性。我们首先介绍现有的排名和评级联合模型（Mallows-Binomial）在获得每个提案的综合分数并生成诱导偏好排序方面的应用。然后，我们将此方法应用于评级和排名数据的几个理论“玩具”示例，旨在演示模型的特定属性。然后，我们描述了一种用于收集前六名提案排名的创新协议，作为典型同行评审评分程序的附加组件，并提供使用实际同行评审数据的案例研究来举例说明输出以及该模型如何正确解决法官的问题评价。对于理论示例，我们展示了该模型如何通过合并排名来为同等评分的提案、使用评分和仅部分排名的提案（以及它们与仅评分方法有何不同）以及法官内部提供的提案提供优先顺序不一致的评级/排名和异常评分。最后，我们讨论如何使用现实世界的面板数据，以适合研究资助决策的格式提供有关资助优先级的信息，并具有一定的准确性。提供了一种方法来在提案提交质量的同行评审评估中收集和使用评级和排名数据，突出了与仅依赖评级的方法相比的几个优点。这种方法利用信息最准确地将审稿人的意见提炼成有用的输出，以做出明智的资助决策，并且足够通用，可以应用于 NIH 小组审评过程等环境。