User interfaces (UI) of desktop, web, and mobile applications involve a hierarchy of objects (e.g., applications, screens, view class, and other types of design objects) with multimodal (e.g., textual and visual) and positional (e.g., spatial location, sequence order, and hierarchy level) attributes. We can therefore represent a set of application UIs as a heterogeneous network with multimodal and positional attributes. Such a network not only represents how users understand the visual layout of UIs but also influences how users would interact with applications through these UIs. To model the UI semantics well for different UI annotation, search, and evaluation tasks, this article proposes the novel Heterogeneous Attention-based Multimodal Positional (HAMP) graph neural network model. HAMP combines graph neural networks with the scaled dot-product attention used in transformers to learn the embeddings of heterogeneous nodes and associated multimodal and positional attributes in a unified manner. HAMP is evaluated with classification and regression tasks conducted on three distinct real-world datasets. Our experiments demonstrate that HAMP significantly out-performs other state-of-the-art models on such tasks. To further provide interpretations of the contribution of heterogeneous network information for understanding the relationships between the UI structure and prediction tasks, we propose Adaptive HAMP (AHAMP), which adaptively learns the importance of different edges linking different UI objects. Our experiments demonstrate AHAMP’s superior performance over HAMP on a number of tasks, and its ability to provide interpretations of the contribution of multimodal and positional attributes, as well as heterogeneous network information to different tasks.

桌面、Web 和移动应用程序的用户界面 (UI) 涉及具有多模式（例如，文本和视觉）和位置（例如，空间）的对象层次结构（例如，应用程序、屏幕、视图类和其他类型的设计对象）。位置、序列顺序和层次结构级别）属性。因此，我们可以将一组应用程序 UI 表示为具有多模式和位置​​属性的异构网络。这样的网络不仅代表用户如何理解 UI 的视觉布局，而且还影响用户如何通过这些 UI 与应用程序交互。为了针对不同的 UI 注释、搜索和评估任务很好地建模 UI 语义，本文提出了新颖的基于异构注意力的多模态位置 (HAMP) 图神经网络模型。HAMP 将图神经网络与 Transformer 中使用的缩放点积注意力相结合，以统一的方式学习异构节点的嵌入以及相关的多模态和位置属性。HAMP 通过在三个不同的现实世界数据集上执行的分类和回归任务进行评估。我们的实验表明，HAMP 在此类任务上的表现明显优于其他最先进的模型。为了进一步解释异构网络信息对理解 UI 结构和预测任务之间关系的贡献，我们提出了自适应 HAMP (AHAMP)，它自适应地学习连接不同 UI 对象的不同边的重要性。我们的实验证明 AHAMP 在许多任务上均优于 HAMP，