Modeling long-range dependencies in sequential data is a crucial step in sequence learning. A recently developed model, the Structured State Space (S4), demonstrated significant effectiveness in modeling long-range sequences. However, It is unclear whether the success of S4 can be attributed to its intricate parameterization and HiPPO initialization or simply due to State Space Models (SSMs). To further investigate the potential of the deep SSMs, we start with exponential smoothing (ETS), a simple SSM, and propose a stacked architecture by directly incorporating it into an element-wise MLP. We augment simple ETS with additional parameters and complex field to reduce the inductive bias. Despite increasing less than 1\% of parameters of element-wise MLP, our models achieve comparable results to S4 on the LRA benchmark.

中文翻译：

---

将指数平滑纳入 MLP：简单但有效的序列模型

对序列数据中的远程依赖关系进行建模是序列学习中的关键步骤。最近开发的模型，结构化状态空间（S4），在模拟长程序列方面表现出显着的有效性。然而，目前尚不清楚 S4 的成功是归因于其复杂的参数化和 HiPPO 初始化，还是仅仅归因于状态空间模型 (SSM)。为了进一步研究深度 SSM 的潜力，我们从指数平滑 (ETS)（一种简单的 SSM）开始，并通过将其直接合并到元素级 MLP 中来提出堆叠架构。我们用额外的参数和复杂的场来增强简单的 ETS，以减少归纳偏差。尽管增加了不到 1% 的逐元素 MLP 参数，但我们的模型在 LRA 基准上取得了与 S4 相当的结果。