Fast Weight Attention for Continual Learning

Fast Weight Attention for Continual Learning

循环快速权重记忆和选择性状态空间模型将扩展的上下文压缩为固定大小的循环状态,使状态转换成为一种在线学习规则。我们在写后读自回归语义下研究了这一规则。 对于这里考虑的前缀预测目标,步骤 \( t \) 中显示的本地快速记忆示例是前缀对齐对 \( (\mathbf{x}_t,\mathbf{y}_t)=(\phi(\mathbf{k}_{t-1}),\mathbf{v}_t) \) 。 常见的同步关联 \( (\phi(\mathbf{k}_t),\mathbf{v}_t) \) 仍然是因果关系,但优化了不同的内部目标。 我们推导了平方误差回归和负内积目标的归一化一阶更新。回归系列包括Falcon-1(标量NLMS更新)、Falcon-2(每列扩展)和Falcon-3(滑动窗口小批量更新); Falcon-1A/Falcon-2A/Falcon-3A是相应的内积变体。我们提供了循环、掩码平行和块平行形式,以及数值稳定的正衰变重整化。代表性变体在语言建模方面仍然具有竞争力,并改进了变量数字加法的长度外推。 该框架将循环序列模型中的时间对齐、可塑性、遗忘和有界重复分开。

Recurrent fast-weight memories and selective state-space models compress an expanding context into a fixed-size recurrent state, making the state transition an online learning rule. We study this rule under read-after-write autoregressive semantics. For the prefix-prediction objective considered here, the local fast-memory example revealed at step \(t\) is the prefix-aligned pair \( (\mathbf{x}_t,\mathbf{y}_t)=(\phi(\mathbf{k}_{t-1}),\mathbf{v}_t) \). The common same-step association \( (\phi(\mathbf{k}_t),\mathbf{v}_t) \) remains causal, but optimizes a different internal objective. We derive normalized first-order updates for squared-error regression and negative inner-product objectives. The regression family comprises Falcon-1 (a scalar NLMS update), Falcon-2 (its per-column extension), and Falcon-3 (a sliding-window mini-batch update); Falcon-1A/Falcon-2A/Falcon-3A are the corresponding inner-product variants. We provide recurrent, masked-parallel, and chunk-parallel forms, together with numerically stable positive-decay renormalization. Representative variants remain competitive in language modeling and improve length extrapolation on variable-digit addition. This framework separates temporal alignment, plasticity, forgetting, and bounded rehearsal in recurrent sequence models.