About
    • AMMDI is an open-notebook hypertext writing experiment, authored by Mike Travers aka mtraven. It's a work in progress and some parts are more polished than others. Comments welcome! More.
Search
Full
Incoming links
from transformer blocks
  • Recurrent networks (including LSTM) are state of the art (in 2017). This proposes flushing them and replacing with nothing but attention, specifically the Transformer.
Twin Pages

TwinPages: RalfBarkow | WebSeitzWiki

attention

17 Jan 2026 - 17 Jan 2026
Open in Logseq
    • Attention (machine learning) - Wikipedia)
      • Attention-like mechanisms were introduced in the 1990s under names like multiplicative modules, sigma pi units, and hyper-networks. Its flexibility comes from its role as "soft weights" that can change during runtime, in contrast to standard weights that must remain fixed at runtime.
      • hard weights from training (backward), soft wweights from use (forward). Weird terminology but OK.