About
    • AMMDI is an open-notebook hypertext writing experiment, authored by Mike Travers aka mtraven. It's a work in progress and some parts are more polished than others. Comments welcome! More.
Search
Full
Incoming links
from transformer blocks
  • Recurrent networks (including LSTM) are state of the art (in 2017). This proposes flushing them and replacing with nothing but attention mechanisms, specifically the Transformer.

attention mechanisms

30 Dec 2022 - 17 Jun 2023
Open in Logseq
    • Attention (machine learning) - Wikipedia)
      • Attention-like mechanisms were introduced in the 1990s under names like multiplicative modules, sigma pi units, and hyper-networks. Its flexibility comes from its role as "soft weights" that can change during runtime, in contrast to standard weights that must remain fixed at runtime.