))
LLM DIFF Transformer with SoftMax Subtraction
Manage episode 444738222 series 3605861
A tartalmat a Brian Carter biztosítja. Az összes podcast-tartalmat, beleértve az epizódokat, grafikákat és podcast-leírásokat, közvetlenül a Brian Carter vagy a podcast platform partnere tölti fel és biztosítja. Ha úgy gondolja, hogy valaki az Ön engedélye nélkül használja fel a szerzői joggal védett művét, kövesse az itt leírt folyamatot https://hu.player.fm/legal.
This paper presents a new architecture for large language models called DIFF Transformer. The paper argues that conventional Transformers over-allocate attention to irrelevant parts of the input, drowning out the signal needed for accurate output. DIFF Transformer tackles this issue by using a differential attention mechanism that subtracts two softmax attention maps, effectively canceling out noise and amplifying attention to relevant content. The paper presents extensive experiments demonstrating that DIFF Transformer outperforms conventional Transformers in various tasks, including language modeling, key information retrieval, hallucination mitigation, and in-context learning. This results in a more efficient model that requires fewer parameters and training data to achieve the same performance as a Transformer.
Read more: https://arxiv.org/pdf/2410.05258
65 epizódok
Megosztás
