Mamba (deep learning architecture)

Mamba is a deep learning architecture for sequence modeling, developed by Albert Gu of Carnegie Mellon University and Tri Dao of Princeton University to address limits of transformer models when they process long sequences. It builds on the Structured State Space sequence model, called S4, which combines continuous-time, recurrent and convolutional views to model long dependencies efficiently. Mamba adds a selection mechanism that makes the state space parameters depend on the input, so the model can focus on relevant information and filter out the rest, which moves it from a time-invariant to a time-varying design. To keep that efficient it uses a hardware-aware algorithm built on kernel fusion, parallel scan and recomputation, which avoids materialising expanded states in memory. It also folds the state space block and an MLP block into one homogeneous layer. The architecture is applied to language, audio and genomics, and a variant called MoE-Mamba combines it with a mixture of experts layer.