Mistral is a Small Language Model originally released in September 2023.
Notable Architectural Features
Sliding Window Attention
A trade off in the application of self-attention that allows the model to use less memory by reducing (but not removing) a token’s influence on other tokens that are further away from it in the input stream.
