Bidirectional
Bidirectional describes a sequence model in which the representation at every position depends on the entire input sequence, both the tokens before it and the tokens after it
Model Architecture
Explore Model Architecture through related topics and the articles other pages reference most.
Explore articles
Browse subtopics: Neural Networks
Articles that also belong to these categories. Counts cover all of Model Architecture.
All articles (65)Deep Learning (21)Transformer Models (16)Neural Networks (15)Large Language Models (14)Machine Learning (13)AI Models (6)Microsoft (6)Computer Vision (5)AI Inference (4)Meta AI (4)AI Companies (3)AI Infrastructure (2)Chinese AI (2)Google (2)Mixture of Experts (2)Natural Language Processing (2)Open Source AI (2)AI History (1)Artificial Intelligence (1)Developer Tools (1)Diffusion Models (1)Google DeepMind (1)Image Generation (1)Information Retrieval (1)MLOps (1)Multimodal AI (1)NVIDIA (1)People (1)Training & Optimization (1)
Showing 1-15 of 15 articles
Depthwise separable convolutional neural network (sepCNN)
A depthwise separable convolutional neural network (often abbreviated sepCNN) is a convolutional neural network that replaces standard convolution layers with depthwise separable convolutions.
Long Short-Term Memory (LSTM)
Long short-term memory (LSTM) is a gated recurrent neural network architecture for processing sequences.
Multi-Head Self-Attention
Multi-head self-attention is the core sequence-mixing mechanism of the Transformer architecture: it runs several scaled dot-product attention operations ("heads") in parallel over different learned projections…
Multi-head Latent Attention
Multi-head Latent Attention (MLA) is an attention mechanism for transformer models that achieves a 93.3% reduction in key-value cache size while maintaining or exceeding the performance of traditional…
Multilayer Perceptron
A multilayer perceptron (MLP) is a feedforward neural network built from layers of fully connected units: an input layer, one or more hidden layers, and an output layer
Node (neural network)
A node in a neural network is the basic computational element, an artificial neuron, that receives one or more inputs, multiplies each by a learned weight, sums them, adds a bias, and passes the result through…
Perceiver
Perceiver is a family of general-purpose neural network architectures from DeepMind built around attention and a small latent bottleneck.
RWKV
RWKV (pronounced "RwaKuv") is an open-source neural network architecture that combines the parallelizable training of Transformers with the constant-time
Recurrent Neural Network
A recurrent neural network (RNN) is a neural network whose computation includes a state that is passed from one step to the next.
Self-attention
Self-attention is a neural network operation in which each position forms a data-dependent mixture of information from positions in the same input sequence or set.
SwiGLU
SwiGLU (Swish-Gated Linear Unit) is the activation function used inside the feed-forward sublayer of most modern transformer large language models, including LLaMA, PaLM, Mistral, Qwen, and DeepSeek.
Titans (neural architecture)
Titans is a family of neural sequence-modeling architectures from Google Research that combines an attention-based "short-term memory" with a deep neural "long-term memory" module that updates its own…
Transformers
A Transformer is a deep learning architecture, introduced by eight Google researchers in the 2017 paper "Attention Is All You Need", that uses attention as the sole mechanism for modeling relationships between…
xLSTM
xLSTM (Extended Long Short-Term Memory) is a recurrent neural network architecture introduced in May 2024 by Maximilian Beck, Korbinian Pöppel, Sepp Hochreiter, and collaborators at Johannes Kepler University…