Neural Networks

Explore Neural Networks through related topics and the articles other pages reference most.

116 articlesSearch and filter Follow updates by RSS

Explore articles

Browse subtopics: Deep Learning

Articles that also belong to these categories. Counts cover all of Neural Networks.

Showing 61-81 of 81 articles

RWKV

RWKV (pronounced "RwaKuv") is an open-source neural network architecture that combines the parallelizable training of Transformers with the constant-time

Deep Learning Machine Learning

ReLU

ReLU, short for rectified linear unit, is an activation function that maps a real input to its positive part:

Deep Learning Machine Learning

The Rectified Linear Unit (ReLU) is the most widely used activation function in deep learning, defined mathematically as $$f(x) = \max(0, x)$$: it returns the input directly when positive and outputs zero…

Deep Learning Machine Learning

Recurrent Neural Network

A recurrent neural network (RNN) is a neural network whose computation includes a state that is passed from one step to the next.

Deep Learning Machine Learning

Register tokens (Vision Transformers Need Registers)

Register tokens are a small set of extra learnable tokens added to the input sequence of a Vision Transformer (ViT) so the network has a dedicated place to carry out internal, image-level computation.

Deep Learning

ResNet

ResNet, short for residual network, is a family of deep convolutional neural networks introduced by Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun at CVPR 2016.

Computer Vision Deep Learning

Self-Extend

Self-Extend (written SelfExtend in the original paper) is a training-free technique that lets a pretrained large language model process inputs much longer than the context window it was trained on, with no…

Deep Learning

Self-attention

Self-attention is a neural network operation in which each position forms a data-dependent mixture of information from positions in the same input sequence or set.

Deep Learning Machine Learning

Sigmoid Function

The sigmoid function is a mathematical function that maps any real number to a value between 0 and 1 using the formula $$\sigma(x) = \frac{1}{1 + e^{-x}}$$, producing a smooth S-shaped curve.

Deep Learning Machine Learning

Soft MoE

Soft MoE (Soft Mixture of Experts) is a fully differentiable variant of the sparse mixture of experts (MoE) layer.

Deep Learning

Softmax

Softmax is a function that converts a finite vector of real-valued scores into a vector of positive numbers that sum to one.

Deep Learning Machine Learning

SwiGLU

SwiGLU (Swish-Gated Linear Unit) is the activation function used inside the feed-forward sublayer of most modern transformer large language models, including LLaMA, PaLM, Mistral, Qwen, and DeepSeek.

Deep Learning Model Architecture

Swin Transformer

The Swin Transformer (Shifted Window Transformer) is a hierarchical vision transformer architecture that computes self-attention within local, non-overlapping windows and introduces a shifted window…

Computer Vision Deep Learning

Tanh (hyperbolic tangent)

The hyperbolic tangent, written tanh, is a smooth, S-shaped activation function that maps any real number into the open interval $$(-1, 1)$$, passing through the origin so that tanh(0) = 0 .

Deep Learning

Transformers

A Transformer is a deep learning architecture, introduced by eight Google researchers in the 2017 paper "Attention Is All You Need", that uses attention as the sole mechanism for modeling relationships between…

Deep Learning Model Architecture

Translational invariance

Translational invariance (also called translation invariance or shift invariance) is the property of a function, system, or machine learning model whose output does not change when its input is translated…

Computer Vision Deep Learning

VGG

VGG (also called VGGNet) is a deep convolutional neural network architecture, introduced in 2014 by Karen Simonyan and Andrew Zisserman of the Visual Geometry Group at the University of Oxford

Computer Vision Deep Learning

Vanishing Gradient Problem

The vanishing gradient problem is a difficulty in training deep neural networks where the gradients used to update the network shrink exponentially as they are propagated backward through the layers, leaving…

Deep Learning Machine Learning

YOLO (object detection)

YOLO (You Only Look Once) is a family of object detection models that treat detection as a single regression problem, predicting bounding boxes and class probabilities directly from full images in one forward…

Computer Vision Deep Learning

tf.keras

tf.keras is the high-level deep learning API built directly into the TensorFlow machine learning framework

Deep Learning Developer Tools

xLSTM

xLSTM (Extended Long Short-Term Memory) is a recurrent neural network architecture introduced in May 2024 by Maximilian Beck, Korbinian Pöppel, Sepp Hochreiter, and collaborators at Johannes Kepler University…

Deep Learning Model Architecture