Activation Function
An activation function is a nonlinear mathematical function applied to the output of each neuron in an artificial neural network, and it is what gives the network the ability to learn complex
Neural Networks
Explore Neural Networks through related topics and the articles other pages reference most.
Explore articles
Browse subtopics: Machine Learning
Articles that also belong to these categories. Counts cover all of Neural Networks.
All articles (116)Deep Learning (81)Machine Learning (58)Computer Vision (17)Model Architecture (15)Interpretability (6)AI History (5)Artificial Intelligence (4)Generative AI (4)Natural Language Processing (4)Transformer Models (4)AI Hardware (3)AI Research (3)Mathematics (3)Google DeepMind (2)Information Retrieval (2)Open Source AI (2)People (2)AI Ethics (1)AI Infrastructure (1)AI for Science (1)Developer Tools (1)Google (1)Large Language Models (1)MLOps (1)Model Evaluation (1)Programming Languages (1)Statistics (1)Training & Optimization (1)
Showing 1-58 of 58 articles
Attention
Attention is a family of operations in neural networks that computes an output by assigning data-dependent weights to a collection of representations and combining them.
Backpropagation
Backpropagation is an algorithm for computing derivatives of a scalar objective with respect to variables in a layered or otherwise composed computation.
Batch Normalization
Batch normalization (BatchNorm or BN) is a normalization method used inside some neural networks.
Bayesian Neural Network
A Bayesian neural network (BNN) is a neural network in which the weights and biases are represented as probability distributions rather than fixed point estimates.
Bias
Bias in artificial intelligence carries three distinct technical meanings: a learnable scalar parameter added inside a neuron, the systematic error component of an estimator (the "bias" in the bias-variance…
Bias (Math) or Bias Term
The bias term is a learnable additive constant b added to the weighted sum of a neuron's inputs before an activation function is applied, computed as $$z = w \cdot x + b$$.
Calibration Layer
A calibration layer is a post-prediction adjustment appended to a trained machine learning model that rescales its raw output scores or predicted probabilities so they better reflect the true likelihood of…
Candidate Sampling
Candidate sampling is a family of training-time optimization techniques used in machine learning to reduce the computational cost of models that must choose among a very large number of output classes.
Co-Adaptation
Co-adaptation in neural networks refers to a phenomenon in which different hidden units develop highly correlated behavior, becoming excessively dependent on one another rather than learning independent
Continual learning
Continual learning, also called lifelong learning or incremental learning, is a machine learning paradigm in which a model learns from a stream of tasks or data distributions over time
Convolutional Neural Network
A convolutional neural network (CNN, or ConvNet) is a neural network that uses convolution-like linear operators in at least some layers.
Convolutional Operation
The convolutional operation is a mathematical procedure that combines two functions to produce a third function expressing how the shape of one is modified by the other.
Decoder
A decoder is the component of a neural network that turns an internal, compressed, or abstract representation into a desired output, such as a translated sentence, a generated image, a reconstructed input, or…
Deep Learning
Deep learning is a subset of machine learning that uses artificial neural networks with multiple layers to automatically learn representations of data at multiple levels of abstraction
Deep Neural Network
A deep neural network (DNN) is an artificial neural network with multiple hidden layers of artificial neurons stacked between its input and output layers
Dense Layer
A dense layer, also called a fully connected (FC) layer, linear layer, or affine layer, is a layer in an artificial neural network where every input neuron is connected to every output neuron.
Discriminator
A discriminator is the neural network in a generative adversarial network (GAN) that is trained to tell real data apart from data produced by the generator
Embedding Layer
An embedding layer is a neural network component that acts as a trainable lookup table, mapping discrete integer indices (such as word IDs, user IDs, or category codes) to dense, continuous-valued vectors.
Epoch
An epoch in machine learning is one complete pass through the entire training dataset, during which every example is presented to the model exactly once to compute gradients and update parameters before the…
Exploding Gradient Problem
The exploding gradient problem is a training failure in deep neural networks where the gradients of the loss function grow exponentially large as they propagate backward through layers during backpropagation…
Feedforward Neural Network (FFN)
A feedforward neural network (FFN), also called a multilayer perceptron (MLP) when it has multiple layers, is a type of artificial neural network in which information flows in one direction only, from the…
Forget Gate
The forget gate is a sigmoid layer inside a Long Short-Term Memory (LSTM) recurrent neural network that decides, element by element, how much of the previous cell state to keep and how much to discard at each…
Full Softmax
Full softmax (also called the standard softmax or exact softmax) is the softmax computation that calculates a probability for every possible output class in a classification model
Fully Connected Layer
A fully connected layer (also called a dense layer or linear layer) is a layer in an artificial neural network in which every input value connects to every output value through a learned weight
Generative adversarial network
Generative adversarial networks (GANs) are a family of generative models trained through competition between two learned functions.
Generator
A generator is a neural network within a generative adversarial network (GAN) that learns to produce synthetic data samples from random noise.
Hidden Layer
A hidden layer is a layer of artificial neurons in a neural network that sits between the input layer and the output layer
Hopfield Network
A Hopfield network is a form of recurrent neural network that stores information as stable patterns in a system of interconnected binary units and retrieves it by content rather than by address.
Input Layer
The input layer is the first layer of a neural network: it receives the raw feature vector for each data sample and passes those values forward to the next layer, performing no learned computation of its own.
Kolmogorov-Arnold Network
A Kolmogorov-Arnold Network (KAN) is a type of neural network architecture proposed as an alternative to the traditional Multi-Layer Perceptron (MLP).
Layer
A layer is the fundamental building block of a neural network: an organized group of neurons (also called nodes or units) that together apply one mathematical transformation to their input and pass the result…
Logits
In machine learning and statistics, logits refer to the raw, unnormalized scores output by a model before they are converted into probabilities.
Mixture of Experts (MoE)
A mixture of experts (MoE) is a machine-learning architecture that contains multiple component functions called experts and a gating or routing mechanism that determines how their outputs contribute to a…
Multi-Head Self-Attention
Multi-head self-attention is the core sequence-mixing mechanism of the Transformer architecture: it runs several scaled dot-product attention operations ("heads") in parallel over different learned projections…
Multi-head Latent Attention
Multi-head Latent Attention (MLA) is an attention mechanism for transformer models that achieves a 93.3% reduction in key-value cache size while maintaining or exceeding the performance of traditional…
Neural Network
A neural network (also called an artificial neural network or ANN) is a computational model, loosely inspired by the networks of biological neurons in animal brains
Neural Tangent Kernel
The neural tangent kernel (NTK) is a kernel function built from the parameter gradients of a neural network.
Neural architecture search
Neural architecture search (NAS) is a technique for automating the design of neural network architectures.
Neuron
A neuron (also called a node or unit) is the fundamental computational element of an artificial neural network: it takes one or more numeric inputs, multiplies each by a learned weight, sums them with a bias…
Output Layer
The output layer is the final layer of a neural network: it takes the features computed by the hidden layers and converts them into the model's prediction, with its size and activation function fixed by the…
Parameter
In machine learning and statistics, a parameter is an internal variable of a model whose value is learned from data during the training process.
Perceptron
A perceptron is the earliest trainable artificial neural network: a single-layer linear model that classifies inputs into two categories by computing a weighted sum and applying a step threshold.
RWKV
RWKV (pronounced "RwaKuv") is an open-source neural network architecture that combines the parallelizable training of Transformers with the constant-time
ReLU
ReLU, short for rectified linear unit, is an activation function that maps a real input to its positive part:
Rectified Linear Unit (ReLU)
The Rectified Linear Unit (ReLU) is the most widely used activation function in deep learning, defined mathematically as $$f(x) = \max(0, x)$$: it returns the input directly when positive and outputs zero…
Recurrent Neural Network
A recurrent neural network (RNN) is a neural network whose computation includes a state that is passed from one step to the next.
Self-attention
Self-attention is a neural network operation in which each position forms a data-dependent mixture of information from positions in the same input sequence or set.
Sigmoid Function
The sigmoid function is a mathematical function that maps any real number to a value between 0 and 1 using the formula $$\sigma(x) = \frac{1}{1 + e^{-x}}$$, producing a smooth S-shaped curve.
Softmax
Softmax is a function that converts a finite vector of real-valued scores into a vector of positive numbers that sum to one.
Sparse Coding
Sparse coding is a representation learning principle in which a signal is encoded as a linear combination of a small number of elements drawn from a (usually overcomplete) dictionary of atoms.
Spatial Pooling
Spatial pooling is a downsampling operation in convolutional neural networks (CNNs) that replaces a local region of a feature map with a single summary statistic, such as the maximum or the average of the…
Two-Tower Model
The two-tower model, also known as the dual encoder, bi-encoder, or Siamese network for retrieval, is a neural network architecture that encodes a query and a candidate item with two separate sub-networks…
Vanishing Gradient Problem
The vanishing gradient problem is a difficulty in training deep neural networks where the gradients used to update the network shrink exponentially as they are propagated backward through the layers, leaving…
Weight
In machine learning and neural networks, a weight is a learnable numerical parameter that determines the strength of the connection between two neurons.
Weighted Sum
A weighted sum is a mathematical operation that combines multiple input values by multiplying each value by a corresponding weight (coefficient) and then summing the results.
Wide Model
A wide model is a type of machine learning model that uses a large number of input features, often with sparse, high-dimensional representations such as one-hot encoding and cross-product feature…
tf.keras
tf.keras is the high-level deep learning API built directly into the TensorFlow machine learning framework