Last reviewed
Apr 27, 2026
Sources
No citations yet
Review status
Needs citations
Revision
v3 ยท 4,305 words
Add missing citations, update stale details, or suggest a clearer explanation.
See also: Machine learning terms, TensorFlow, Computational graph, Tensor
In the context of machine learning, a node is a fundamental unit within a computational graph, which is a directed, acyclic graph (DAG) used to represent the flow of data and operations in a TensorFlow model. A TensorFlow graph is composed of multiple nodes, each representing an operation or a variable, which are connected by edges representing the flow of data between these nodes. The TensorFlow graph is a core component of the TensorFlow library, which is an open-source software library for numerical computation and machine learning, developed by the Google Brain team and first released in November 2015 under the Apache 2.0 license.
In the TensorFlow Python API, a node is concretely represented by an instance of the tf.Operation class (often abbreviated as an op), and the data that flows along an edge between two nodes is represented by an instance of the tf.Tensor class. A whole graph is represented by an instance of tf.Graph. Together, these three abstractions form what the official TensorFlow documentation calls a dataflow graph, the data structure that describes a TensorFlow computation independently of the language used to construct it.[^tf-intro]
The node concept is central to TensorFlow because the graph it lives inside is what makes the framework portable, parallelizable, and deployable. A graph can be saved to disk as a GraphDef protocol buffer, restored on a server, sent to a mobile phone, or compiled to GPU and TPU machine code by XLA, all without needing the original Python program. Understanding what a node is, what it stores, and how it relates to its neighbors is therefore a prerequisite for understanding how TensorFlow trains, serves, and optimizes models.
A computational graph in mathematics and computer science is a directed graph whose vertices represent operations and whose edges represent operands. Every modern deep learning framework, including TensorFlow, PyTorch, JAX, and MXNet, is built around some version of this idea. The graph captures the structure of a calculation as a static data structure that can be analyzed, rewritten, scheduled, and differentiated.
A TensorFlow graph is more specifically a dataflow graph, a model that comes from the dataflow architectures studied by Jack Dennis at MIT in the 1970s. In a dataflow graph, an operation fires whenever all of its inputs are available. This makes the graph naturally parallel: any two nodes whose dependencies are satisfied can run at the same time, possibly on different devices. The TensorFlow whitepaper by Abadi and colleagues at Google describes this design explicitly, noting that TensorFlow "uses dataflow graphs to represent computation, shared state, and the operations that mutate that state" and maps the nodes of those graphs across many machines and across many devices within a single machine.[^abadi-osdi]
The basic mathematical idea is simple. Suppose you want to compute y = relu(matmul(x, W) + b). As a dataflow graph this becomes four nodes (a MatMul, an Add, and a Relu, plus the input x, the variable W, and the variable b as additional source nodes) connected by edges that carry intermediate tensors. The graph encodes both the values that flow and the order in which they have to be produced. From this single description, TensorFlow can derive the forward pass, the backward pass (by adding gradient nodes), the device placement, and the runtime schedule.
In the TensorFlow runtime, every node is encoded as a NodeDef protocol buffer message defined in tensorflow/core/framework/node_def.proto. A NodeDef is small and self-describing, which is what allows TensorFlow graphs to be serialized and restored without the original Python program.[^tool-dev] At runtime each node is also wrapped by a tf.Operation Python object that exposes the same information through a higher-level API.
The table below summarizes the most important fields exposed by tf.Operation and the corresponding NodeDef proto.
| Field | Type | Meaning |
|---|---|---|
name | string | Unique identifier for this node within its graph (e.g. dense_1/MatMul). |
op (or type) | string | The kind of operation, registered in the TensorFlow op registry (e.g. MatMul, Conv2D, Relu, Identity). |
inputs | list of Tensor | Tensors consumed by this node. Each entry references an output of another node. |
outputs | list of Tensor | Tensors produced by this node. Other nodes can take these as inputs. |
attr | map of name to AttrValue | Static attributes such as dtype, padding, strides, kernel shape, etc. |
device | string | Device specification, e.g. /job:worker/replica:0/task:0/device:GPU:0. |
control_inputs | list of Operation | Other nodes that must finish executing before this node fires, even though no tensor flows between them. |
The op field is the bridge between the abstract description of the graph and the concrete kernel that will run on a device. When a graph is loaded, the op name is looked up in a registry (for example, MatMul resolves to a CUDA kernel on GPU, an Eigen kernel on CPU, and a TPU implementation when running on TPU). This indirection is what allows the same GraphDef to run on a phone, a server, and an accelerator.
Nodes can have zero inputs (sources, such as Const, Variable, or Placeholder), zero outputs (sinks, such as NoOp or Save), or any number of each. They can also have control edges, which are dashed edges in TensorBoard that represent ordering constraints with no data attached. Control edges are how variable assignments, queue operations, and side effects are sequenced in a graph that otherwise only sees pure data dependencies.
TensorFlow ships with several thousand built-in op kinds, but they fall into a small number of categories that every user encounters.
| Category | Examples | Role |
|---|---|---|
| Arithmetic ops | Add, Sub, Mul, Div, MatMul, Conv2D | Perform numerical computation on input tensors and produce output tensors. |
| Activation and reduction | Relu, Sigmoid, Softmax, ReduceSum, ReduceMean | Element-wise nonlinearities and reductions used in neural networks. |
| Variables | Variable, VariableV2, VarHandleOp, ReadVariableOp, AssignVariableOp | Hold mutable state that persists across calls (weights, biases, optimizer slots). |
| Constants | Const | Immutable tensors baked into the graph. |
| Placeholders | Placeholder, PlaceholderWithDefault | Symbolic input slots. Used to feed external data in TF 1.x style programs. |
| Control flow | Switch, Merge, Enter, Exit, NextIteration, LoopCond, tf.cond, tf.while_loop | Implement conditionals and loops within a graph. AutoGraph translates Python if and while into these ops. |
| Queue and dataset ops | FIFOQueue, IteratorGetNext, MapDataset, BatchDataset | Stage and pipeline input data for tf.data input pipelines. |
| I/O and serialization | RestoreV2, SaveV2, ReadFile, DecodeJpeg | Read and write tensors from disk or other sources. |
| Communication | Send, Recv, CollectiveReduce, AllReduce | Move tensors between devices and between machines in distributed runs. |
| No-ops and grouping | NoOp, Identity, Group | Used by the runtime to coordinate execution and as anchors for control dependencies. |
Variables deserve a special mention because they are the only nodes that carry mutable state. A Variable op produces a tensor whose value can be modified in place by AssignVariableOp and read by ReadVariableOp. During training, the optimizer inserts these read and assign ops next to each variable so that gradient updates can be applied without breaking the dataflow contract.
The meaning of a node is the same across TensorFlow versions, but the way users interact with the graph has changed considerably between TensorFlow 1.x and TensorFlow 2.x.
In TensorFlow 1.x, the graph was the primary interface. A typical program had two phases: a construction phase, where Python code added nodes to the default graph by calling functions like tf.matmul, tf.constant, or tf.placeholder, and an execution phase, where the user opened a tf.Session and called session.run(fetches, feed_dict=...) to actually compute tensor values. Nothing numeric happened in the construction phase; calling tf.matmul(a, b) simply added a MatMul node to the graph and returned a symbolic tf.Tensor handle. Inputs were supplied through tf.placeholder nodes whose values were filled in by the feed_dict argument at session run time.
This model is sometimes called define-and-run or static graph programming. It is fast and very deployable because the whole computation is known up front, but it is also awkward: control flow has to be written using ops like tf.cond and tf.while_loop, debugging requires special graph-aware tools, and intermediate tensor values are not visible without an explicit session.run call.
In TensorFlow 2.x, released in September 2019, eager execution became the default. Operations now run immediately when the corresponding Python function is called, in the same way as NumPy or PyTorch. Calling tf.matmul(a, b) with concrete tensors returns a concrete tensor, not a graph node. Sessions and placeholders are no longer needed in regular user code; the equivalent classes live in tf.compat.v1 for backward compatibility.
Graphs did not disappear in TensorFlow 2. They were instead pushed under a single new abstraction: tf.function. When a Python function is decorated with @tf.function, TensorFlow runs the function once with tracing turned on, records every TensorFlow op it calls into a fresh tf.Graph, and afterwards executes that graph instead of the Python code on subsequent calls. This gives users the interactive feel of eager execution during development and the performance and portability of a static graph in production.[^tf-intro]
The table below contrasts the two execution modes.
| Aspect | TF 1.x graph mode | TF 2.x eager mode | TF 2.x with tf.function |
|---|---|---|---|
| Default behavior | Graph construction | Immediate execution | Trace once, run as a graph |
| Interface for inputs | tf.placeholder + feed_dict | Regular Python arguments | Regular Python arguments |
| Driver | tf.Session.run() | Python interpreter | Cached ConcreteFunction |
| Control flow | tf.cond, tf.while_loop | Native Python | AutoGraph rewrites Python into graph ops |
| Debugging | Graph-aware tools, tf.Print | print, pdb, breakpoints | Eager during dev via tf.config.run_functions_eagerly(True) |
| Typical use | Production training and serving | Research and prototyping | Production paths in modern code |
tf.function is the modern way to build TensorFlow graphs. It takes an ordinary Python function and turns it into a callable object (a PolymorphicFunction) that lazily compiles one or more ConcreteFunction objects, each backed by a tf.Graph. The first time the function is called with a new combination of input shapes and dtypes, TensorFlow performs a process called tracing: it runs the Python body once, recording every TensorFlow op it issues and discarding the rest of the Python side effects. The resulting graph is then executed natively by the TensorFlow runtime on every subsequent call with the same input signature.[^tf-function]
A simple example illustrates the idea.
import tensorflow as tf
@tf.function
def linear(x, W, b):
return tf.nn.relu(tf.matmul(x, W) + b)
W = tf.Variable(tf.random.normal((4, 3)))
b = tf.Variable(tf.zeros((3,)))
x = tf.random.normal((2, 4))
y = linear(x, W, b) # First call traces a graph with 5 nodes.
y = linear(x, W, b) # Second call runs the cached graph directly.
During the first call, TensorFlow records a graph that contains a MatMul node, an Add node, a Relu node, the variable read ops for W and b, and the implicit input placeholders for x. On subsequent calls with tensors of the same shape and dtype, this graph runs directly with no Python overhead.
Because tracing only happens during the first call, plain Python print statements run only once, while tf.print runs every time. This is a common debugging gotcha. Adding a print("tracing") line to a tf.function is in fact the standard way to detect unwanted retracing.
AutoGraph is the library inside tf.function that allows ordinary Python control flow to participate in the graph. AutoGraph transforms a subset of Python code into graph-compatible TensorFlow ops at trace time. if statements that branch on a tf.Tensor become tf.cond nodes; while and for loops over tensors become tf.while_loop nodes; break, continue, and return get translated into the corresponding loop control signals.[^autograph] You can inspect the rewritten code with tf.autograph.to_code(my_function).
A short example shows AutoGraph in action.
@tf.function
def relu_like(x):
if tf.reduce_sum(x) > 0:
return x
else:
return tf.zeros_like(x)
AutoGraph rewrites the if into a tf.cond so that the conditional becomes part of the dataflow graph rather than a Python-side branch. Loops over tf.range are similarly rewritten into tf.while_loop. Loops over plain Python range, by contrast, are unrolled at trace time, which is why mixing the two in performance-critical code can produce surprises.
Because every node is a first-class object, you can walk a graph and print its nodes from Python. Inside a tf.function, the graph for a particular concrete signature is reachable through get_concrete_function.
import tensorflow as tf
@tf.function
def f(x, y):
return tf.nn.relu(tf.matmul(x, y))
concrete = f.get_concrete_function(
tf.TensorSpec((None, 4), tf.float32),
tf.TensorSpec((4, 3), tf.float32),
)
for op in concrete.graph.get_operations():
print(op.name, op.type,
[t.name for t in op.inputs],
[t.shape for t in op.outputs])
For a typical run this prints the input placeholder ops, a MatMul, a Relu, an Identity that exposes the function's return value, and a small number of bookkeeping nodes inserted by the runtime. The op.name, op.type, op.inputs, op.outputs, op.control_inputs, and op.device properties together describe everything the runtime needs to execute that node.
A tf.Graph is the in-memory representation of a TensorFlow graph. To serialize one, TensorFlow uses GraphDef, a protocol buffer message defined in tensorflow/core/framework/graph.proto. A GraphDef is essentially a list of NodeDef messages plus a versions field that records which set of op semantics the graph was built against.[^tool-dev]
The usual way to obtain a GraphDef from Python is graph.as_graph_def(). The result can be written to disk in either binary protobuf format (typically with the suffix .pb) or text protobuf format (.pbtxt), and reloaded later with tf.graph_util.import_graph_def. Because protocol buffers are language-neutral, the same GraphDef can be consumed from C++, Java, Go, JavaScript, Swift, and several other languages.
On top of GraphDef, TensorFlow defines several richer formats for full models:
| Format | Container | Purpose |
|---|---|---|
GraphDef | graph.proto | Bare list of nodes plus op versions. No variable values. |
MetaGraphDef | meta_graph.proto | A GraphDef plus signatures, collections, and variable metadata. |
SavedModel | Directory with saved_model.pb and variables/ subdirectory | One or more MetaGraphDefs plus checkpoint files. The recommended format for serving and deployment. |
| Frozen graph | Single .pb file | A GraphDef where variables have been replaced by Const nodes containing the trained values. Used for legacy mobile and TF Lite pipelines. |
| Checkpoint | .ckpt, .index, .data-* | Variable values only, no graph structure. Used during training. |
A SavedModel is the canonical way to ship a TensorFlow model for inference. It bundles the graph (or several graphs, one per tag_set), the values of every variable in a separate variables/ directory, and a list of named signatures that map input and output tensor names to friendly keys. Tools such as TensorFlow Serving, TensorFlow Lite, and TensorFlow.js all consume SavedModel directly, which is why TF 2.x users almost never have to touch GraphDef by hand.
TensorBoard is the standard tool for inspecting TensorFlow graphs visually. Its Graphs dashboard reads the GraphDef recorded alongside training logs and renders it as an interactive diagram with zoom, pan, and click-to-expand support. Op nodes appear as ellipses, namespaces appear as rounded rectangles that group related ops, tensor edges are drawn as solid arrows whose thickness reflects tensor size, and control dependencies are drawn as dashed arrows.[^tb-graphs]
TensorBoard provides two complementary views:
| View | Source | What it shows |
|---|---|---|
| Op-level graph | GraphDef written by tf.summary.trace_on or tf.summary.graph | Every tf.Operation in the graph, including bookkeeping ops added by the optimizer or the input pipeline. |
| Conceptual graph | Keras model summary | The high-level layer structure of a tf.keras.Model, with one node per layer rather than one node per op. |
For a tf.function, the typical recipe is to bracket the call with the summary trace API:
from datetime import datetime
import tensorflow as tf
@tf.function
def step(x, y):
return tf.nn.relu(tf.matmul(x, y))
logdir = "logs/func/" + datetime.now().strftime("%Y%m%d-%H%M%S")
writer = tf.summary.create_file_writer(logdir)
tf.summary.trace_on(graph=True, profiler=False)
step(tf.random.uniform((3, 3)), tf.random.uniform((3, 3)))
with writer.as_default():
tf.summary.trace_export(name="step", step=0)
For Keras models, passing keras.callbacks.TensorBoard(log_dir=logdir) to model.fit records the graph automatically. Once the logs are written, running tensorboard --logdir logs opens the dashboard and the Graphs tab shows the rendered model. Beyond visualization, the dashboard supports searching for a node by name, highlighting all upstream nodes that influence a selected node ("trace inputs"), and color-coding nodes by device placement, structure, or TPU compatibility.
A raw graph produced by tracing is rarely what actually runs on the hardware. Before execution, TensorFlow passes the graph through Grappler, the default graph optimizer. Grappler is a meta-optimizer that applies a sequence of rewrite passes to a tf.Graph, simplifying it and improving its memory and runtime characteristics. Grappler runs automatically whenever a tf.function is executed, and most users never see it directly.[^grappler]
The table below summarizes the most important Grappler passes.
| Optimizer | What it does |
|---|---|
| Pruning | Removes nodes whose outputs are not needed by any fetch or side effect. Runs first to shrink the graph. |
| Constant folding | Replaces subgraphs whose inputs are all constants with a single Const node holding the precomputed value. |
| Arithmetic | Removes common subexpressions, simplifies expressions like x * 1 and x + 0, and reorders associative operations. |
| Layout | Switches between NHWC and NCHW tensor layouts to match the format preferred by the target device, especially for convolution operations. |
| Remapper | Fuses common patterns, for example Conv2D + BiasAdd + Relu, into a single optimized kernel. |
| Memory | Reduces peak memory usage, including swapping tensors between GPU and host memory when needed. |
| Dependency | Removes redundant control dependencies and no-op nodes. |
| Function | Inlines small tf.function calls to expose more optimization opportunities. |
| Loop | Hoists loop-invariant work out of tf.while_loop bodies. |
| Auto mixed precision | Casts compatible operations to float16 or bfloat16 on supporting hardware. |
| Debug stripper | Removes tf.debugging and assertion ops in production builds. |
Users who want to tune Grappler can call tf.config.optimizer.set_experimental_options(...) to toggle individual passes. For example, disabling constant_folding is a common debugging step when investigating numerical mismatches between eager and graph execution.
For maximum performance, a TensorFlow graph can be compiled all the way down to native machine code by XLA, the Accelerated Linear Algebra compiler. XLA replaces TensorFlow's default per-op execution with a fused, hardware-specific binary. It targets CPUs, GPUs, and TPUs from a common intermediate representation called HLO (High Level Optimizer IR), which represents the program as functional, statically typed linear algebra operations such as Convolution, Dot, Reduce, and SelectAndScatter.[^xla]
The usual way to opt in is through tf.function:
@tf.function(jit_compile=True)
def matmul_relu(x, W, b):
return tf.nn.relu(tf.matmul(x, W) + b)
With jit_compile=True, TensorFlow takes the traced graph, identifies the largest contiguous subgraphs (clusters) whose ops are all supported by the XLA backend, and replaces each cluster with a single XlaRun node that invokes the compiled binary. Inside that binary, multiple ops are typically fused into a single GPU kernel or CPU loop, which reduces memory bandwidth use and removes per-op dispatch overhead.
XLA is the default compiler on Cloud TPUs, where every TensorFlow program is JIT-compiled before execution. On CPU and GPU, XLA is opt-in because some ops are still unsupported and because the compilation itself takes time on the first call. The compiler has since been spun out into the OpenXLA project and is shared with JAX and PyTorch/XLA, which means the same HLO-level optimizations now back several major frameworks.
PyTorch and TensorFlow took opposite stances on graphs at the start. PyTorch, released in 2016, embraced eager execution as its only mode and built its autograd engine on a dynamic, define-by-run graph that is built fresh on every forward pass and torn down after the corresponding backward pass. TensorFlow 1.x, by contrast, required users to construct a static graph up front. Both frameworks have since converged toward a hybrid model in which eager execution is the default and a compiler turns hot Python code into a graph for performance. The table below summarizes the current state.
| Aspect | TensorFlow 1.x | TensorFlow 2.x (eager + tf.function) | PyTorch (eager) | PyTorch 2.x with torch.compile |
|---|---|---|---|---|
| Default mode | Static graph | Eager | Eager | Eager |
| Graph construction | Explicit, ahead of time | Traced lazily by tf.function | Built on the fly during the forward pass | Captured by TorchDynamo at bytecode level |
| Intermediate representation | GraphDef of NodeDefs | tf.Graph plus optional XLA HLO | torch.fx.Graph (when using torch.fx); ATen-level autograd graph internally | FX graph, then torch.inductor IR or other backends |
| Backward pass | Built by adding gradient nodes to the graph | Same, computed by tf.GradientTape | Built dynamically as the forward runs | Captured together with the forward graph |
| Control flow | tf.cond, tf.while_loop | AutoGraph rewrites Python into graph ops | Native Python | Captured by Dynamo; complex flow falls back to eager |
| Compiler | Grappler, XLA via jit_compile | Grappler, XLA | None by default | Inductor, NVFuser, OpenXLA backends |
| Deployment | SavedModel, frozen graph | SavedModel, TF Lite, TF.js | TorchScript, ONNX | AOT-compiled artifact (experimental) |
| Python at runtime | Not required | Not required inside a tf.function | Required | Optional inside compiled regions |
The upshot is that the node concept exists in both frameworks. In TensorFlow it is a tf.Operation recorded in a tf.Graph. In PyTorch it is a torch.fx.Node in an fx.Graph, or a Function instance in the autograd tape, depending on which path you take. The two frameworks differ less in graph semantics than they did five years ago. They mostly differ in when the graph is built, how visible it is to the user, and which compilation toolchain ingests it.
The following self-contained snippet shows how to build a tiny TensorFlow graph, list its nodes, and write it to disk as a GraphDef. It works in TensorFlow 2.x.
import tensorflow as tf
@tf.function
def tiny_model(x):
W = tf.constant([[2.0], [3.0]])
b = tf.constant([1.0])
return tf.nn.relu(tf.matmul(x, W) + b)
# Trigger tracing and obtain the concrete function.
concrete = tiny_model.get_concrete_function(
tf.TensorSpec((None, 2), tf.float32)
)
# 1. Walk the nodes.
for op in concrete.graph.get_operations():
print(f"{op.name:30s} type={op.type:10s} inputs={len(op.inputs)} outputs={len(op.outputs)}")
# 2. Serialize the graph to GraphDef.
graph_def = concrete.graph.as_graph_def()
with open("tiny_model.pb", "wb") as f:
f.write(graph_def.SerializeToString())
print(f"Wrote graph with {len(graph_def.node)} NodeDef messages.")
The output lists the constant nodes for W and b, the placeholder for x, the MatMul, Add (or BiasAdd), Relu, and an Identity node that returns the result. Each row corresponds to one entry in the GraphDef.node repeated field. The same graph can later be loaded into another TensorFlow program, or visualized in TensorBoard by writing it through tf.summary.graph.
Imagine you are playing with building blocks, and each block represents a step in solving a math problem. Some blocks have numbers on them, and others have symbols like "+" or "x" for addition or multiplication. You can arrange these blocks in different ways to create different math problems and solve them step by step.
In machine learning, a TensorFlow graph works like these building blocks. Each block is called a node, and they can represent different things, like math operations or numbers that can change (like the scores in a game). When you connect the blocks (nodes) with lines (edges), you show the order of the steps to solve the problem. The whole setup of blocks and lines creates a flow of information, which helps the computer solve the problem and learn from it.
A helpful way to think about modern TensorFlow is that you usually write your math the way you would write any normal Python program, but if you wrap your code with @tf.function, TensorFlow secretly builds a block tower out of it the first time you run it. The next time you call your function, it does not rebuild the tower. It just pours marbles down the existing tower, which is much faster than rolling each marble by hand.