Groq Hardware

Groq hardware is a family of artificial-intelligence accelerators and multi-chip systems built around a statically scheduled streaming architecture. Groq's 2020 architecture paper called the processor the Tensor Streaming Processor (TSP); the company later adopted Language Processing Unit (LPU) as its product name.^[1][5] The first implementation organized memory, vector, matrix, switching, and instruction-control resources as functional slices in one logical core. A compiler schedules both operations and data movement before execution instead of leaving those decisions to caches, arbiters, or dynamic instruction schedulers at run time.^[1][2]

This article covers the processor architecture, memory system, chip-to-chip fabric, cards and racks, and the compiler-facing software stack. For corporate history, leadership, financing, and commercial partnerships, see Groq. Company events do not belong in a hardware specification, and the LPU product name is not a substitute for the broader system architecture.

Two hardware generations must be kept separate. The published first-generation GroqChip design used a 14 nm process and appeared in GroqCard, GroqNode, and GroqRack products. NVIDIA introduced the Groq 3 LPU and its 256-chip LPX rack on March 16, 2026, as part of the Vera Rubin platform.^[3][7][8] Figures for one generation do not describe the other.

Scope and terminology

TSP is the architecture name used in Groq's peer-reviewed 2020 and 2022 papers. LPU is Groq's later product term.^[1][4][5] It is not an industry standard for a processor class, and it should not be read as meaning that the chip can execute only language models. The original paper evaluated deep-learning workloads, and Groq's 2022 software material described computer vision, natural-language processing, and linear-algebra support.^[1][6]

The hardware is an AI accelerator, but it differs from a conventional many-core GPU. Its functional units collectively form one statically scheduled logical core. The architecture gives the compiler explicit control over instruction order, SRAM placement, and the timing and route of tensor streams. This shifts complexity from run-time control hardware into compilation; it does not eliminate complexity from the system.

Single-chip architecture

The first TSP divided a two-dimensional mesh into columns of function-specific tiles. A functional slice contains 20 tiles, and each tile operates on 16 elements, giving a maximum vector length of 320 elements. Instructions propagate along a slice while operands and results travel across slices as streams. An operation occurs when an instruction and its scheduled operand stream meet at a functional unit.^[1]

Scroll sideways for more →

The compiler sees 144 independently controlled instruction queues. It tracks a logical time for each queue and inserts waits or repeated no-operations when needed to align producers and consumers. The instruction set exposes operation latency and instruction-to-operand skew so that scheduling can be cycle-aware. This is the basis for deterministic execution: for a fixed binary and hardware configuration, the processor follows the same instruction and data-movement schedule on each run.^[1]

Groq's TSP patent record describes the same central idea in broader terms: functional slices receive separate instruction and data flows, and the compiler arranges for them to intersect at predetermined times.^[2] A patent documents the claimed design and its history; it is not independent evidence of performance.

SRAM and rated specifications

The first-generation chip used on-die SRAM as accelerator-local working storage rather than as a hardware-managed cache. The ISCA paper reports 220 MiB of globally addressed SRAM. Groq's product brief reports 230 MB. These are effectively the same capacity in different units: 230 million bytes is about 219.3 MiB. They should not be presented as two different versions of the chip.^[1][3]

The memory slices and stream registers let operands move directly between storage and arithmetic units. This provides high on-chip bandwidth and avoids cache misses, but it also imposes a capacity constraint. Models larger than one chip's local SRAM must be partitioned across multiple chips. Host memory and system DRAM may still be present; the architectural claim is about the LPU's primary local working storage, not the absence of memory elsewhere in a server.

Groq's 2022 product brief lists the following first-generation ratings:^[3]

Scroll sideways for more →

These are vendor specifications, not measured application throughput. TOPS and TFLOPS also depend on number format and operation-counting conventions, so the INT8 and FP16 figures cannot be compared as if they measured the same work.

Multi-chip execution

Groq's 2022 ISCA paper extended the single-chip schedule across a direct chip-to-chip fabric. It described each TSP as both a processing endpoint and a forwarding element. Local SRAM remains physically distributed, but software can treat the combined capacity as a global address space. The compiler partitions a graph, schedules sends and receives, and selects routes before the program runs.^[4]

In the first-generation packaging described by the paper, one PCIe card carries one TSP, a four-rack-unit node contains eight cards, and a rack contains nine nodes, or 72 TSPs. The topology uses direct links and a software-scheduled Dragonfly organization. The paper analyzes a maximum topology of 10,440 TSPs across 145 racks; that is a design-scale analysis, not evidence that a cluster of that size was deployed.^[4]

Independent clocks drift even when nominal frequencies match. The system therefore exchanges hardware-aligned counter values, aligns program start, and uses run-time deskew operations to preserve a shared notion of time. The network also avoids dynamic backpressure and uses forward error correction instead of timing-variable link retries. These choices make communication latency more predictable, but they require the compiler to know link capacity, route, and timing in advance.^[4]

Hardware generations and systems

The public specifications below describe different products and dates. First-generation figures come from Groq's 2022 brief and ISCA system paper. Groq 3 figures come from NVIDIA's March 2026 LPX materials.^[3][4][7][8]

Scroll sideways for more →

NVIDIA describes each Groq 3 compute tray as a liquid-cooled 1U unit with eight LP30 chips, 4 GB of on-chip SRAM in aggregate, and host and fabric-expansion memory. Its proposed heterogeneous inference path assigns long-context prefill and decode attention to Rubin GPUs while using LPUs for latency-sensitive feed-forward and mixture-of-experts work during decode.^[7] That division is a platform design described by the vendor. It should not be treated as an independently measured result, and the first-generation measurements below do not establish Groq 3 performance.

The cited NVIDIA materials do not specify the LP30 process node, die size, or transistor count. Those fields should remain undisclosed here instead of being filled from unsourced or conflicting secondary reports.

Compiler and software stack

Static scheduling makes the compiler part of the architecture rather than an interchangeable convenience layer. For a graph that spans one or more processors, the compiler must choose operations, allocate SRAM, place work on functional slices, order instruction queues, partition the graph across chips, and schedule network traffic. A deterministic chip does not guarantee high utilization if that mapping is poor.^[1][4]

Groq's 2022 GroqWare brief described a stack containing Groq Compiler, a lower-level Groq API, GroqFlow import tools, the GroqView profiler and visualizer, a performance estimator, and an open-source driver and run time. At that date, the brief listed PyTorch, TensorFlow, and ONNX as supported model sources.^[6] Those statements describe the 2022 product and should not be assumed to be a complete list of current compiler or operator support.

Groq's current architecture page continues to describe a custom compiler that statically schedules compute and networking.^[5] Current hosted users generally access selected models through Groq's API, which exposes an OpenAI-compatible chat-completions endpoint.^[9] API compatibility concerns request and response formats; it does not prove that an arbitrary framework model can be compiled for the hardware.

Measured performance

Groq-authored papers report architecture evaluations, including batch-one ResNet-50 on the first chip and matrix multiplication, collectives, and BERT on multi-chip systems.^[1][4] Those results are useful for understanding the design, but they are not independent comparisons.

A June 2026 study accepted to the HPAI4S workshop at IEEE IPDPS compared one NVIDIA A100 SXM 40 GB GPU with a first-generation GroqRack containing 72 GroqCards. Both ran Llama 2 7B in FP16. The A100 used vLLM 0.12.0, while GroqRack used GroqFlow and a compiled binary supplied by Groq. The study separated prompt prefill from autoregressive decode and reported the following results:^[10]

Scroll sideways for more →

The A100 was much faster in prefill. GroqRack had about four times lower time per output token at batch one, while the A100 passed GroqRack's single-request decode throughput above batch four in the tested configurations. The result is phase- and workload-specific, not a general ranking.^[10]

The GroqRack binary in that study accepted one request and processed prompt tokens sequentially. A dedicated binary was required for each batch size, and the evaluated rack's SRAM capacity left insufficient space for a second maximum-context request. The authors excluded tokenization and detokenization, did not assess output correctness or semantic quality, and tested only Llama 2 7B with synthetic equal-length inputs and outputs. The measurements therefore should not be generalized to GroqCloud, other models, other precisions, or Groq 3.^[10]

MLPerf Inference uses a standard load generator, accuracy validation, declared scenarios, and submission checks.^[11] A review of MLCommons' public result repositories from v0.5 through v6.0 found no Groq-named submitter directory in either the Closed or Open division as of July 28, 2026.^[12] The independent GroqRack study is not an MLPerf submission, and vendor token-rate demonstrations should not be labeled MLPerf results.

Limits and comparison rules

The architecture exchanges memory capacity for bandwidth. Keeping the active working set in SRAM can reduce decode latency, but large models require many chips, so placement and communication become first-order costs. Static schedules can make execution repeatable, yet compilation, service queues, host sampling, network transport, multi-tenancy, and failures outside the scheduled fabric can still vary end-to-end latency.

Peak compute, SRAM bandwidth, and tokens per second answer different questions. A fair comparison must hold constant the model, numerical format, quantization, quality target, prompt and output lengths, batch and concurrency, latency percentile, power boundary, software version, and whether prompt processing is included. It should report time to first token separately from time per output token and aggregate throughput. These controls matter because the independent study found opposite winners for prefill latency and batch-one decode latency.

Groq hardware is therefore best understood as a compiler-orchestrated accelerator system optimized for predictable, low-batch execution, not as a universal replacement for GPUs. The published evidence supports a strong decode-latency result for one first-generation configuration and detailed vendor specifications for Groq 3. It does not yet support an independent, standardized performance claim for the newer LPX system.

References

1. ^Abts, Dennis, et al. "Think Fast: A Tensor Streaming Processor (TSP) for Accelerating Deep Learning Workloads." 2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture, pp. 145-158. doi.org/...ISCA45697.2020.00023
2. ^United States Patent and Trademark Office. "Tensor streaming processor architecture." US11360934B1, published June 14, 2022. patents.google.com/...en
3. ^Groq. "GroqChip Processor Product Brief v1.5." 2022. cdn.sanity.io/...f76a4a5c16ac63cc064058460376b.pdf
4. ^Abts, Dennis, et al. "A Software-defined Tensor Streaming Multiprocessor for Large-scale Machine Learning." Proceedings of the 49th Annual International Symposium on Computer Architecture, 2022, pp. 567-580. doi.org/...3470496.3527405
5. ^Groq. "Introducing the LPU." Accessed July 28, 2026. groq.com/lpu-architecture
6. ^Groq. "GroqWare Suite Product Brief v1.5." 2022. cdn.sanity.io/...d18a83eada79132db1b8a956c4725.pdf
7. ^Aubrey, Kyle, and Farshad Ghodsian. "Inside NVIDIA Groq 3 LPX: The Low-Latency Inference Accelerator for the NVIDIA Vera Rubin Platform." NVIDIA Technical Blog, March 16, 2026. developer.nvidia.com/...nvidia-vera-rubin-platform
8. ^NVIDIA. "NVIDIA Groq 3 LPX." Accessed July 28, 2026. nvidia.com/...lpx
9. ^Groq. "API Reference." GroqDocs, accessed July 28, 2026. console.groq.com/...api-reference
10. ^Usami, Shun, Venkatram Vishwanath, and E. Wes Bethel. "Prefill/Decode-Aware Evaluation of LLM Inference on Emerging AI Accelerators." arXiv:2606.17104, accepted to HPAI4S at IEEE IPDPS 2026. arxiv.org/...2606.17104
11. ^MLCommons. "MLPerf Inference Submission Guide." Accessed July 28, 2026. docs.mlcommons.org/...submission
12. ^MLCommons. "Repositories matching inference_results." GitHub, accessed July 28, 2026. github.com/...repositories