d-Matrix Raptor

Raptor is the second-generation AI inference accelerator from d-Matrix, a Santa Clara semiconductor startup, and the first commercial chip built on the company's 3D stacked digital in-memory compute technology, which d-Matrix markets as 3DIMC. Raptor bonds a logic die directly on top of a custom DRAM die instead of placing memory beside compute on an interposer, an arrangement d-Matrix describes as a "two-story" package. The company presented early silicon results at Hot Chips 2026 and published the design in a paper at the 2026 International Symposium on Computer Architecture.^[1][3][5]

On September 10, 2026 d-Matrix announced a collaboration with NVIDIA that makes it an NVLink Fusion partner and brings Raptor into the NVIDIA MGX rack reference architecture. d-Matrix calls Raptor an XPU rather than a GPU or an accelerator card, reflecting its positioning as a specialized processor that sits alongside GPUs in the same rack rather than replacing them. Initial availability of Raptor XPUs integrated into an NVIDIA MGX rack is a company projection for the fourth quarter of 2027, and as of the announcement the chip had not yet taped out.^[1][2]

From Corsair to 3DIMC

Raptor follows Corsair, d-Matrix's first commercial product, which entered full production in June 2026.^[10] Corsair's design bet was on SRAM: 2 GB of on-card performance memory with compute embedded in the memory macros, backed by commodity LPDDR5X for capacity. That combination delivers very high bandwidth at very low latency, but SRAM does not scale in capacity. d-Matrix's own Hot Chips material put a bridged Corsair card pair at roughly 300 TB/s at about 1 ns latency while holding only about 4 GB, noting that a 6T SRAM cell is around ten times larger than a DRAM cell and that leakage runs into tens of watts at gigabyte scale.^[3]

The company announced 3DIMC at Hot Chips in August 2025, alongside a test chip called Pavehawk that co-founder and CTO Sudeep Bhoja described as more than two years in the making and operational in d-Matrix's labs.^[8] Bhoja returned a year later to present Raptor itself, in a session titled "3D DRAM based Accelerator for Generative Inference" that he shared with Aayush Ankit of Meta during the memory technology tutorial on August 23, 2026.^[6] In a March 2026 follow-up the company said it had validated Pavehawk across voltage and temperature corners and was measuring roughly 0.4 pJ/bit in worst-case conditions, against a stated target of 10 times better memory bandwidth and 10 times better energy efficiency than HBM4.^[9]

On November 18, 2025 d-Matrix and the Taiwanese ASIC house Alchip announced a collaboration on what both companies called the world's first 3D DRAM-based datacenter inference accelerator, with Alchip supplying ASIC design and advanced packaging expertise. That release named Raptor explicitly as the commercial debut vehicle for 3DIMC.^[7]

Architecture

Inverted 3D stacking

Raptor puts the logic die on top of the memory die, the reverse of the usual arrangement in which compute sits underneath a stack. The practical consequence is thermal: a cold plate sits directly on the compute silicon, which is where nearly all the heat is generated, while the DRAM die below doubles as the interposer and carries PCIe and die-to-die signals down through its through-silicon vias.^[4] d-Matrix presented a one-high logic-on-top stack held at no more than 0.5 W/mm2 as liquid-coolable while keeping the DRAM below 100 C.^[3]

The compute die is fabricated by TSMC on a 4 nm process, reported as N4 by ServeTheHome and N4P by Tom's Hardware, and is bonded face to face to the DRAM die at a 36 micrometer pitch, a technique d-Matrix characterized as proven, low-cost and high-yield rather than exotic.^[3][4]

Bandwidth and energy

Bhoja called the 0.37 pJ/bit number "a measured number" from working silicon.^[4] The energy advantage comes from geometry rather than process: the vertical path is a PHY-less, millimeter-scale connection, where an HBM link is a centimeter-scale route across an interposer through a physical layer block.^[3] The trade-off is capacity. Raptor's 32 GB per card is a fraction of what an HBM4 accelerator carries, and Bhoja acknowledged in the Hot Chips question session that the storage density gap traces back to d-Matrix using a less advanced DRAM process: "A lot of the drop for us was also because we used a not-so-advanced DRAM. And so if we used a more mainline DRAM, just like the HBM4 guys are doing, we would be able to push that up almost all the way to the HBM4 numbers."^[4]

d-Matrix's answer to the capacity question is to size deployments at rack scale rather than per card. A 72-card scale-up domain carries about 2.3 TB, which the company says is enough to hold a frontier model's weights at 4-bit precision with headroom for roughly 54 concurrent users at 1 million tokens of context.^[4] Bhoja summarized the position as "Even with 32 gigabytes of memory capacity, we are able to solve SOTA models in a scale-up network, so no bits are wasted."^[4]

Memory subsystem engineering

Much of the Hot Chips material dealt with problems created by the custom DRAM die rather than by the compute logic. Because the memory is co-designed rather than bought off a catalogue, its bank geometry does not match a conventional part, and the design work shows up in three places.

The first is bank-to-channel mapping. Each tensor engine requires a 128-byte flit per access, but the die's 840 banks (768 usable after 72 spares) are spread across 256 channels, giving three banks per channel and 96 bytes per access. Fetching a 128-byte flit in two accesses would pull 192 bytes and waste about a third of the bandwidth. d-Matrix's solution, which it calls stream blocking, shares one partial 32-byte access across three flits so that four accesses at 96 bytes feed three flits at 128 bytes with no overfetch and no shifting network.^[3]

The second is I/O power. Moving 100 TB/s at 0.37 pJ/bit costs roughly 296 W of what the ISCA paper gives as a 422 W per-package budget.^[4] Conventional data-bus inversion would recover about 20 percent, but it depends on multi-cycle bursts and a sideband pin, neither of which exists on a single-cycle 256-bit vertical link. d-Matrix instead compares each flit to the previous one and carries a one-bit inversion tag alongside the ECC metadata, which it calls stream flipping and puts at 0.8 percent overhead with no change to the physical layer.^[3][4]

The third is reliability at temperature. The DRAM is designed for a 105 C junction temperature, where retention falls from a standard 32 ms to 4 ms and refresh must run eight times more often. d-Matrix shrank each microbank to 1,366 rows and about 5.33 MB so that a full refresh sweep costs about 1.37 percent of bandwidth, and pairs commodity ECC with a [132,128] Reed-Solomon code on the logic die that corrects two symbol errors per 128 bytes, with a CRC behind it.^[3][4] Bank failures are absorbed by a two-level multiplexer chain the company calls bank chaining, which lets any two failed banks anywhere on the die be swapped for spares while channel widths stay symmetric.^[4]

Performance claims

d-Matrix's headline result at Hot Chips 2026 was roughly 988 tokens per second per user serving the 2.8-trillion-parameter Kimi K3 model at 1 million tokens of context, and the accompanying ISCA paper projects about 4.7 times higher throughput per card than HBM-based designs.^[4][5] Every one of these figures is a d-Matrix projection built on early silicon characterization rather than a measurement of a finished product, and no third party has benchmarked Raptor. Tom's Hardware noted that the closest published reference point is Cerebras's November 2024 claim of 969 tokens per second on Llama 3.1 405B, which Artificial Analysis verified on live hardware.^[4]

The workload argument behind the design is that decode, not prefill, dominates wall-clock inference time. Prefill processes many prompt tokens in parallel and is compute-bound; decode emits one token at a time and is usually memory-bandwidth-bound, and mixture-of-experts layers stay memory-bound even at modest batch sizes.^[3] Weights and KV cache both grow, the latter with context length multiplied by batch size, which d-Matrix illustrated with a figure of roughly 935 GB of KV cache for 64 users at 1 million tokens of context.^[3]

NVIDIA collaboration

The September 10, 2026 announcement covers a multi-year product roadmap and makes d-Matrix an NVLink Fusion partner, with Raptor as the first engagement point. d-Matrix will build a rack around the NVIDIA MGX reference design using Vera CPUs, NVLink switches, BlueField-4 DPUs, ConnectX-9 SuperNICs and Spectrum-X Ethernet networking, with modular cable-free trays.^[1] Astera Labs is named as the connectivity partner for the system.^[1]

What NVLink Fusion supplies to a third-party accelerator vendor is less a chip than a platform: NVLink scale-up networking, the MGX rack architecture, and the surrounding supply chain, power and cooling validation and systems software. NVIDIA's own framing is that building a custom XPU is only the first step and that deploying one at scale otherwise costs billions of dollars and years of work.^[2] d-Matrix's racks are also intended to run alongside NVIDIA Vera Rubin NVL72 systems for disaggregated inference rather than only standalone.^[2]

Quotes in the announcement come from Sid Sheth, founder and CEO of d-Matrix, who called the collaboration "a defining moment on our journey to infinite inference, accessible to all"; Jensen Huang, founder and CEO of NVIDIA, who said NVLink Fusion "gives partners like d-Matrix a path to integrate seamlessly with NVIDIA compute platforms"; and Jitendra Mohan, CEO of Astera Labs, who said the partnership delivers "high-throughput for low latency AI inference."^[1] Speaking at a press briefing the day before, Sheth added that NVLink Fusion and MGX let d-Matrix "integrate our Raptor XPUs into a broadly deployed, liquid-cooled architecture."^[2]

NVIDIA lists AWS, which is designing Trainium4 for NVLink Fusion, as another adopter, and names Alchip, Arm, Astera Labs, Ayar Labs, Cadence, Fujitsu, GUC, Intel, Lightmatter, Marvell, MediaTek, Samsung, SiFive and Synopsys in the surrounding technology ecosystem.^[2] ServeTheHome noted that the arrangement cuts both ways: d-Matrix avoids reinventing rack-scale networking, while NVIDIA supplies the CPUs and interconnect in any rack d-Matrix wins, taking a socket that AMD, Intel or Qualcomm would otherwise contest.^[12]

Positioning

d-Matrix frames Raptor around what it calls the premium token economy, a marketing term for agentic and interactive workloads where users pay more for speed: AI coding assistants, real-time chatbots and voice agents.^[1] The deployment model is heterogeneous disaggregation, splitting an inference request so that GPUs handle the compute-intensive prefill phase while d-Matrix XPUs handle the latency-sensitive decode phase.^[1] This is the same pattern d-Matrix used to sell Corsair, where the company reported customer speed-ups of more than 10x from mixing GPUs, CPUs and accelerators in one pipeline.^[10]

Sheth described the commercial logic on X the day after the announcement, saying customers "will be able to run a Raptor rack standalone, or alongside NVIDIA GPUs."^[11]

Status and open questions

Three questions remain open. The first is supply: d-Matrix has named TSMC for the logic die and Alchip for ASIC design and packaging, but neither operates a DRAM fab, and the company has not said who fabricates the custom DRAM die. Tom's Hardware pointed out that leading-edge DRAM capacity is committed to HBM4 lines and that a startup asking for a bespoke die with non-standard bank geometry is negotiating from a weak position.^[4]

The second is scaling beyond one rack. Aayush Ankit, who led Raptor's SoC architecture at d-Matrix before joining Meta's MTIA team and co-presented the Hot Chips session, put it as "We are making this unit of compute blazingly fast. Communication becomes a bottleneck soon enough."^[4] Once a model and its context no longer fit a scale-up domain, inference falls back on inter-card synchronization, which the ISCA paper acknowledges.

The third is the roadmap past one-high stacking. Asked at Hot Chips about multi-layer plans, Bhoja said the roadmap "is still a work in progress, and we have a hard enough time trying to get one-high to work and work around all of the thermal issues of that."^[4] Samsung presented a related concept at the same conference, zHBM, which stacks HBM directly on the processor, and unlike d-Matrix it owns its own DRAM fabs.^[4]

Competitive context

Raptor is unusual in the group for attacking capacity and bandwidth with the same mechanism. Groq and Cerebras get bandwidth from very large SRAM arrays and accept a capacity ceiling; HBM-based designs get capacity and accept a bandwidth and power ceiling. d-Matrix's claim is that stacking a custom DRAM die under the compute die gets a useful fraction of both, at the cost of depending on a memory supply chain it does not control.

See also

• d-Matrix Corsair
• NVLink Fusion
• NVIDIA MGX
• High Bandwidth Memory
• Disaggregated serving
• AI accelerator

References

1. ^1 ^2 ^3 ^4 ^5 ^6 ^7"d-Matrix Adopts NVIDIA NVLink Fusion Rackscale Infrastructure for Ultra-Low Latency AI Inference." d-Matrix. September 10, 2026. d-matrix.ai/...d-matrix-rackscale-nvidia
2. ^1 ^2 ^3 ^4 ^5Jesse Clayton. "d-Matrix Adopts NVIDIA NVLink Fusion for Rack-Scale XPU Deployment." NVIDIA Blog. September 10, 2026. blogs.nvidia.com/...d-matrix-nvlink-fusion
3. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10Patrick Kennedy. "d-Matrix Raptor 3D-DRAM Accelerator for Generative Inference at Hot Chips 2026." ServeTheHome. August 23, 2026. servethehome.com/...ve-inference-at-hot-chips-2026
4. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16Luke James. "Hot Chips 2026: d-Matrix stacks AI accelerator directly on custom DRAM for 100 TB/s per card." Tom's Hardware. August 26, 2026. tomshardware.com/...stom-dram-for-100-tbs-per-card
5. ^1 ^2Prashant J. Nair, Ramyad Hadidi, Subramani Ganesh, Sangamesh Kodge, Shubhankit Rathore, Neil Thanawala, Nikitha Reddy, Gyanesh Saharia, Vinayak Patankar, Arun Tiruvur, Nithesh Kurella and Sudeep Bhoja. "Early Silicon of Raptor: The First 3D-DRAM Accelerator for Generative Inference." 2026 ACM/IEEE 53rd Annual International Symposium on Computer Architecture (ISCA), pp. 2632-2647. June 2026. doi.org/...ISCA66397.2026.00183
6. ^"Hot Chips 2026 Program." Hot Chips, accessed September 15, 2026. Tutorial 1: Memory technology, Sunday August 23, 2026. hotchips.org
7. ^"d-Matrix and Alchip Announce Collaboration on World's First 3D DRAM Solution to Supercharge AI Inference." d-Matrix. November 18, 2025. d-matrix.ai/...olution-to-supercharge-ai-inference
8. ^Sudeep Bhoja. "Blazing the Trail Toward More Scalable, Affordable AI with 3DIMC." d-Matrix. August 25, 2025. d-matrix.ai/scaling-ai-inference-with-3dimc
9. ^"Going Vertical: Why we created a 3D DRAM solution to advance low latency AI inference." d-Matrix. March 16, 2026. d-matrix.ai/...to-advance-low-latency-ai-inference
10. ^1 ^2"d-Matrix Corsair AI Inference Platform Enters Full Production to Meet Customer Demand." d-Matrix. June 9, 2026. d-matrix.ai/...-production-to-meet-customer-demand
11. ^Sid Sheth (@sidsheth). Post on X. September 12, 2026. twitter.com/...2098572794472902791
12. ^Cliff Robinson. "d-Matrix Joins the NVIDIA NVLink Fusion Platform." ServeTheHome. September 12, 2026. servethehome.com/...-nvidia-nvlink-fusion-platform