NVIDIA DeepStream

NVIDIA DeepStream is a GPU-accelerated software development kit for building real-time streaming analytics pipelines, principally for video. It is built on the open-source GStreamer multimedia framework and supplies a set of NVIDIA plugins that run hardware video decode, batched neural-network inference, object tracking and message-broker output on NVIDIA GPUs, from Jetson edge modules to data-center cards.^[1][4] DeepStream 9.1, released on 14 July 2026, is the current version; it moved SDK distribution to GitHub, added support for NVIDIA JetPack 7.2 on Jetson Orin, and shipped a set of packaged "agent skills" that let coding agents stand up multi-camera 3D tracking pipelines from natural-language prompts.^[3][8][10]

Scroll sideways for more →

What DeepStream is

DeepStream is not a model and not an application. It is a set of GStreamer plugins plus reference applications that assemble into a pipeline: sources feed decoders, decoded frames are batched, the batch goes through one or more inference stages, a tracker assigns persistent IDs, analytics rules run over the results, and metadata is drawn on screen or published to a message broker.^[4]

The pipeline model

The plugins that matter most are:

Scroll sideways for more →

Batching is the design decision that makes the rest work. nvstreammux requests a pad per source, collects frames round-robin, and pushes a batch downstream either when batch-size frames have been gathered or when the batched-push-timeout expires, so a slow or stalled camera cannot block the pipeline indefinitely. Frames that do not match the muxer's output resolution are scaled into the batch buffer, and the muxer supports adding and removing sources at runtime through pad-added and pad-deleted events.^[5]

Why this makes many-camera deployment economical

A plain-language version of the argument: decoding an H.264 stream on a CPU is expensive, and running a detector on one frame at a time wastes most of a GPU's arithmetic units. DeepStream sends decode to the fixed-function NVDEC block rather than the CPU or the shader cores, keeps the decoded surface in GPU memory so no frame is copied back to host memory between stages, and then hands the inference engine a stack of frames from different cameras at once. A GPU reaches high utilization on a batch of 30 frames where it would idle on one. Cost per camera therefore falls as camera count rises, which is what makes a 200-camera site affordable on a handful of accelerators rather than one box per camera.^[4][5]

Version history

Dates below are taken from NVIDIA release notes, NVIDIA announcement posts and the GitHub release metadata.

Scroll sideways for more →

DeepStream predates this table by several years. Releases before 7.0 are omitted because their dates could not be confirmed against a primary NVIDIA source.

Object tracking

Tracking is the technical heart of DeepStream. Gst-nvtracker does not implement tracking itself: it loads a low-level library that implements the NvDsTracker API, queries it for the frame formats and memory types it needs, converts buffers accordingly, and passes the whole multi-stream batch through a single API call so the tracker can run in batched mode on the GPU.^[6]

The reference trackers

Scroll sideways for more →

NvDCF is the only one of the four original trackers that produces a per-target confidence value, because it does visual tracking rather than pure box association.^[6] A sub-batching feature splits the input batch across several tracker library instances on separate threads, which matters above 128 streams because each VPI-backend tracker instance supports at most 128.^[6]

Why cross-camera identity is hard

Keeping one identity for a person walking from one camera's view into another's is a different problem from tracking within a single view. Within a view, an object moves a little between frames, so proximity plus a motion model does most of the work. Across views, the same person appears at a different scale, from a different angle, under different lighting, often with a gap in time between sightings. Appearance embeddings help but degrade with viewpoint change and clothing similarity. The geometric fix is calibration: if every camera's mapping from image pixels to a shared world coordinate system is known, two detections can be compared by where they are on the floor rather than by what they look like, and "the same person" becomes "the same point in the room". That is why multi-camera systems make calibration a prerequisite, and why calibration has been the deployment bottleneck.

Multi-View 3D Tracking

Multi-View 3D Tracking (MV3DT) is NVIDIA's distributed answer. Each camera runs detection and single-view 3D tracking, back-projecting 2D boxes into 3D world coordinates through a 3x4 projection matrix stored in a YAML calibration file, using a ground-plane assumption. Cameras with overlapping fields of view then exchange tracklets over MQTT and negotiate a globally unique ID for each new object, with no central aggregator. The documented mechanisms include peer-target re-association, late re-association for missed handovers, ID correction when a handover assigns the wrong ID, Kalman-filter fusion of measurements from several cameras, and "see-through" tracking that starts a track for a target completely occluded in one camera's view using a peer camera's observation.^[14]

DeepStream 9.1 ships MV3DT with three detectors: PeopleNet Transformer (the default for pedestrian scenes), PeopleNet v2.6.3 (DetectNet_v2-based), and an RT-DETR 2D warehouse variant trained on classes including person, forklift and humanoid robots. Output goes to a tiled on-screen display with 2D and 3D boxes, a bird's-eye-view trajectory map, and Kafka protobuf messages carrying sensor ID, object ID and 3D box per frame.^[2][19]

The underlying algorithm is published. Hernandez and colleagues report 96.5% IDF1, 93.1% MOTA and 94.6% MOTP on WILDTRACK and 41.7% IDF1 with 50.9% MOTA on the SCOUT benchmark, sustaining 30 FPS across 100 cameras with under 10 ms inter-camera latency and 2.2% communication overhead; they describe the system as operating zero-shot given camera calibrations.^[16] Those are the authors' own reported figures.

Contrary to some coverage of the 9.1 release, MV3DT is not new in 9.1. NVIDIA's documentation states it was "newly introduced in DeepStream 8.0", where the feature page carried a Developer Preview label, and the 9.0 release notes list it again among that release's tracker additions.^[14][15][3]

AutoMagicCalib

AutoMagicCalib (AMC) is the calibration tool that MV3DT depends on. Instead of walking a checkerboard target through every camera's view, AMC runs DeepStream detection and tracking over ordinary operational footage, collects the trajectories of moving people, and derives each camera's intrinsic parameters (focal length, principal point, lens distortion) and extrinsic parameters (rotation, translation, world position) from them. The pipeline runs per-camera trajectory extraction, single-view calibration and rectification, multi-view tracklet matching anchored by a handful of manually placed alignment points on a layout image, and bundle adjustment across all views. An optional path uses VGGT, a learned geometry model, when object movement is too limited for the trajectory-based method.^[2][17][18] AMC is delivered as a microservice with a REST API and a six-step web interface; its documentation page is already present in the DeepStream 9.0 documentation set, so it is not a 9.1 debut either.^[17]

AMC's documentation is unusually candid about what it needs: ten or more unique people walking through the space, five minutes or more of footage per camera at 1920x1080, time-synchronized streams, cameras listed in order of overlapping field of view, and at least 30% overlap between consecutive pairs. Tracklets shorter than 90 frames are discarded, and multi-camera calibration wants at least six matched cross-camera tracklets per pair.^[18]

DeepStream 9.1 and agent skills

The headline of the 9.1 launch is not a new tracker but a new way of driving the SDK. A DeepStream "skill" is a directory containing a SKILL.md file plus reference material, following the convention that coding agents such as Claude Code, Codex and Cursor already support. Copying the directory into the agent's skills folder gives the agent condensed, guardrailed knowledge of DeepStream APIs and runbooks, so a developer can write "deploy mv3dt on the 12-camera sample dataset" and have the agent check prerequisites, pull containers, start Kafka and Mosquitto, download detector weights, generate configuration and launch the pipeline.^[2][9][13]

The counts reported around this release do not all agree, and the discrepancy is worth stating plainly. NVIDIA's launch blog says DeepStream 9.1 gives "access to 13 agentic skills".^[2] The repository at tag v9.1.0 contains ten skill directories: deepstream-dev, deepstream-generate-pipeline, deepstream-profile-pipeline, deepstream-import-vision-model, deepstream-run-mv3dt, deepstream-sop, and four AMC skills.^[7][9] The documentation describes "six DeepStream pipeline skills" plus the AMC calibration skills, again ten.^[13] The skills README itself says the project ships "nine complementary skills" immediately above a table listing ten.^[9] Ten is the number a reader can verify by cloning the repository.

Alongside the skills, NVIDIA ships an Inference Builder MCP server that lets an agent generate pipeline configurations, build Docker images and run smoke tests.^[13] NVIDIA's own documentation attaches a warning to all of it: generated code is a starting point and must go through code review, testing and security validation before production use.^[13]

Licensing, cost and what "open source" means here

This is where preliminary reporting overstated things. DeepStream's source code is genuinely open: the monorepo at github.com/NVIDIA/DeepStream carries a dual license, Apache-2.0 for source and CC-BY-4.0 for documentation and skills, and contains GStreamer plugin sources, utility libraries, sample and reference applications, TAO integration apps and the Service Maker SDK.^[7][11] That is a change in kind from the older model, where sources were scattered across NVIDIA-AI-IOT repositories and NGC downloads.

But the SDK is not fully open source. NVIDIA's own FAQ distinguishes the full package, which contains the complete SDK including both open-source components and proprietary libraries, from the deepstream-binaries package, which contains only the proprietary binary libraries for people building from source.^[12] The repository README calls this the "DeepStream 9.1 proprietary runtime" and notes that build/build.sh downloads it automatically before compiling.^[7] Release assets, including those binaries and the sample data, are governed by the NVIDIA Software Development Kit license agreement rather than Apache-2.0.^[7] The AutoMagicCalib repository draws the same line: Apache-2.0 scripts that pull proprietary auto-magic-calib containers from NGC.^[18] The repository is also explicitly not accepting code contributions.^[7]

The SDK itself carries no download fee; users accept the DeepStream SDK software license agreement and the NVIDIA AI Product License at download.^[33] Production support and enterprise terms come through NVIDIA AI Enterprise rather than a separate DeepStream runtime charge.^[1]

Performance

NVIDIA publishes end-to-end throughput for its pre-trained models, measured on a 1080p H.265 clip with rendering disabled. A sample of the DeepStream 9.1 figures:^[19]

Scroll sideways for more →

These are NVIDIA's own measurements. The spread between PeopleNet 2.6.3 and PeopleNet Transformer under the same tracker shows that detector choice, not the tracking stack, usually sets the camera budget.

Where it sits in NVIDIA's stack

DeepStream is the perception layer of NVIDIA Metropolis. Models come from TAO Toolkit or NGC; inference runs on TensorRT or Triton; deployment targets are Jetson modules under JetPack at the edge or Blackwell, Hopper, Ada and Turing GPUs in the data center.^[3][7] Above it sit the blueprints. NVIDIA's AI Blueprint for video search and summarization (VSS) uses vision language models and retrieval to answer questions about long video, and its ingestion path uses DeepStream's NvDCF tracker with SAM2 for mask generation; the 9.1 blog notes that the prebuilt MV3DT microservice can be combined with it.^[28][2] NVIDIA also markets a multi-camera tracking workflow built on DeepStream, a re-identification embedding model, a Kafka broker and Kubernetes.^[26]

Applications and deployments

NVIDIA positions DeepStream around four families of use case: warehouse and industrial safety (tracking workers, forklifts and autonomous mobile robots across a facility), retail analytics (customer flow, dwell time, self-checkout), smart-building monitoring, and traffic and smart-city work.^{[1][2][25][26]} The MV3DT reference datasets shipped with 9.1 are warehouse scenes, and the warehouse RT-DETR variant is trained on person, forklift and humanoid-robot classes, which indicates where NVIDIA expects the first multi-camera deployments.^[2][19]

Named, dated deployments are harder to come by than marketing categories. One documented example: the City of Raleigh, North Carolina, working with Metropolis partner Quantiphi, applied DeepStream to hundreds of existing traffic cameras to count vehicles entering and leaving intersections in real time, reaching what the city's chief information officer described as 95% accuracy after tuning NVIDIA pre-trained models; the city said it planned to extend the system to road flooding, parking utilization, bus-stop wait times and sanitation.^[27] The 9.1 SOP skill points at another line of work: step-sequence compliance for industrial operators, built from event boundary detection plus a vision language model.^[9][13]

Limitations and criticism

DeepStream's learning curve is GStreamer's learning curve. The Service Maker and pyservicemaker APIs exist precisely because assembling a pipeline by hand requires understanding pads, caps negotiation, and probe callbacks.^[23] NVIDIA's documentation frames its agent skills the same way, as a response to the demand for "deep expertise in GStreamer internals".^[13]

The documented limitations are extensive. From the 9.1 release notes: DeepStream's Triton integration supports a single GPU; moving a model from nvinfer to nvinferserver can cost 5 to 15% throughput; network width must be a multiple of 4 and height a multiple of 2; dynamic resolution change is alpha quality; XID errors and occasional kernel stalls have been seen above 200 streams; and performance under WSL2 lags a native Ubuntu install.^[3]

Version churn is a real cost. Each major DeepStream release is pinned to specific CUDA, TensorRT, Triton and JetPack versions, so an upgrade tends to be a platform migration rather than a package bump: 9.1 requires Ubuntu 24.04, CUDA 13.2, TensorRT 10.16.x and driver 595 or later on x86, and JetPack 7.2 GA on Jetson, a JetPack release NVIDIA published on 2 June 2026 that brought the Orin family onto the JetPack 7 line alongside Thor.^[7][24] NVIDIA maintains a dedicated migration guide for each release. Features are also removed: DeepStream 8.0 dropped TensorFlow, UFF and Caffe model support along with a long list of pre-trained models, and the Python bindings (pyds) are now deprecated in favor of pyservicemaker, which obliges existing Python applications to be rewritten.^[3]

Community friction is visible in NVIDIA's own forum. When 8.0 shipped in September 2025, developers posted a thread asking why there had been no forum announcement a week after release, noting that previous versions had been announced there.^[32] NVIDIA posted the announcement three days later.^[21]

Alternatives

Scroll sideways for more →

DeepStream's advantage is throughput per dollar on NVIDIA hardware, and its disadvantage is that the advantage exists only there. A pipeline built around nvinfer, nvstreammux and nvtracker does not port to another vendor's accelerator without a rewrite.

Surveillance and dual use

Multi-camera person tracking is dual-use: the same capability that counts shoppers or detects a worker entering a forklift aisle can follow named individuals through a building. NVIDIA's position, stated on its multi-camera tracking workflow page, is that identity in these systems "is tracked through visual embeddings/appearance rather than any personal biometric information, so privacy is fully maintained".^[26] That claim is narrower than it sounds: an appearance embedding is not a face template and does not survive a change of clothes, but it does support following a specific person across a camera network for the duration of a visit. Separately, NVIDIA removed FaceDetect, FaceDetectIR, gaze, emotion and facial-landmark models from DeepStream starting with version 8.0; the release notes record the removal without giving a reason.^[3] DeepStream is not facial recognition software, and NVIDIA does not ship a face-identification model with it, but nothing in the SDK prevents a developer from attaching one as a secondary inference stage.

See also

• NVIDIA
• TensorRT
• NVIDIA Triton Inference Server
• NVIDIA Jetson
• Jetson Thor
• NVIDIA Holoscan
• Computer vision
• Edge AI

References

1. ^"DeepStream SDK", NVIDIA Developer. developer.nvidia.com/deepstream-sdk
2. ^Shubham Agrawal and Debraj Sinha, "Build a Multi-Camera 3D Tracking Application with NVIDIA DeepStream 9.1 Skills", NVIDIA Technical Blog, 15 July 2026. developer.nvidia.com/...idia-deepstream-9-1-skills
3. ^"DeepStream SDK 9.1 for NVIDIA dGPU/X86 and Jetson" (release notes), NVIDIA DeepStream documentation. docs.nvidia.com/...DS_Release_notes
4. ^"Welcome to the DeepStream Documentation" (overview and architecture), NVIDIA. docs.nvidia.com/...DS_Overview
5. ^"Gst-nvstreammux", DeepStream Plugin Guide, NVIDIA. docs.nvidia.com/...DS_plugin_gst-nvstreammux
6. ^"Gst-nvtracker", DeepStream Plugin Guide, NVIDIA. docs.nvidia.com/...DS_plugin_gst-nvtracker
7. ^NVIDIA/DeepStream repository README (tag v9.1.0), GitHub. github.com/...DeepStream
8. ^"DeepStream 9.1.0" release and assets, GitHub. github.com/...v9.1.0
9. ^"DeepStream Coding Agent", skills/README.md, NVIDIA/DeepStream, GitHub. github.com/...README.md
10. ^CHANGELOG.md, NVIDIA/DeepStream, GitHub. github.com/...CHANGELOG.md
11. ^LICENSE, NVIDIA/DeepStream, GitHub. github.com/...LICENSE
12. ^"FAQ for DeepStream GitHub Mono-repo", NVIDIA DeepStream documentation. docs.nvidia.com/...DS_Monorepo_FAQ
13. ^"DeepStream Coding Agent", NVIDIA DeepStream documentation. docs.nvidia.com/...DS_AI_Agent
14. ^"Multi-View 3D Tracking", NVIDIA DeepStream 9.1 documentation. docs.nvidia.com/...DS_MV3DT
15. ^"Multi-View 3D Tracking (Developer Preview)", NVIDIA DeepStream 8.0 documentation. docs.nvidia.com/...DS_MV3DT
16. ^Byron Hernandez, Fangyu Li, Aotian Wu, Paul J. Shin, Kaustubh Purandare and Henry Medeiros, "Fully Distributed Multi-View 3D Tracking in Real-Time", arXiv:2606.13127, submitted 11 June 2026. arxiv.org/...2606.13127
17. ^"AutoMagicCalib", NVIDIA DeepStream documentation. docs.nvidia.com/...DS_AutoMagicCalib
18. ^NVIDIA-AI-IOT/auto-magic-calib repository README, GitHub. github.com/...auto-magic-calib
19. ^"Performance", NVIDIA DeepStream documentation. docs.nvidia.com/...DS_Performance
20. ^"DeepStream SDK 8.0 for NVIDIA dGPU/X86 and Jetson" (release notes), NVIDIA. docs.nvidia.com/...DS_Release_notes
21. ^"NVIDIA DeepStream SDK Version 8 is Now Available", NVIDIA Developer Forums, 23 September 2025. forums.developer.nvidia.com/...345860
22. ^"DeepStream 7.1 is now available!", NVIDIA Developer Forums, 18 October 2024. forums.developer.nvidia.com/...310284
23. ^"NVIDIA DeepStream 7.0 Milestone Release for Next-Gen Vision AI Development", NVIDIA Technical Blog, 14 May 2024. developer.nvidia.com/...-gen-vision-ai-development
24. ^"JetPack SDK Downloads and Notes", NVIDIA Developer (JetPack 7.2, 2 June 2026). developer.nvidia.com/...downloads
25. ^"AI-Powered Multi-Camera Tracking for Smart Spaces", NVIDIA. nvidia.com/...ai-powered-multi-camera-tracking
26. ^"The Multi-Camera Tracking AI Workflow", NVIDIA. nvidia.com/...multi-camera-tracking
27. ^"AI Getting Green Light: City of Raleigh Taps NVIDIA Metropolis to Improve Traffic", NVIDIA Blog, 11 March 2024. blogs.nvidia.com/...-of-raleigh-metropolis-traffic
28. ^"Advance Video Analytics AI Agents Using the NVIDIA AI Blueprint for Video Search and Summarization", NVIDIA Technical Blog, 18 May 2025. developer.nvidia.com/...o-search-and-summarization
29. ^Intel Deep Learning Streamer (DL Streamer) Pipeline Framework README, GitHub. github.com/...dlstreamer
30. ^Frigate NVR README, GitHub. github.com/...frigate
31. ^"NVIDIA Holoscan SDK", NVIDIA Developer. developer.nvidia.com/holoscan-sdk
32. ^"Feedback on DeepStream 8.0 Announcement", NVIDIA Developer Forums, 20 September 2025. forums.developer.nvidia.com/...345533
33. ^"DeepStream SDK, Get Started", NVIDIA Developer. developer.nvidia.com/deepstream-getting-started