Citation and evidence

NVIDIA NeMo

14 min full readUpdated 28 references

This article's verification

Report a problem with this article

Edit article
More

Use this article

Raw MarkdownExplore connections

Improve this page

Revision historyDiscuss this article

Topics

AI InfrastructureDeveloper ToolsNVIDIA

Cite this article

NVIDIA NeMo is an open, end-to-end framework from NVIDIA for building, customizing, and deploying generative AI models, described by NVIDIA as "a scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI." [1][2] It started in 2019 as a toolkit for speech and conversational AI and grew into a broader platform that covers large language model training, multimodal models, and automatic speech recognition and text to speech. NeMo runs on NVIDIA GPUs and ties together the pieces a team needs to take a model from raw data through pretraining, fine-tuning, alignment, and serving. [1][2][3]

A common point of confusion is the difference between NeMo and Nemotron. NeMo is the software framework you use to train and customize models. Nemotron is a family of actual models that NVIDIA built and released, and NVIDIA uses NeMo internally to train them. One is the workshop, the other is what comes out of it. [8][9]

When was NeMo released and where did it come from?

NVIDIA introduced NeMo in 2019. The name is short for Neural Modules, and the project was first described in a paper, "NeMo: a toolkit for building AI applications using Neural Modules," posted to arXiv on September 14, 2019. [12] The early design centered on reusable building blocks that took typed inputs and produced typed outputs, which you could connect to assemble a model. The first release leaned into conversational AI, with collections for automatic speech recognition, natural language processing, and speech synthesis. For several years NeMo was best known in the speech community as a way to train and fine-tune ASR and TTS models. [3][4][12]

As the field shifted toward large transformer models trained on huge clusters, NeMo expanded to match. It picked up dedicated collections for language models and multimodal models, adopted distributed training techniques built for thousands of GPUs, and added tooling for the stages around training such as data preparation, alignment, and safety. The result today describes itself as a cloud native generative AI framework that lets users "efficiently create, customize, and deploy new generative AI models by leveraging existing code and pretrained model checkpoints." [1][2]

"Building and Deploying Generative AI Models with NVIDIA NeMo Framework", official video from NVIDIA Developer, published August 7, 2023. Watch on YouTube.

How is NeMo built?

NeMo is written in Python and built on PyTorch. For large model training it sits on top of two NVIDIA libraries that do the heavy lifting. The first is Megatron Core, an open source library of GPU optimized building blocks and system level techniques for training transformer models at scale. The second is Transformer Engine, which handles mixed precision and FP8 training on NVIDIA Hopper and newer GPUs. NeMo wraps both behind higher level APIs so teams can configure a training run without reimplementing the low level kernels themselves. [1][2][5]

Training a model with hundreds of billions of parameters does not fit on one GPU, so NeMo supports several forms of model parallelism that can be combined. Tensor parallelism splits the math inside each layer across GPUs. Pipeline parallelism puts different layers on different GPUs and streams batches through them. Data parallelism replicates the model and feeds each copy a different slice of the batch. Sequence and context parallelism split work along the length of the input, which matters for long contexts, and expert parallelism distributes the experts in a mixture of experts model. NeMo also supports fully sharded data parallelism, which shards the parameters and optimizer state to save memory. The point of combining these is to spread a single model and its data across many GPUs and many nodes while keeping the hardware busy. It all runs on NVIDIA's CUDA platform and the GPU libraries built on top of it. [2]

Mixed precision and FP8 are part of why the framework targets recent hardware. Transformer Engine lets NeMo train in FP8 on Hopper and newer GPUs, which cuts memory use and speeds up training compared with higher-precision formats, as long as the GPU supports it. Older or non-NVIDIA hardware does not get that path. [1][2][5]

Because NeMo is a deep learning framework rather than a single model, it ships with pretrained checkpoints and recipes you can start from instead of training a neural network from scratch. The recipes capture the parallelism layout, the data configuration, and the hyperparameters for a given model size, so a team can reproduce a known-good setup rather than tuning everything by hand. NeMo also reads and writes formats that line up with the wider community, so checkpoints can move between NeMo and projects like Hugging Face. [1]

What are the main components of NeMo?

Over time NeMo has grown into a set of components that each handle one stage of the model lifecycle. Several of them were also offered as managed NeMo microservices, containerized versions meant to run on Kubernetes in production; NVIDIA sunset the NeMo microservices on October 1, 2026, and says there will be no new releases [7]. The table below lists the main pieces. [1][6][7]

ComponentRoleNotes
NeMo CuratorData curationGPU accelerated download, cleaning, filtering, and deduplication of text, image, video, and audio data for pretraining and fine-tuning. Uses RAPIDS with Ray to scale across multi-node, multi-GPU clusters (the 2023 original used MPI and Dask). In one NVIDIA benchmark it deduplicated 1.96 trillion tokens in about 0.5 hours on 32 NVIDIA H100 GPUs, and NVIDIA's 2023 launch post reported GPU deduplication running roughly 20x faster and 5x cheaper than a CPU pipeline. [8][13][24]
NeMo CustomizerFine-tuning serviceFormer microservice (sunset October 2026) for fine-tuning and adapting models, with methods such as LoRA, P-tuning, and supervised fine-tuning. [6][7]
NeMo RL (successor to NeMo Aligner)Post-training and alignmentLibrary for model alignment and reinforcement learning, including SFT, DPO, and GRPO, with Megatron Core or PyTorch-native AutoModel training backends, vLLM generation, and Ray scheduling; it runs from a single GPU up to multi-node clusters. The older NeMo Aligner, which supported reinforcement learning from human feedback, has not been maintained since May 15, 2025, and its repository is archived. [9][25]
NeMo GuardrailsSafety and controlOpen source toolkit for adding programmable rails to LLM applications, covering input, output, dialog, retrieval, and execution rails. [10]
NeMo RetrieverRetrieval and RAGMicroservices for embedding, reranking, and multimodal extraction that power retrieval augmented generation over enterprise data; NVIDIA now brands the retrieval models as Nemotron Retriever. NVIDIA reported up to 15x faster multimodal PDF extraction than open-source alternatives and 3x higher embedding throughput for its FP8 NIM than an FP16 open-source setup, and its Llama 3.2 embedding model was evaluated on 26 languages. [11][14][28]
NeMo EvaluatorEvaluationOpen-source library (formerly also a microservice, sunset October 2026) for benchmarking models on academic and custom datasets, including LLM-as-a-judge methods. [6][7]
NeMo Data StoreArtifact storageFormer microservice (sunset October 2026) that stored and managed datasets, models, and related artifacts across the workflow. [6][7]

Expanded article table

The components are designed to chain together into one workflow. A team can curate a dataset with Curator, train or fine-tune with the core framework and Customizer, align the result with NeMo RL, attach Guardrails, connect it to private data through Retriever, and check quality with Evaluator. The microservices versions share an API surface and run as containers, which is what lets them be wired into a continuous loop where a deployed model is evaluated, then fine-tuned again on fresh data, then redeployed. That data flywheel framing was how NVIDIA pitched the microservices for agentic applications that need to keep improving after launch, until it sunset them on October 1, 2026. [6][7][15]

What does NeMo Guardrails do?

Guardrails is worth a closer look because safety sits at a different layer from training. NVIDIA open-sourced NeMo Guardrails on April 25, 2023, as a toolkit for keeping generative AI chatbots accurate and on-topic. [16] The companion research paper describes the approach this way: "using a runtime inspired from dialogue management, NeMo Guardrails allows developers to add programmable rails to LLM applications, these are user-defined, independent of the underlying LLM, and interpretable." [17]

Rather than baking behavior into the weights, it inserts programmable rails between the application and the model. Input rails can reject or rewrite what a user sends, for example masking sensitive data. Dialog rails shape how the model is prompted and can hold it to a defined conversation flow. Retrieval rails act on the chunks pulled into a retrieval augmented generation pipeline. Execution rails govern the tools the model is allowed to call. Output rails check or alter the response before it reaches the user, covering things like content moderation, fact-checking, and jailbreak detection. Because it is provider-agnostic, Guardrails can sit in front of many different models rather than only NVIDIA's own, and it integrates with toolkits such as LangChain. [10][16][17]

What are the Parakeet and Canary speech models?

The speech roots are still part of NeMo. It keeps dedicated collections for automatic speech recognition and text to speech, with model families and pretrained checkpoints for transcription, speech synthesis, and related tasks such as speaker identification and spoken language understanding. NeMo has been a common starting point for state-of-the-art ASR work, since the collections include strong baseline architectures and recipes that researchers can fine-tune on their own audio rather than training from scratch. [1][4]

NVIDIA ships two flagship ASR families built with NeMo. Parakeet is a line of high-throughput English and multilingual transcription models, and Canary is a multilingual recognition and translation line. Parakeet TDT 0.6B v2, a 600 million parameter FastConformer-TDT model released in 2025, posted a 6.05 average word error rate on the Hugging Face Open ASR Leaderboard and an RTFx of 3,386, meaning it transcribes audio roughly 3,386 times faster than real time at batch size 128, which NVIDIA said ranked it first on that leaderboard as of June 2025. [18][19] NVIDIA's Canary-Qwen-2.5B, released in June 2025, reported a record 5.63% word error rate on the English Open ASR Leaderboard, and in August 2025 NVIDIA released Parakeet V3 and Canary V2 with speech recognition and translation for 25 European languages. [21] These results are a reminder that NeMo was a serious ASR and TTS toolkit before the language model era and did not drop that focus when it broadened. The same modular design that shaped the original Neural Modules still shows up here, where audio, language, and acoustic pieces can be assembled into a pipeline. [1][4]

In 2026 NeMo's main GitHub repository was refocused on speech: the old NeMo repository URLs now redirect to github.com/NVIDIA-NeMo/Speech, whose README presents the project as NVIDIA NeMo Speech, says the repository "has pivoted to focus on audio, speech, and multimodal LLMs," and names v2.7.3 as the final NeMo release before the repository split, with NeMo Speech 3.0 as release v3.0.0. [21] Large language model and multimodal training now lives in separate libraries in the NVIDIA-NeMo GitHub organization, chiefly Megatron Bridge and NeMo AutoModel, alongside NeMo RL, Curator, Evaluator, Export-Deploy, Run, and Guardrails. [2][23] NVIDIA released Nemotron 3.5 Transcribe, a speech-augmented language model, through NeMo Speech on 28 October 2026. NVIDIA trained it with NeMo Speech on the NeMo Automodel backend, and NeMo Speech supplies both the vLLM plugin used to serve it and its fine-tuning toolkit. [22]

How does NeMo fit with Nemotron, NIM, and NVIDIA AI Enterprise?

NeMo sits inside a larger NVIDIA software stack, and the parts are easy to mix up, so it helps to separate them. NeMo is the framework for building and customizing models. Nemotron is a family of NVIDIA models that are built with that framework. NVIDIA NIM, short for NVIDIA Inference Microservices, packages models as optimized containers for serving, and a model customized in NeMo can be deployed through NIM. NeMo models can also be exported to TensorRT-LLM for optimized inference. [1][3]

The whole collection is offered commercially through NVIDIA AI Enterprise, which adds support, security updates, and a supported path to production for the NeMo framework (and, until their October 2026 sunset, the NeMo microservices). [7] The NeMo microservices reached general availability on April 23, 2025; NVIDIA named SAP and ServiceNow among the enterprise platforms using NeMo Retriever microservices, and Constellation Research reported that NVIDIA cited Amdocs telecom agents delivering a 64% improvement in average handling time. [15][20][26][27] NVIDIA sunset the microservices on October 1, 2026. [7] So a developer can use the open source NeMo libraries for free, or adopt supported NeMo framework containers and NIM under NVIDIA AI Enterprise when running in production. [1][3][7]

Who uses NeMo?

NeMo is aimed at organizations that train or heavily customize their own models on NVIDIA hardware rather than only calling a hosted API. That includes research labs publishing new models, enterprises building domain-specific assistants and agents, and teams that need to fine-tune a foundation model on private data and keep it inside their own environment for privacy or compliance reasons. NVIDIA itself uses NeMo to produce the Nemotron models, which makes the framework a working example of its own use rather than a tool NVIDIA only ships to others. The audience splits roughly in two. Researchers and individual developers tend to reach for the open source libraries directly. Larger companies that want a supported path lean on the NVIDIA AI Enterprise packaging (and, until their October 2026 sunset, the NeMo microservices) so they get vendor support and security updates instead of maintaining the stack on their own. [1][6][15]

Limitations

NeMo is built for NVIDIA GPUs and the CUDA platform, so it is not a portable, hardware-neutral framework. Teams without access to capable NVIDIA hardware, especially the Hopper and newer GPUs needed for FP8 training, get less out of it. The framework is also large and aimed at people doing real training and customization, which makes it heavier to learn than a simple inference library or an API client. The full value shows up most for groups that actually pretrain, fine-tune, or align models at scale, while a team that only needs to call an existing model through an endpoint will find most of NeMo unnecessary. The NeMo name can also be confusing, since it has covered free open source libraries, commercial microservices (sunset in October 2026), and, after the 2026 reorganization, a family of separately named repositories such as NeMo Speech, Megatron Bridge, and NeMo RL. [1][7][23]

Continue reading

More to read

Topics:AI InfrastructureDeveloper ToolsNVIDIA

References

  1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13NVIDIA. "NVIDIA/NeMo." GitHub. github.com/...NeMo
  2. ^1 ^2 ^3 ^4 ^5 ^6 ^7NVIDIA. "NeMo Framework Overview." NVIDIA NeMo Framework User Guide. docs.nvidia.com/...overview
  3. ^1 ^2 ^3 ^4NVIDIA. "NVIDIA NeMo Framework Megatron Backend." NGC Catalog (developer.nvidia.com/nemo-framework redirects here). catalog.ngc.nvidia.com/...nemo
  4. ^1 ^2 ^3NVIDIA. "NeMo Toolkit." NVIDIA NeMo Framework User Guide. docs.nvidia.com/...nemotoolkit
  5. ^1 ^2NVIDIA. "Megatron-Core." NVIDIA Docs. docs.nvidia.com/megatron-core
  6. ^1 ^2 ^3 ^4 ^5 ^6NVIDIA. "NeMo Microservices." NVIDIA Docs. docs.nvidia.com/...about
  7. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10NVIDIA. "NVIDIA NeMo Microservices." developer.nvidia.com. developer.nvidia.com/nemo-microservices
  8. ^1 ^2NVIDIA. "NVIDIA-NeMo/Curator." GitHub. github.com/...Curator
  9. ^1 ^2NVIDIA. "NVIDIA-NeMo/RL." GitHub. github.com/...RL
  10. ^1 ^2NVIDIA. "NVIDIA/NeMo-Guardrails." GitHub. github.com/...NeMo-Guardrails
  11. ^NVIDIA. "NVIDIA NeMo Retriever." developer.nvidia.com (archived June 2, 2025). web.archive.org/...nemo-retriever
  12. ^1 ^2Kuchaiev, Oleksii, et al. "NeMo: a toolkit for building AI applications using Neural Modules." arXiv:1909.09577, September 14, 2019. arxiv.org/...1909.09577
  13. ^NVIDIA. "Curating Trillion-Token Datasets: Introducing NVIDIA NeMo Data Curator." NVIDIA Technical Blog. developer.nvidia.com/...roducing-nemo-data-curator
  14. ^NVIDIA. "NVIDIA NeMo Retriever Delivers Accurate Multimodal PDF Data Extraction 15x Faster." NVIDIA Technical Blog. developer.nvidia.com/...data-extraction-15x-faster
  15. ^1 ^2 ^3NVIDIA. "Enhance Your AI Agent with Data Flywheels Using NVIDIA NeMo Microservices." NVIDIA Technical Blog. developer.nvidia.com/...-nvidia-nemo-microservices
  16. ^1 ^2NVIDIA. "Right on Track: NVIDIA Open-Source Software Helps Developers Add Guardrails to AI Chatbots." NVIDIA Blog, April 25, 2023. blogs.nvidia.com/...ai-chatbot-guardrails-nemo
  17. ^1 ^2Rebedea, Traian, et al. "NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails." arXiv:2310.10501, October 16, 2023. arxiv.org/...2310.10501
  18. ^NVIDIA. "nvidia/parakeet-tdt-0.6b-v2." Hugging Face model card. huggingface.co/...parakeet-tdt-0.6b-v2
  19. ^NVIDIA. "NVIDIA Speech AI Models Deliver Industry-Leading Accuracy and Performance." NVIDIA Technical Blog. developer.nvidia.com/...g-accuracy-and-performance
  20. ^CIO. "Nvidia says NeMo microservices now generally available." April 23, 2025. cio.com/...o-microservices-now-generally-available
  21. ^1 ^2NVIDIA, "NVIDIA-NeMo/Speech" GitHub repository README. Retrieved 29 October 2026. github.com/...Speech
  22. ^NVIDIA, "NVIDIA-Nemotron-3.5-Transcribe-30B-A4B" model card, Hugging Face. Retrieved 29 October 2026. huggingface.co/...-Nemotron-3.5-Transcribe-30B-A4B
  23. ^1 ^2NVIDIA. "NVIDIA-NeMo." GitHub organization. Retrieved 2 November 2026. github.com/NVIDIA-NeMo
  24. ^NVIDIA. "Key Features." NVIDIA NeMo Curator documentation. docs.nvidia.com/...key-features
  25. ^NVIDIA. "NVIDIA/NeMo-Aligner" (archived repository). GitHub. github.com/...NeMo-Aligner
  26. ^Conway, Joey. "Enterprises Onboard AI Teammates Faster With NVIDIA NeMo Tools to Scale Employee Productivity." NVIDIA Blog, April 23, 2025. blogs.nvidia.com/...eammates-employee-productivity
  27. ^Dignan, Larry. "Nvidia NeMo Microservices generally available, aims for AI agent data flywheel." Constellation Research, April 23, 2025. constellationr.com/...-aims-ai-agent-data-flywheel
  28. ^NVIDIA. "nvidia/llama-3.2-nv-embedqa-1b-v2." Hugging Face model card. huggingface.co/...llama-3.2-nv-embedqa-1b-v2

Improve this article

Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.

9 revisions · v9 · 2,764 words · full history

Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify

Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here

Reviewer note: Independent fact-check of the full article against its cited sources (70 claims, 31 sources fetched); 3 material + 18 minor fixed

Cite this page: AI Wiki. "NVIDIA NeMo." aiwiki.ai, updated 29 Oct 2026, fact-checked 2 Nov 2026. CC BY 4.0. https://aiwiki.ai/wiki/nvidia_nemo

Edit article