# NVIDIA NeMo

> Source: https://aiwiki.ai/wiki/nvidia_nemo
> Updated: 2026-10-29
> Fact-checked: 2026-11-02
> Categories: AI Infrastructure, Developer Tools, NVIDIA
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "NVIDIA NeMo." aiwiki.ai, 29 Oct 2026. https://aiwiki.ai/wiki/nvidia_nemo
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

**NVIDIA NeMo** is an open, end-to-end framework from [NVIDIA](https://aiwiki.ai/wiki/nvidia) for building, customizing, and deploying [generative AI](https://aiwiki.ai/wiki/generative_ai) models, described by NVIDIA as "a scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI." [1][2] It started in 2019 as a toolkit for speech and conversational AI and grew into a broader platform that covers [large language model](https://aiwiki.ai/wiki/large_language_model) training, [multimodal](https://aiwiki.ai/wiki/multimodal_model) models, and [automatic speech recognition](https://aiwiki.ai/wiki/speech_recognition) and text to speech. NeMo runs on NVIDIA GPUs and ties together the pieces a team needs to take a model from raw data through pretraining, fine-tuning, alignment, and serving. [1][2][3]

A common point of confusion is the difference between NeMo and Nemotron. NeMo is the software framework you use to train and customize models. [Nemotron](https://aiwiki.ai/wiki/nemotron) is a family of actual models that NVIDIA built and released, and NVIDIA uses NeMo internally to train them. One is the workshop, the other is what comes out of it. [8][9]

## When was NeMo released and where did it come from?

NVIDIA introduced NeMo in 2019. The name is short for Neural Modules, and the project was first described in a paper, "NeMo: a toolkit for building AI applications using Neural Modules," posted to [arXiv](https://aiwiki.ai/wiki/arxiv) on September 14, 2019. [12] The early design centered on reusable building blocks that took typed inputs and produced typed outputs, which you could connect to assemble a model. The first release leaned into conversational AI, with collections for automatic speech recognition, [natural language processing](https://aiwiki.ai/wiki/natural_language_processing), and speech synthesis. For several years NeMo was best known in the speech community as a way to train and fine-tune ASR and TTS models. [3][4][12]

As the field shifted toward large transformer models trained on huge clusters, NeMo expanded to match. It picked up dedicated collections for [language model](https://aiwiki.ai/wiki/language_model)s and multimodal models, adopted [distributed training](https://aiwiki.ai/wiki/distributed_training) techniques built for thousands of GPUs, and added tooling for the stages around training such as data preparation, alignment, and safety. The result today describes itself as a cloud native generative AI framework that lets users "efficiently create, customize, and deploy new generative [AI models](https://aiwiki.ai/wiki/model) by leveraging existing code and pretrained model checkpoints." [1][2]

<figure class="embedvideo">
<iframe src="https://www.youtube-nocookie.com/embed/gTwLLhebOcQ" title="Building and Deploying Generative AI Models with NVIDIA NeMo Framework"></iframe>
<figcaption>"Building and Deploying Generative AI Models with NVIDIA NeMo Framework", official video from NVIDIA Developer, published August 7, 2023. <a href="https://www.youtube.com/watch?v=gTwLLhebOcQ">Watch on YouTube</a>.</figcaption>
</figure>

## How is NeMo built?

NeMo is written in Python and built on [PyTorch](https://aiwiki.ai/wiki/pytorch). For large [model training](https://aiwiki.ai/wiki/model_training) it sits on top of two NVIDIA libraries that do the heavy lifting. The first is Megatron Core, an open source library of GPU optimized building blocks and system level techniques for training [transformer](https://aiwiki.ai/wiki/transformers) models at scale. The second is Transformer Engine, which handles mixed precision and FP8 training on [NVIDIA Hopper](https://aiwiki.ai/wiki/nvidia_hopper) and newer GPUs. NeMo wraps both behind higher level APIs so teams can configure a [training run](https://aiwiki.ai/wiki/training_run) without reimplementing the low level kernels themselves. [1][2][5]

Training a model with hundreds of billions of parameters does not fit on one GPU, so NeMo supports several forms of [model parallelism](https://aiwiki.ai/wiki/model_parallelism) that can be combined. [Tensor parallelism](https://aiwiki.ai/wiki/tensor_parallelism) splits the math inside each layer across GPUs. [Pipeline parallelism](https://aiwiki.ai/wiki/pipeline_parallelism) puts different layers on different GPUs and streams batches through them. [Data parallelism](https://aiwiki.ai/wiki/data_parallelism) replicates the model and feeds each copy a different slice of the batch. Sequence and [context parallelism](https://aiwiki.ai/wiki/context_parallelism) split work along the length of the input, which matters for long contexts, and [expert parallelism](https://aiwiki.ai/wiki/expert_parallelism) distributes the experts in a [mixture of experts](https://aiwiki.ai/wiki/mixture_of_experts) model. NeMo also supports fully sharded data parallelism, which shards the parameters and optimizer state to save memory. The point of combining these is to spread a single model and its data across many GPUs and many nodes while keeping the hardware busy. It all runs on NVIDIA's [CUDA](https://aiwiki.ai/wiki/cuda) platform and the GPU libraries built on top of it. [2]

Mixed precision and FP8 are part of why the framework targets recent hardware. Transformer Engine lets NeMo train in FP8 on Hopper and newer GPUs, which cuts memory use and speeds up training compared with higher-precision formats, as long as the GPU supports it. Older or non-NVIDIA hardware does not get that path. [1][2][5]

Because NeMo is a [deep learning](https://aiwiki.ai/wiki/deep_learning) framework rather than a single model, it ships with pretrained checkpoints and recipes you can start from instead of training a [neural network](https://aiwiki.ai/wiki/neural_network) from scratch. The recipes capture the parallelism layout, the data configuration, and the [hyperparameter](https://aiwiki.ai/wiki/hyperparameter)s for a given model size, so a team can reproduce a known-good setup rather than tuning everything by hand. NeMo also reads and writes formats that line up with the wider community, so checkpoints can move between NeMo and projects like [Hugging Face](https://aiwiki.ai/wiki/hugging_face). [1]

## What are the main components of NeMo?

Over time NeMo has grown into a set of components that each handle one stage of the model lifecycle. Several of them were also offered as managed NeMo microservices, containerized versions meant to run on [Kubernetes](https://aiwiki.ai/wiki/kubernetes) in production; NVIDIA sunset the NeMo microservices on October 1, 2026, and says there will be no new releases [7]. The table below lists the main pieces. [1][6][7]

| Component | Role | Notes |
|---|---|---|
| NeMo Curator | Data curation | GPU accelerated download, cleaning, filtering, and deduplication of text, image, video, and audio data for pretraining and fine-tuning. Uses RAPIDS with Ray to scale across multi-node, multi-GPU clusters (the 2023 original used MPI and Dask). In one NVIDIA benchmark it deduplicated 1.96 trillion tokens in about 0.5 hours on 32 NVIDIA H100 GPUs, and NVIDIA's 2023 launch post reported GPU deduplication running roughly 20x faster and 5x cheaper than a CPU pipeline. [8][13][24] |
| NeMo Customizer | Fine-tuning service | Former microservice (sunset October 2026) for [fine-tuning](https://aiwiki.ai/wiki/fine_tuning) and adapting models, with methods such as LoRA, P-tuning, and [supervised fine-tuning](https://aiwiki.ai/wiki/supervised_fine-tuning). [6][7] |
| NeMo RL (successor to NeMo Aligner) | Post-training and alignment | Library for model [alignment](https://aiwiki.ai/wiki/ai_alignment) and reinforcement learning, including SFT, DPO, and GRPO, with Megatron Core or PyTorch-native AutoModel training backends, [vLLM](https://aiwiki.ai/wiki/vllm) generation, and Ray scheduling; it runs from a single GPU up to multi-node clusters. The older NeMo Aligner, which supported [reinforcement learning from human feedback](https://aiwiki.ai/wiki/rlhf), has not been maintained since May 15, 2025, and its repository is archived. [9][25] |
| NeMo Guardrails | Safety and control | Open source toolkit for adding programmable rails to LLM applications, covering input, output, dialog, retrieval, and execution rails. [10] |
| NeMo Retriever | Retrieval and RAG | Microservices for embedding, reranking, and multimodal extraction that power [retrieval augmented generation](https://aiwiki.ai/wiki/retrieval_augmented_generation) over enterprise data; NVIDIA now brands the retrieval models as Nemotron Retriever. NVIDIA reported up to 15x faster multimodal PDF extraction than open-source alternatives and 3x higher embedding throughput for its FP8 NIM than an FP16 open-source setup, and its Llama 3.2 embedding model was evaluated on 26 languages. [11][14][28] |
| NeMo Evaluator | Evaluation | Open-source library (formerly also a microservice, sunset October 2026) for benchmarking models on academic and custom datasets, including LLM-as-a-judge methods. [6][7] |
| NeMo Data Store | Artifact storage | Former microservice (sunset October 2026) that stored and managed datasets, models, and related artifacts across the workflow. [6][7] |

The components are designed to chain together into one workflow. A team can curate a dataset with Curator, train or fine-tune with the core framework and Customizer, align the result with NeMo RL, attach Guardrails, connect it to private data through Retriever, and check quality with Evaluator. The microservices versions share an API surface and run as containers, which is what lets them be wired into a continuous loop where a deployed model is evaluated, then fine-tuned again on fresh data, then redeployed. That data flywheel framing was how NVIDIA pitched the microservices for agentic applications that need to keep improving after launch, until it sunset them on October 1, 2026. [6][7][15]

## What does NeMo Guardrails do?

Guardrails is worth a closer look because safety sits at a different layer from training. NVIDIA open-sourced NeMo Guardrails on April 25, 2023, as a toolkit for keeping generative AI [chatbot](https://aiwiki.ai/wiki/chatbot)s accurate and on-topic. [16] The companion research paper describes the approach this way: "using a runtime inspired from dialogue management, NeMo Guardrails allows developers to add programmable rails to LLM applications, these are user-defined, independent of the underlying LLM, and interpretable." [17]

Rather than baking behavior into the weights, it inserts programmable rails between the application and the model. Input rails can reject or rewrite what a user sends, for example masking sensitive data. Dialog rails shape how the model is prompted and can hold it to a defined conversation flow. Retrieval rails act on the chunks pulled into a retrieval augmented generation pipeline. Execution rails govern the tools the model is allowed to call. Output rails check or alter the response before it reaches the user, covering things like content moderation, fact-checking, and jailbreak detection. Because it is provider-agnostic, Guardrails can sit in front of many different models rather than only NVIDIA's own, and it integrates with toolkits such as [LangChain](https://aiwiki.ai/wiki/langchain). [10][16][17]

## What are the Parakeet and Canary speech models?

The speech roots are still part of NeMo. It keeps dedicated collections for automatic speech recognition and text to speech, with model families and pretrained checkpoints for transcription, speech synthesis, and related tasks such as speaker identification and spoken language understanding. NeMo has been a common starting point for state-of-the-art ASR work, since the collections include strong baseline architectures and recipes that researchers can fine-tune on their own audio rather than training from scratch. [1][4]

NVIDIA ships two flagship ASR families built with NeMo. Parakeet is a line of high-throughput English and multilingual transcription models, and Canary is a multilingual recognition and translation line. Parakeet TDT 0.6B v2, a 600 million parameter FastConformer-TDT model released in 2025, posted a 6.05 average [word error rate](https://aiwiki.ai/wiki/word_error_rate) on the Hugging Face Open ASR Leaderboard and an RTFx of 3,386, meaning it transcribes audio roughly 3,386 times faster than real time at batch size 128, which NVIDIA said ranked it first on that leaderboard as of June 2025. [18][19] NVIDIA's Canary-Qwen-2.5B, released in June 2025, reported a record 5.63% word error rate on the English Open ASR Leaderboard, and in August 2025 NVIDIA released Parakeet V3 and Canary V2 with speech recognition and translation for 25 European languages. [21] These results are a reminder that NeMo was a serious ASR and TTS toolkit before the language model era and did not drop that focus when it broadened. The same modular design that shaped the original Neural Modules still shows up here, where audio, language, and acoustic pieces can be assembled into a pipeline. [1][4]

In 2026 NeMo's main GitHub repository was refocused on speech: the old NeMo repository URLs now redirect to github.com/NVIDIA-NeMo/Speech, whose README presents the project as NVIDIA NeMo Speech, says the repository "has pivoted to focus on audio, speech, and multimodal LLMs," and names v2.7.3 as the final NeMo release before the repository split, with NeMo Speech 3.0 as release v3.0.0. [21] Large language model and multimodal training now lives in separate libraries in the NVIDIA-NeMo GitHub organization, chiefly Megatron Bridge and NeMo AutoModel, alongside NeMo RL, Curator, Evaluator, Export-Deploy, Run, and Guardrails. [2][23] NVIDIA released [Nemotron 3.5 Transcribe](https://aiwiki.ai/wiki/nemotron_3_5_transcribe), a speech-augmented language model, through NeMo Speech on 28 October 2026. NVIDIA trained it with NeMo Speech on the NeMo Automodel backend, and NeMo Speech supplies both the vLLM plugin used to serve it and its fine-tuning toolkit. [22]

## How does NeMo fit with Nemotron, NIM, and NVIDIA AI Enterprise?

NeMo sits inside a larger NVIDIA software stack, and the parts are easy to mix up, so it helps to separate them. NeMo is the framework for building and customizing models. Nemotron is a family of NVIDIA models that are built with that framework. [NVIDIA NIM](https://aiwiki.ai/wiki/nvidia_nim), short for NVIDIA Inference Microservices, packages models as optimized containers for serving, and a model customized in NeMo can be deployed through NIM. NeMo models can also be exported to TensorRT-LLM for optimized inference. [1][3]

The whole collection is offered commercially through [NVIDIA AI Enterprise](https://aiwiki.ai/wiki/nvidia_ai_enterprise), which adds support, security updates, and a supported path to production for the NeMo framework (and, until their October 2026 sunset, the NeMo microservices). [7] The NeMo microservices reached general availability on April 23, 2025; NVIDIA named SAP and [ServiceNow](https://aiwiki.ai/wiki/servicenow) among the enterprise platforms using NeMo Retriever microservices, and Constellation Research reported that NVIDIA cited Amdocs telecom agents delivering a 64% improvement in average handling time. [15][20][26][27] NVIDIA sunset the microservices on October 1, 2026. [7] So a developer can use the open source NeMo libraries for free, or adopt supported NeMo framework containers and NIM under NVIDIA AI Enterprise when running in production. [1][3][7]

## Who uses NeMo?

NeMo is aimed at organizations that train or heavily customize their own models on NVIDIA hardware rather than only calling a hosted API. That includes research labs publishing new models, enterprises building domain-specific assistants and agents, and teams that need to fine-tune a [foundation model](https://aiwiki.ai/wiki/foundation_models) on private data and keep it inside their own environment for privacy or compliance reasons. NVIDIA itself uses NeMo to produce the Nemotron models, which makes the framework a working example of its own use rather than a tool NVIDIA only ships to others. The audience splits roughly in two. Researchers and individual developers tend to reach for the open source libraries directly. Larger companies that want a supported path lean on the NVIDIA AI Enterprise packaging (and, until their October 2026 sunset, the NeMo microservices) so they get vendor support and security updates instead of maintaining the stack on their own. [1][6][15]

## Limitations

NeMo is built for NVIDIA GPUs and the CUDA platform, so it is not a portable, hardware-neutral framework. Teams without access to capable NVIDIA hardware, especially the Hopper and newer GPUs needed for FP8 training, get less out of it. The framework is also large and aimed at people doing real training and customization, which makes it heavier to learn than a simple inference library or an API client. The full value shows up most for groups that actually pretrain, fine-tune, or align models at scale, while a team that only needs to call an existing model through an endpoint will find most of NeMo unnecessary. The NeMo name can also be confusing, since it has covered free open source libraries, commercial microservices (sunset in October 2026), and, after the 2026 reorganization, a family of separately named repositories such as NeMo Speech, Megatron Bridge, and NeMo RL. [1][7][23]

## References

1. NVIDIA. "NVIDIA/NeMo." GitHub. https://github.com/NVIDIA/NeMo
2. NVIDIA. "NeMo Framework Overview." NVIDIA NeMo Framework User Guide. https://docs.nvidia.com/nemo-framework/user-guide/latest/overview.html
3. NVIDIA. "NVIDIA NeMo Framework Megatron Backend." NGC Catalog (developer.nvidia.com/nemo-framework redirects here). https://catalog.ngc.nvidia.com/orgs/nvidia/containers/nemo
4. NVIDIA. "NeMo Toolkit." NVIDIA NeMo Framework User Guide. https://docs.nvidia.com/nemo-framework/user-guide/latest/nemotoolkit/index.html
5. NVIDIA. "Megatron-Core." NVIDIA Docs. https://docs.nvidia.com/megatron-core/index.html
6. NVIDIA. "NeMo Microservices." NVIDIA Docs. https://docs.nvidia.com/nemo/microservices/latest/about/index.html
7. NVIDIA. "NVIDIA NeMo Microservices." developer.nvidia.com. https://developer.nvidia.com/nemo-microservices
8. NVIDIA. "NVIDIA-NeMo/Curator." GitHub. https://github.com/NVIDIA-NeMo/Curator
9. NVIDIA. "NVIDIA-NeMo/RL." GitHub. https://github.com/NVIDIA-NeMo/RL
10. NVIDIA. "NVIDIA/NeMo-Guardrails." GitHub. https://github.com/NVIDIA/NeMo-Guardrails
11. NVIDIA. "NVIDIA NeMo Retriever." developer.nvidia.com (archived June 2, 2025). https://web.archive.org/web/20250602211240/https://developer.nvidia.com/nemo-retriever
12. Kuchaiev, Oleksii, et al. "NeMo: a toolkit for building AI applications using Neural Modules." arXiv:1909.09577, September 14, 2019. https://arxiv.org/abs/1909.09577
13. NVIDIA. "Curating Trillion-Token Datasets: Introducing NVIDIA NeMo Data Curator." NVIDIA Technical Blog. https://developer.nvidia.com/blog/curating-trillion-token-datasets-introducing-nemo-data-curator/
14. NVIDIA. "NVIDIA NeMo Retriever Delivers Accurate Multimodal PDF Data Extraction 15x Faster." NVIDIA Technical Blog. https://developer.nvidia.com/blog/nvidia-nemo-retriever-delivers-accurate-multimodal-pdf-data-extraction-15x-faster/
15. NVIDIA. "Enhance Your AI Agent with Data Flywheels Using NVIDIA NeMo Microservices." NVIDIA Technical Blog. https://developer.nvidia.com/blog/enhance-your-ai-agent-with-data-flywheels-using-nvidia-nemo-microservices/
16. NVIDIA. "Right on Track: NVIDIA Open-Source Software Helps Developers Add Guardrails to AI Chatbots." NVIDIA Blog, April 25, 2023. https://blogs.nvidia.com/blog/ai-chatbot-guardrails-nemo/
17. Rebedea, Traian, et al. "NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Rails." arXiv:2310.10501, October 16, 2023. https://arxiv.org/abs/2310.10501
18. NVIDIA. "nvidia/parakeet-tdt-0.6b-v2." Hugging Face model card. https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2
19. NVIDIA. "NVIDIA Speech AI Models Deliver Industry-Leading Accuracy and Performance." NVIDIA Technical Blog. https://developer.nvidia.com/blog/nvidia-speech-ai-models-deliver-industry-leading-accuracy-and-performance/
20. CIO. "Nvidia says NeMo microservices now generally available." April 23, 2025. https://www.cio.com/article/3968114/nvidia-says-nemo-microservices-now-generally-available.html
21. NVIDIA, "NVIDIA-NeMo/Speech" GitHub repository README. Retrieved 29 October 2026. https://github.com/NVIDIA-NeMo/Speech
22. NVIDIA, "NVIDIA-Nemotron-3.5-Transcribe-30B-A4B" model card, Hugging Face. Retrieved 29 October 2026. https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Transcribe-30B-A4B
23. NVIDIA. "NVIDIA-NeMo." GitHub organization. Retrieved 2 November 2026. https://github.com/NVIDIA-NeMo
24. NVIDIA. "Key Features." NVIDIA NeMo Curator documentation. https://docs.nvidia.com/nemo/curator/latest/about/key-features.html
25. NVIDIA. "NVIDIA/NeMo-Aligner" (archived repository). GitHub. https://github.com/NVIDIA/NeMo-Aligner
26. Conway, Joey. "Enterprises Onboard AI Teammates Faster With NVIDIA NeMo Tools to Scale Employee Productivity." NVIDIA Blog, April 23, 2025. https://blogs.nvidia.com/blog/nemo-enterprises-ai-teammates-employee-productivity/
27. Dignan, Larry. "Nvidia NeMo Microservices generally available, aims for AI agent data flywheel." Constellation Research, April 23, 2025. https://www.constellationr.com/insights/news/nvidia-nemo-microservices-generally-available-aims-ai-agent-data-flywheel
28. NVIDIA. "nvidia/llama-3.2-nv-embedqa-1b-v2." Hugging Face model card. https://huggingface.co/nvidia/llama-3.2-nv-embedqa-1b-v2

