Build Real-Time, Multi-Speaker AI with NVIDIA Nemotron 3 Diarization**

Why speaker diarization matters

Every conversation carries two layers of information: what was said and who said it. Speech recognition captures and transcribes the words. Speaker diarization classifies who spoke when, helping applications connect what was said to the right participant.

Consider a transcript from a meeting, customer call, or podcast in which every sentence is correct but none is attributed to a speaker. You can read the words, but you cannot reliably tell who made a commitment, who raised an objection, or which participant interrupted. Search, summaries, action items, conversation analytics, and voice-agent memory all become less useful.

Speaker diarization identifies the time intervals during which each speaker is active, including intervals where people talk over one another. Those speaker timestamps can then be combined with automatic speech recognition (ASR) to create a speaker-attributed transcript.

NVIDIA Nemotron 3 Diarization is an open-weight, 100M-parameter model that ranks #1 on Voice Arena’s Diarization-Bench leaderboard with a 14.72% Diarization Error Rate (DER). Supporting up to eight speakers across live and recorded conversations, it handles overlapping speech, chunked processing for flexible recording lengths, and customizable streaming latency.

Earlier models like NVIDIA Streaming Sortformer established this approach for four-speaker diarization, including the streaming diar_streaming_sortformer_4spk-v2.1 checkpoint used as the baseline below. Nemotron 3 Diarization expands support to eight speakers and improves the accuracy and throughput measured in the following evaluations.

How Nemotron 3 Diarization works

One model for offline and streaming conversations

Diarization systems must solve two related problems. First, they must detect speech and assign it to the correct speaker. Second, they must preserve that assignment throughout the conversation, even after silence, interruptions, or long gaps between a speaker’s turns.

Streaming makes the second problem harder. An offline model can inspect an entire recording at once. A streaming system receives only a small chunk of new audio and limited context. Without an effective memory mechanism, the speaker assigned to one output channel in the current chunk can be assigned to a different channel in the next.

Nemotron 3 Diarization follows the Sortformer approach of ordering output speakers by when they first appear. The first new voice becomes the first speaker channel, the next new voice becomes the second, and so on. This arrival-time ordering makes the model’s generic speaker labels stable and removes the need to solve a new speaker permutation for every chunk.

Nemotron 3 Diarization was trained using public and licensed speech data, including multispeaker-annotated, real-world conversations licensed from David AI. Additional licensed David AI audio provided source material for large-scale simulated English and multilingual mixtures spanning 21 languages. Adding David AI data to our training decreased compound diarization error rate (DER) by 0.77 absolute points, from 11.19% to 10.42%, at both the offline-style and ultra-low-latency operating points.

The model supports as many as eight speaker channels. These are anonymous labels, not real-world identities: the model can report that speaker_2 spoke from one timestamp to another, but it does not determine that speaker_2 is a particular person. Downstream applications can map these anonymous channel IDs to explicit speaker identities by pairing timestamps with meeting metadata, user profiles, or active speaker verification models.

From audio to speaker activity

Architecture flow for Nemotron 3 Diarization
The model accepts 16 kHz, single-channel audio and converts it into Mel-spectrogram features with a 10 ms frame step. It stacks those features by a factor of eight, producing 80 ms frames for a 31-layer Transformer encoder with rotary positional embeddings (RoPE).

Figure 1. Nemotron 3 Diarization converts audio into arrival-ordered speaker-activity probabilities. AOSC and FIFO context support streaming inference.

Above the Transformer, a Conv1D layer upsamples the predictions to the input-feature resolution. The default output is a [T, 8] floating-point tensor: T time steps by eight possible speaker channels. Each value is the probability that a speaker is active at that time. The default stride is 10 ms and can be configured to another multiple of 10 ms.

This representation handles overlap naturally. If two people speak at the same time, two channels can be active in the same frame. Postprocessing converts these probabilities into generic speaker labels with start and end timestamps.

During streaming inference, two forms of memory provide context:

• The Arrival-Order Speaker Cache (AOSC) retains useful information about speakers observed in earlier chunks, organized according to their arrival-ordered channels.
• A first-in, first-out (FIFO) queue supplies recent frame context before the current chunk.

The input buffer also includes right context, which is audio immediately after the current chunk. More right context can help the model interpret speaker transitions, while less right context reduces the time it must wait before producing a result. Together, the current chunk, right context, FIFO queue, and speaker cache enable one model to operate at several latency points.

Chunked inference removes a fixed model-imposed maximum audio duration. Performance can still degrade on unusually long recordings or audio with severe noise, reverberation, far-field capture, or domain shift.

Diarization and Speaker Attributed ASR (Multi-speaker ASR) are different tasks

Standalone diarization produces speaker activity and timestamps, not the words being spoken. ASR produces text but does not necessarily preserve speaker attribution. A speaker-attributed transcription pipeline combines both outputs:

Figure 2. End-to-end speaker-attributed transcription pipeline combining audio diarization timestamps with automatic speech recognition (ASR) to map spoken words to specific speakers.

That separation matters when designing a system. Diarization errors include missed speech, false speech detections, incorrect speaker assignments, and boundary errors. ASR errors affect the words. Applications should evaluate both components and the combined pipeline on their intended audio.

Balance latency and accuracy

The same model supports recommended input-buffer latencies of 30.4, 1.04, 0.64, and 0.32 seconds. Shorter buffers let the system respond sooner, while more context generally improves accuracy and throughput. These values measure the audio buffered before inference; computation, networking, ASR, and application processing add to end-to-end latency. Although the model can technically use an 80 ms input buffer, 0.32 seconds is the lowest recommended configuration. The configuration table in “Get started” shows how to select an operating point.

Benchmark results: Ranked #1 in Voice Arena’s initial Diarization-Bench

In Voice Arena’s initial Diarization-Bench results, NVIDIA Nemotron 3 Diarization ranked first among 12 systems and 17 total system configurations evaluated across 139 English-language conversations totaling approximately 22 hours. With overlapping speech scored, system-generated speech activity detection, and no boundary collar, Nemotron 3 Diarization achieved a 14.72% diarization error rate (DER), compared with 19.3% for the next-ranked system – a ~24% relative reduction. It also ranked first with 100 ms and 250 ms collars and across both in-person and online recordings. These initial results may change as Voice Arena completes its Version 1 evaluation and paired statistical analysis.

Figure 3. Voice Arena Diarization-Bench v1 results (English, 0 ms collar, DER lower is better), showing NVIDIA Nemotron 3 Diarization ranking #1 with 14.72% DER.

The figure below uses the Nemotron 3 Diarization evaluation results. They compare the model with NVIDIA’s previous four-speaker Streaming Sortformer under the disclosed scoring and throughput settings.

Figure 4. Diarization Error Rate (DER) performance across eight public benchmarks at 1.04-second output latency, showing Nemotron 3 Diarization state-of-the-art results compared to the previous NVIDIA Sortformer baseline.

Measuring diarization accuracy

The primary metric used to evaluate the model is diarization error rate (DER). It combines three kinds of error:

• Missed speech: a reference speaker was active, but the system detected no corresponding speech.
• False alarm: the system marked a speaker as active when the reference contained no corresponding speech.
• Speaker confusion: the system detected speech at the right time but assigned it to the wrong speaker.

Figure 5. DER adds missed speech, false alarms, and speaker confusion, then divides by total reference speaker time. Overlapping reference speakers each contribute to the denominator.

Benchmark settings can materially change DER, so the evaluation protocol is part of the result. The Nemotron 3 evaluation contains 901 condition-specific recordings spanning multilingual telephone speech, meetings, near-field and far-field microphones, multi-microphone capture, and difficult acoustic environments. Overlapping speech is scored in every evaluation.

DIHARD III, AliMeeting, AMI, and NOTSOFAR1 use a zero-second collar, meaning no boundary tolerance is excluded from scoring. CALLHOME-Part2 uses a 0.25-second collar. Results were generated with the NeMo e2e_diarize_speech.py evaluation script.

The comparison below uses diar_streaming_sortformer_4spk-v2.1, NVIDIA’s previous four-speaker streaming Sortformer, as the baseline. It uses the final Nemotron-3-Diarization values rather than preview-model results.

An average 40% relative DER reduction at 1.04-second latency

Figure 6. Full-set DER for the final model and previous NVIDIA baseline at 1.04-second input-buffer latency. Lower is better.

At 1.04-second input-buffer latency, Nemotron 3 Diarization reduces DER on all eight listed evaluation conditions. The relative reductions range from 9.0% on CALLHOME-Part2 to 65.2% on NOTSOFAR1 MHM.

The unweighted mean of the eight per-dataset relative reductions is 41.0%. In other words, this is an average of relative improvements across evaluation conditions; it is not a pooled DER computed by combining every recording into one score.

The improvement is also consistent across operating points. At each latency shared by the two models (30.4, 1.04, and 0.32 seconds), the final model has lower full-set DER on every evaluated dataset.

Figure 7. Full-set DER across input-buffer latency settings. The previous baseline does not have a 0.64-second configuration.

Improvements increase in higher-speaker-count conditions

Support for eight speakers enables meetings and group conversations with more than four participants. The benchmark advantage also widens in the higher-speaker-count subsets of DIHARD III, CALLHOME-Part2, and NOTSOFAR1.

Figure 8. DER at 30.4-second input-buffer latency, broken out by speaker count. Shaded regions contain more than four speakers.

The figure also preserves an important nuance: on the two-speaker CALLHOME subset, the final model records 5.98% DER compared with 5.68% for the previous baseline. Across the full CALLHOME-Part2 evaluation, however, DER improves from 10.32% to 9.10%, with larger gains in the higher-speaker-count subsets.

DIHARD III groups recordings with five through nine speakers into one result. Nine speakers exceed Nemotron 3 Diarization’s supported maximum of eight, so that aggregate includes audio outside the specified speaker-count limit.

Accuracy and throughput

A deployable diarization system must balance accuracy and throughput. The model card reports real-time factor speedup (RTFx), calculated as total audio duration divided by total processing time. Higher RTFx means the system processes more audio per unit of compute time.

Figure 9. DIHARD III full-set DER versus batch-size-32 compiled throughput. Results use BF16 with the NeMo PyTorch backend on an NVIDIA RTX PRO 5000.

At the 30.4-second configuration, Nemotron 3 Diarization reaches 15,113× RTFx at batch size 32 with torch.compile(), compared with 2,619× for the previous baseline, while lowering DIHARD III DER from 19.09% to 12.73%. At the 1.04-second configuration, it reaches 865× compared with 136×, while lowering DER from 19.60% to 13.18%.

These results measure batched throughput on the disclosed test system. They should not be interpreted as single-stream, end-to-end application latency. Developers should benchmark the complete pipeline on their target hardware, including data movement, diarization, ASR, and downstream processing.

See Nemotron 3 Diarization in action

Explore the Nemotron Diarization live model demo to connect the model’s speaker timestamps with a conversation you can follow. The application offers synthetic conversations, an eight-speaker mode, live microphone input, and stress-test scenarios. We also added a live mic for a multlingual ASR models, so you can perform live streaming diarization with speakers of different languages.

Start in Conversation with a preloaded topic, then follow the speaker activity and live transcript as the conversation plays. Listen for interruptions and compare the timing of the voices with the displayed speaker activity. For your own conversation, use Live Mic or the Multilingual Live Mic. Finally you can upload an audio file you have prerecorded in the Audio File tab.

The following invented exchange illustrates the difference that speaker attribution makes; it is not a measured output from the demo:

With speaker labels, an application can associate the commitment with the same participant who answers the follow-up. The timeline adds information that plain text cannot show: two speakers can be active at once. A downstream summarizer can use the attributed transcript to extract action items while retaining the source speaker and timestamps.

The prepared scenarios use synthetic audio. Some eight-speaker scenarios provide a short speaker-context primer before the audible conversation; they illustrate behavior with that context and are not an unprimed benchmark. Participant names and roles belong to the application’s scenario logic. Nemotron 3 Diarization supplies generic speaker channels and timestamps, not identity verification.

Bring real-time, speaker-aware transcription on device

Argmax has added support for NVIDIA Nemotron 3 Diarization in Argmax Pro SDK 3, enabling real-time speaker attribution for conversations with up to eight speakers, and introduced a pre-diarized transcription API that separates speakers before speech recognition, designed to improve transcription in complex conversations with overlapping speech.

Watch the demo to see Nemotron 3 Diarization running with Argmax Pro SDK 3.

Get started with NVIDIA NeMo Speech

Install the dependencies

Install the system packages and NVIDIA NeMo speech dependencies after setting up Python 3.12 or later, Cython, and a recent PyTorch version.

apt-get update && apt-get install -y libsndfile1 ffmpeg
uv pip install Cython packaging
uv pip install 'nemo-toolkit[asr]'

Implement streaming diarization with streaming ASR (multi-speaker ASR) in real-world scenarios

To evaluate ASR and diarization (multi-speaker ASR) performance, refer to the Quick start guide to Diarization with ASR.

Run an offline diarization on a recording

Use a 16 kHz mono recording containing two or more speakers, and replace /path/to/conversation.wav with its path. The following example loads the checkpoint and uses the recommended 30.4-second offline-style configuration.

from nemo.collections.asr.models import SortformerEncLabelModel

diar_model = SortformerEncLabelModel.from_pretrained(
    "nvidia/Nemotron-3-Diarization"
)
diar_model.eval()


diar_model.sortformer_modules.spkcache_len = 264
diar_model.sortformer_modules.fifo_len = 40
diar_model.sortformer_modules.chunk_len = 340
diar_model.sortformer_modules.chunk_right_context = 40
diar_model.sortformer_modules.spkcache_update_period = 300
diar_model._check_streaming_parameters()

predicted_segments = diar_model.diarize(
    audio=["/path/to/conversation.wav"],
    batch_size=1,
)

for segment in predicted_segments[0]:
    print(segment)

The diarize() method returns speaker-marked segments as strings in the form start_seconds end_seconds speaker_id. For example, the output structure can look like this (illustrative timestamps, not a measured model result):

0.400 2.100 speaker_0
1.800 3.250 speaker_1
3.600 4.700 speaker_0

Here, both speakers are active between 1.800 and 2.100 seconds. A speaker can appear in multiple segments, and the intervals need not be mutually exclusive. To obtain the underlying speaker-activity tensors as well, set include_tensor_outputs=True.

The API accepts an audio path, a list of paths, NumPy arrays, or a line-delimited JSON manifest. When providing NumPy arrays, pass the correct integer sample_rate; the default file-audio sample rate is 16 kHz. Nemotron 3 Diarization supports 16 kHz, single-channel .wav, .flac, .opus, and .mp3 audio. It is designed for Linux systems with supported NVIDIA Ampere, Hopper, or Blackwell GPUs.

Choose the latency-quality operating point

Nemotron 3 Diarization exposes streaming parameters in units of 80 ms encoder frames. The recommended configurations cover offline-style processing through ultra-low-latency streaming.

The latency values in this table are input-buffer latency, calculated as:(CHUNK_LEN + RIGHT_CONTEXT) × 80 ms

They do not include model computation, network transport, ASR, or application processing. Although the model can technically use an input buffer as short as 80 ms, 0.32 seconds is the lowest recommended configuration. Lowering latency generally reduces both accuracy and throughput, so developers should select the operating point against end-to-end product requirements rather than buffer duration alone.

Use all five parameter values from the same row in the inference example, then call _check_streaming_parameters() before running diarization. The speaker cache retains earlier speaker context; FIFO controls the recent history; chunk and right context set the input-buffer latency; the cache update period controls how much FIFO context is used for a cache update.

Combine diarization with Offline ASR

Run ASR and diarization on the same recording and time base. For example, You can use Nemotron ASR 3.5 or Parakeet TDT 0.6B v3 that can return word timestamps. The following minimal offline example reuses predicted_segments from above and associates each word with the speaker active at its midpoint:

from nemo.collections.asr.models import ASRModel

asr_model = ASRModel.from_pretrained(
    model_name="nvidia/parakeet-tdt-0.6b-v3"
)
words = asr_model.transcribe(
    ["/path/to/conversation.wav"], timestamps=True
)[0].timestamp["word"]

turns = []
for segment in predicted_segments[0]:
    start, end, speaker = segment.split()
    turns.append((float(start), float(end), speaker))

def speaker_at(midpoint):
    active = sorted({
        speaker for start, end, speaker in turns if start <= midpoint < end
    })
    if len(active) == 1:
        return active[0]
    return "overlap/ambiguous" if active else "unassigned"

for word in words:
    midpoint = (word["start"] + word["end"]) / 2
    label = speaker_at(midpoint)
    print(f"{word['start']:.2f}-{word['end']:.2f} {label}: {word['word']}")

This midpoint rule is a simple alignment heuristic. It marks simultaneous speaker activity as ambiguous and leaves words outside detected speech unassigned. It does not separate overlapping voices or determine which active speaker produced an ASR word. For a production transcript, evaluate word-boundary alignment, overlap handling, and both models’ errors on representative audio before combining adjacent words into speaker turns.

Deployment considerations

The model supports a maximum of eight speakers. If a recording contains more, speech can be missed or assigned to the wrong channel. Noise, severe reverberation, far-field recording, domain shift, and long conversations can also increase missed speech, false alarms, boundary errors, or speaker confusion.

Downstream applications should preserve useful uncertainty instead of treating every speaker assignment as infallible. Evaluate the complete system on audio representative of the intended environment, especially before using speaker attribution in regulated, safety-relevant, or consequential workflows.

Use of the model is governed by the OpenMDW License Agreement, version 1.1.

Resources and next steps

Build with NVIDIA ecosystem partners