> For the complete documentation index, see [llms.txt](https://docs.understand.tech/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.understand.tech/documentation/english-version/ai-in-a-box/hardware-experiments/nemotron-3.5-bolt-74b-a9b.md).

# Nemotron 3.5 Bolt 74B-A9B

A hybrid Mamba-Transformer mixture-of-experts: 74B parameters, 9B active per token, quantized to mixed NVFP4 and FP8. Measured on RTX PRO 6000 Blackwell with vLLM 0.32.0.

`MoE` `74B / 9B active` `262,144 ctx` `vLLM 0.32.0` `NVFP4 + FP8`

<a href="https://huggingface.co/primitive-ai/Nemotron-3.5-Bolt-74B-A9B-mixed-NVFP4-FP8" class="button secondary">View on Hugging Face</a>

## At a glance

| Hardware             | TP | Weights per GPU |      KV cache | Speed for 1 user | Chat at 25 tok/s | Agentic at 40 tok/s |
| -------------------- | -: | --------------: | ------------: | ---------------: | ---------------: | ------------------: |
| RTX PRO 6000 · 96 GB |  1 |        44.0 GiB |  4.0 M tokens |      121.7 tok/s |           29.8\* |               9.4\* |
| RTX PRO 6000 · 96 GB |  2 |        22.5 GiB | 13.3 M tokens |      150.9 tok/s |             33.2 |                10.6 |

## Serve

The command used for the measurements. Pick the configuration in the tabs.

{% tabs %}
{% tab title="RTX PRO 6000 · TP=1" %}
{% code title="RTX PRO 6000 · TP=1 · NVFP4 + FP8" overflow="wrap" %}

```shellscript
VLLM_USE_V2_MODEL_RUNNER=1 vllm serve primitive-ai/Nemotron-3.5-Bolt-74B-A9B-mixed-NVFP4-FP8 \
  --max-model-len 262144 \
  --max-num-seqs 512 \
  --gpu-memory-utilization 0.92 \
  --enable-prefix-caching \
  --moe-backend b12x \
  --attention-backend FLASHINFER \
  --mamba-backend flashinfer \
  --mamba-cache-mode align \
  --limit-mm-per-prompt '{"image":1,"video":0}' \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser-plugin <model directory>/ultra_v3_reasoning_parser.py \
  --reasoning-parser ultra_v3 \
  --trust-remote-code
```

{% endcode %}

<figure><picture><source srcset="https://2654650065-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FXgNPw9jE3xyWIcnvYeCB%2Fuploads%2Fgit-blob-0486a92f9afd61803af4386c1432a940480d470f%2Fmemory-bolt-tp1-en-dark.png?alt=media" media="(prefers-color-scheme: dark)"><img src="https://2654650065-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FXgNPw9jE3xyWIcnvYeCB%2Fuploads%2Fgit-blob-6a0e0cca372c71c9527aace2cd08f5d85cdab2ba%2Fmemory-bolt-tp1-en-light.png?alt=media" alt="Bolt on RTX PRO 6000, TP=1: memory per GPU split into weights, KV cache, working memory, CUDA context and free memory"></picture><figcaption></figcaption></figure>

#### Capacity per instance

| Workload | Sessions at 40 tok/s | Sessions at 25 tok/s | Ceiling | Speed at the ceiling |
| -------- | -------------------: | -------------------: | ------: | -------------------: |
| Chat     |               14.4\* |               29.8\* |      64 |           14.7 tok/s |
| Agentic  |                9.4\* |               18.1\* |      48 |           12.3 tok/s |
| Code     |               14.1\* |               25.3\* |      32 |           20.7 tok/s |

\* Confirmed by 3 repeats: speed spread under 3%, sessions under 3.5%. Agentic and code are measured with the session history already cached.

|                    |                  |
| ------------------ | ---------------- |
| Speed for 1 user   | 121.7 tok/s      |
| KV cache           | 3,993,951 tokens |
| Prefix-cache block | 2,112 tokens     |
| Start-up           | 3 to 5 min       |

* **Prompt computation is the limit, not memory.** Throughput tops out near 870 tok/s with the cache only 30% full.
* **Cap agentic sessions at 32.** With the history cached, throughput still rises up to 48 sessions, but each user gets 12.3 tok/s there. With a cap of 32 sessions and 64 requests, an admitted session keeps 17.1 tok/s (9.8 without a cap) and the others wait 21 s for their first token (p95). At 128 requests the wait exceeds 2 minutes and throughput drops by 38%: put a queue in front of the engine.
* **Close to the model card.** 107.6 tok/s for 1 request and 572 tok/s for 32, against 112.7 and 607 on the model card (−4.5% and −5.7%).

<details>

<summary>Measured speed by number of sessions</summary>

| Sessions | Chat: tok/s per user | Chat: tok/s total | Chat: first token p95 | Agentic: tok/s per user | Agentic: tok/s total | Agentic: first token p95 |
| -------: | -------------------: | ----------------: | --------------------: | ----------------------: | -------------------: | -----------------------: |
|        1 |                121.7 |               112 |                 0.6 s |                   121.6 |                   96 |                    0.6 s |
|        8 |                 55.1 |               421 |                 1.0 s |                    44.5 |                  331 |                    0.9 s |
|       16 |                 38.3 |               600 |                 0.8 s |                    28.1 |                  430 |                    0.6 s |
|       24 |                 29.3 |               697 |                 0.6 s |                       – |                    – |                        – |
|       32 |                 24.0 |               794 |                 0.6 s |                    17.8 |                  509 |                    3.5 s |
|       48 |                    – |                 – |                     – |                    12.3 |                  540 |                    7.2 s |
|       64 |                 14.7 |               859 |                 9.2 s |                     9.8 |                  567 |                   12.9 s |

</details>
{% endtab %}

{% tab title="RTX PRO 6000 · TP=2" %}
{% code title="RTX PRO 6000 · TP=2 · NVFP4 + FP8" overflow="wrap" %}

```shellscript
VLLM_USE_V2_MODEL_RUNNER=1 vllm serve primitive-ai/Nemotron-3.5-Bolt-74B-A9B-mixed-NVFP4-FP8 \
  --tensor-parallel-size 2 \
  --max-model-len 262144 \
  --max-num-seqs 512 \
  --gpu-memory-utilization 0.92 \
  --enable-prefix-caching \
  --moe-backend b12x \
  --attention-backend FLASHINFER \
  --mamba-backend flashinfer \
  --mamba-cache-mode align \
  --limit-mm-per-prompt '{"image":1,"video":0}' \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_coder \
  --reasoning-parser-plugin <model directory>/ultra_v3_reasoning_parser.py \
  --reasoning-parser ultra_v3 \
  --trust-remote-code
```

{% endcode %}

<figure><picture><source srcset="https://2654650065-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FXgNPw9jE3xyWIcnvYeCB%2Fuploads%2Fgit-blob-6146b21316ac31f49a887b01815e7a4d96d379f0%2Fmemory-bolt-tp2-en-dark.png?alt=media" media="(prefers-color-scheme: dark)"><img src="https://2654650065-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FXgNPw9jE3xyWIcnvYeCB%2Fuploads%2Fgit-blob-55862da03757826efe4b3f9ff67d4fc01b9dccdd%2Fmemory-bolt-tp2-en-light.png?alt=media" alt="Bolt on RTX PRO 6000, TP=2: memory per GPU split into weights, KV cache, working memory, CUDA context and free memory"></picture><figcaption></figcaption></figure>

#### Capacity per instance

| Workload | Sessions at 40 tok/s | Sessions at 25 tok/s | Ceiling | Speed at the ceiling |
| -------- | -------------------: | -------------------: | ------: | -------------------: |
| Chat     |                 17.2 |                 33.2 |      32 |           25.7 tok/s |
| Agentic  |                 10.6 |                 21.1 |      32 |           18.6 tok/s |
| Code     |                 20.8 |                 ≥ 32 |    ≥ 32 |           27.6 tok/s |

|                    |                                                         |
| ------------------ | ------------------------------------------------------- |
| Speed for 1 user   | 150.9 tok/s                                             |
| KV cache           | 13,339,384 tokens                                       |
| Prefix-cache block | 2,112 tokens                                            |
| Start-up           | about 20 min the first time, mostly loading the weights |

* **Faster for one user, not per GPU.** One user gets 150.9 tok/s (+24%), but a TP=1 instance on each of the two GPUs serves 67% to 80% more chat sessions. Bolt is best run in TP=1.
* **Same throughput ceiling.** About 856 tok/s in chat, as on a single GPU.

<details>

<summary>Measured speed by number of sessions</summary>

| Sessions | Chat: tok/s per user | Chat: tok/s total | Chat: first token p95 | Agentic: tok/s per user | Agentic: tok/s total | Agentic: first token p95 |
| -------: | -------------------: | ----------------: | --------------------: | ----------------------: | -------------------: | -----------------------: |
|        1 |                150.9 |               127 |                 0.7 s |                   150.9 |                  127 |                    0.7 s |
|        8 |                 62.8 |               467 |                 1.2 s |                    48.1 |                  357 |                    1.1 s |
|       16 |                 41.9 |               643 |                 1.2 s |                    29.7 |                  450 |                    1.1 s |
|       32 |                 25.7 |               839 |                 1.9 s |                    18.6 |                  509 |                    4.6 s |
|       48 |                    – |                 – |                     – |                    12.4 |                  511 |                    9.7 s |
|       64 |                 14.7 |               856 |                16.7 s |                       – |                    – |                        – |

</details>
{% endtab %}
{% endtabs %}

## Variant

| Checkpoint                                                                                                                              | Precision   | Size    | Context tested |
| --------------------------------------------------------------------------------------------------------------------------------------- | ----------- | ------- | -------------- |
| [primitive-ai/Nemotron-3.5-Bolt-74B-A9B-mixed-NVFP4-FP8](https://huggingface.co/primitive-ai/Nemotron-3.5-Bolt-74B-A9B-mixed-NVFP4-FP8) | NVFP4 + FP8 | 46.7 GB | 262,144 tokens |

The `--reasoning-parser-plugin` file ships with the model repository. KV cache measured at 3,993,951 tokens on one GPU, against 4,074,466 on the model card.

## Strategy

|                           |                                                  |                                                                                                         |
| ------------------------- | ------------------------------------------------ | ------------------------------------------------------------------------------------------------------- |
| **Tensor parallel, TP=1** | One instance per GPU.                            | Most sessions per GPU, in every workload. The recommended setting.                                      |
| **Tensor parallel, TP=2** | One instance across two GPUs of the same server. | Faster for one user (+24%) and 3.3 times more cache. TP=1 serves 67% to 80% more chat sessions per GPU. |

Workloads, load procedure and definitions: [How we measure](https://docs.understand.tech/documentation/english-version/ai-in-a-box/hardware-experiments.md#how-we-measure).
