VRAM
40GB
Generation
ampere
MSRP
$10,000
Vendor
nvidia
Runs Perfectly (S/A/B)
108
With Offloading (C/D)
1
Cannot Run (F)
0
S
Excellent (108 models)
Runs great with room for large context windows
Qwen 2.5 32B
Alibaba
32BChat / LLMQ4_K_M19.0GB
Gemma 3 27B
27BChat / LLMQ4_K_M15.9GB
Mistral Small 22B
Mistral AI
22BChat / LLMQ4_K_M12.9GB
Phi-4
Microsoft
14BChat / LLMQ8_015.0GB
Qwen 2.5 14B
Alibaba
14BChat / LLMQ8_015.1GB
Qwen 2.5 Coder 14B
Alibaba
14BCodingQ8_015.1GB
Code Llama 13B Instruct
Meta
13BCodingQ4_K_M7.8GB
Gemma 3 12B
12BChat / LLMQ8_012.2GB
Mistral Nemo 12B
Mistral AI
12BChat / LLMQ8_012.6GB
FLUX.1 Schnell (GGUF)
Black Forest Labs
12BImage GenerationQ5_014.0GB
FLUX.1 Dev (GGUF)
Black Forest Labs
12BImage GenerationQ5_014.0GB
Solar 10.7B
Upstage
10.7BChat / LLMQ8_011.1GB
Falcon 3 10B
TII
10BChat / LLMQ8_010.7GB
Gemma 2 9B Instruct
9.2BChat / LLMQ8_09.7GB
Yi 1.5 9B Chat
01.AI
9BChat / LLMQ8_09.2GB
Yi Coder 9B
01.AI
9BCodingQ8_09.2GB
CodeGemma 7B
8.5BCodingQ8_08.9GB
DeepSeek R1 Distill 8B
DeepSeek
8BChat / LLMQ8_08.4GB
Llama 3.1 8B Instruct
Meta
8BChat / LLMFP1617.0GB
Granite 3.3 8B
IBM
8BChat / LLMQ8_08.6GB
EXAONE 3.5 7.8B
LG AI
7.8BChat / LLMQ8_08.2GB
InternLM 2.5 7B
Shanghai AI Lab
7.7BChat / LLMQ8_08.2GB
Qwen 2.5 7B Instruct
Alibaba
7.6BChat / LLMQ8_09.0GB
Qwen 2.5 Coder 7B
Alibaba
7.6BCodingQ8_08.0GB
Mistral 7B Instruct v0.3
Mistral AI
7.3BChat / LLMFP1615.5GB
LLaVA 1.6 7B
LLaVA
7BMultimodalQ8_08.5GB
Falcon 3 7B
TII
7BChat / LLMQ8_08.3GB
OLMo 2 7B
Allen AI
7BChat / LLMQ8_07.7GB
OpenChat 3.5 7B
OpenChat
7BChat / LLMQ8_07.7GB
StarCoder2 7B
BigCode
7BCodingQ8_07.6GB
Code Llama 7B
Meta
7BCodingQ8_07.2GB
DeepSeek Coder 6.7B
DeepSeek
6.7BCodingQ8_07.2GB
Yi 1.5 6B Chat
01.AI
6BChat / LLMQ8_06.5GB
Phi-3.5 Vision
Microsoft
4.2BMultimodalQ4_K_M3.2GB
Gemma 3 4B
4BChat / LLMQ8_04.3GB
Nemotron Mini 4B
NVIDIA
4BChat / LLMQ8_04.7GB
Danube 3 4B
H2O.ai
4BChat / LLMQ8_04.4GB
Phi-3.5 Mini 3.8B
Microsoft
3.8BChat / LLMQ8_04.3GB
Phi-4 Mini 3.8B
Microsoft
3.8BChat / LLMQ8_04.3GB
Stable Diffusion XL (CoreML)
Stability AI
3.5BImage GenerationCoreML3.3GB
SDXL Turbo (GGUF)
Stability AI
3.5BImage GenerationQ5_05.0GB
Llama 3.2 3B Instruct
Meta
3.2BChat / LLMQ8_03.7GB
Qwen 2.5 3B
Alibaba
3BChat / LLMQ8_03.9GB
Qwen 2.5 Coder 3B
Alibaba
3BCodingQ8_03.9GB
Falcon 3 3B
TII
3BChat / LLMQ8_03.8GB
StableLM Zephyr 3B
Stability AI
3BChat / LLMQ8_03.3GB
Rocket 3B
Pansophic
3BChat / LLMQ8_03.3GB
StarCoder2 3B
BigCode
3BCodingQ8_03.5GB
Stable Code 3B
Stability AI
3BCodingQ8_03.3GB
PaliGemma 3B
3BMultimodalQ4_K_M2.5GB
Gemma 2 2B
2.6BChat / LLMQ8_03.1GB
Stable Diffusion 3 Medium (GGUF)
Stability AI
2.5BImage GenerationQ8_09.2GB
EXAONE 3.5 2.4B
LG AI
2.4BChat / LLMQ8_03.1GB
Qwen2-VL 2B
Alibaba
2.2BMultimodalQ8_02.0GB
CodeGemma 2B
2BCodingQ8_03.0GB
MiniCPM-V 2.6
OpenBMB
2BMultimodalQ8_03.0GB
Granite 3.3 2B
IBM
2BChat / LLMQ8_03.0GB
Moondream 2
Moondream
1.8BMultimodalQ4_K_M1.5GB
SmolLM2 1.7B
HuggingFace
1.7BChat / LLMQ8_02.2GB
Whisper Large v3
OpenAI
1.55BSpeech RecognitionQ8_03.4GB
Qwen 2.5 1.5B
Alibaba
1.5BChat / LLMQ8_02.3GB
DeepSeek R1 Distill 1.5B
DeepSeek
1.5BChat / LLMQ8_02.3GB
Qwen 2.5 Coder 1.5B
Alibaba
1.5BCodingQ8_02.3GB
Yi Coder 1.5B
01.AI
1.5BCodingQ8_02.0GB
DeepSeek Coder 1.3B
DeepSeek
1.3BCodingQ8_01.8GB
Llama 3.2 1B Instruct
Meta
1.24BChat / LLMFP162.8GB
TinyLlama 1.1B
TinyLlama
1.1BChat / LLMQ8_01.6GB
Gemma 3 1B
1BChat / LLMQ8_01.5GB
Falcon 3 1B
TII
1BChat / LLMQ8_02.2GB
Stable Diffusion 2.1 Base (CoreML)
Stability AI / Apple
0.86BImage GenerationCoreML-Palettized1.6GB
Stable Diffusion 1.5 (CoreML)
Runway
0.86BImage GenerationCoreML-Palettized2.5GB
Stable Diffusion 1.5 (GGUF)
Runway / GPUStack
0.86BImage GenerationQ8_02.3GB
Stable Diffusion 2.1 (GGUF)
Stability AI
0.86BImage GenerationQ8_02.7GB
Whisper Large v3 Turbo
OpenAI
0.81BSpeech RecognitionQ8_02.0GB
Whisper Medium
OpenAI
0.77BSpeech RecognitionQ8_01.9GB
Distil-Whisper Large v3
HuggingFace
0.76BSpeech RecognitionQ8_01.9GB
BGE Reranker v2 M3
BAAI
0.568BRerankerFP161.6GB
Qwen 2.5 0.5B
Alibaba
0.5BChat / LLMQ8_01.1GB
Qwen 2.5 Coder 0.5B
Alibaba
0.5BCodingQ8_01.1GB
Danube 3 500M
H2O.ai
0.5BChat / LLMQ8_01.0GB
SmolLM2 360M
HuggingFace
0.36BChat / LLMQ8_00.9GB
BGE Large EN v1.5
BAAI
0.335BEmbeddingFP161.1GB
MusicGen Small
Meta
0.3BAudio GenerationONNX-Q4F160.8GB
Whisper Small
OpenAI
0.24BSpeech RecognitionQ8_00.9GB
Nomic Embed Text v1.5
Nomic AI
0.137BEmbeddingFP160.8GB
SmolLM2 135M
HuggingFace
0.135BChat / LLMFP160.8GB
Kokoro 82M TTS
Kokoro
0.082BText to SpeechONNX-Q8F160.6GB
Whisper Base
OpenAI
0.074BSpeech RecognitionQ8_00.3GB
Whisper Base English
OpenAI
0.074BSpeech RecognitionQ8_00.3GB
Whisper Tiny English (Quantized)
OpenAI
0.039BSpeech RecognitionQ5_10.1GB
Whisper Tiny
OpenAI
0.039BSpeech RecognitionQ8_00.2GB
BGE Small EN v1.5
BAAI
0.033BEmbeddingQ8_00.1GB
Snowflake Arctic Embed S
Snowflake
0.033BEmbeddingQ8_00.1GB
Jina Reranker Tiny EN
Jina AI
0.033BRerankerFP160.1GB
all-MiniLM-L6-v2
Sentence Transformers
0.023BEmbeddingQ8_00.1GB
Piper TTS - Amy (English)
Rhasspy
0.02BText to SpeechONNX0.1GB
Piper TTS - Lessac (English)
Rhasspy
0.02BText to SpeechONNX0.1GB
Piper TTS - LibriTTS-R (English)
Rhasspy
0.02BText to SpeechONNX0.6GB
Piper TTS - Spanish (MLS)
Rhasspy
0.02BText to SpeechONNX0.1GB
Piper TTS - French (Siwis)
Rhasspy
0.02BText to SpeechONNX0.5GB
Piper TTS - German (Thorsten)
Rhasspy
0.02BText to SpeechONNX0.1GB
Piper TTS - Chinese (Huayan)
Rhasspy
0.02BText to SpeechONNX0.1GB
Piper TTS - Japanese (Kokoro)
Rhasspy
0.02BText to SpeechONNX0.1GB
Piper TTS - Korean
Rhasspy
0.02BText to SpeechONNX0.1GB
Piper TTS - Russian (Irina)
Rhasspy
0.02BText to SpeechONNX0.1GB
Piper TTS - Portuguese (Faber)
Rhasspy
0.02BText to SpeechONNX0.1GB
Piper TTS - Italian (Riccardo)
Rhasspy
0.02BText to SpeechONNX0.5GB
Piper TTS - Arabic (Kareem)
Rhasspy
0.02BText to SpeechONNX0.1GB
C
Possible (1 model)
Needs partial CPU offloading, slower performance
Similar GPUs
Frequently Asked Questions
What AI models can I run on the NVIDIA A100 40GB?
The NVIDIA A100 40GB with 40GB VRAM can run 108 AI models smoothly (grade B or better), including Qwen 2.5 32B, Gemma 3 27B, Mistral Small 22B and 105 more.
How much VRAM does the NVIDIA A100 40GB have?
The NVIDIA A100 40GB has 40GB of VRAM, which is sufficient for running many local AI models.
Is the NVIDIA A100 40GB good for AI?
With 40GB VRAM, the NVIDIA A100 40GB can run 108 out of 109 models in our database at grade B or better. This is an excellent choice for local AI workloads.