Can RTX 4090 run Phi-3.5 MoE?
Yes — runs locally
~28 tok/sec · Good — slight pause, then text streams smoothly.
The verdict
The RTX 4090 (24 GB VRAM) handles Phi-3.5 MoE comfortably using the Q4_K_M quantization, which fits in 24.1 GB. Expected throughput is around 28 tokens/second, which feels Good — slight pause, then text streams smoothly. in interactive use. Microsoft MoE — 16 experts of 3.8 B, 6.6 B active per token. Strong reasoning at modest cost.
Setup tutorial: Phi-3.5 MoE on RTX 4090
AI-generated, GPU-specific. Verified commands for your exact hardware.
Phi-3.5 MoE runs on an NVIDIA GeForce RTX 4090 with a Grade C performance, using the Q4_K_M quantization, achieving ~26 tokens per second.
Prerequisites
Before starting, ensure you have at least 25GB of free disk space, a 64-bit version of Windows or Linux, the latest NVIDIA drivers (version 525.60.12 or later), and CUDA 11.8 installed.
Expected performance
With the Q4_K_M quantization, you can expect the model to run at approximately 26 tokens per second, using around 24.1GB of VRAM. This leaves a headroom of about -0.1GB for context, allowing for a full context window of 131072 tokens with minimal overhead.
1. Install runtimeOllama
pip install ollama
ollama init2. Download the model
Download the Phi-3.5 MoE Q4_K_M quantized model (23.6GB) from the Hugging Face repository.
ollama pull bartowski/Phi-3.5-MoE-instruct-GGUF:Phi-3.5-MoE-instruct-Q4_K_M.gguf3. Run it
ollama run Phi-3.5-MoE-instruct-Q4_K_M --n-gpu-layers 32 --flash-attn
ollama chat Phi-3.5-MoE-instruct-Q4_K_M4. Optimize for RTX 4090
For optimal performance on the NVIDIA GeForce RTX 4090 with 24GB VRAM, set --n-gpu-layers to 32 to utilize most of the GPU memory. Enable flash attention (--flash-attn) to reduce memory usage and improve speed. Given the 24GB VRAM, you can achieve a practical context window of up to 131072 tokens with a small margin for overhead.
Troubleshooting
Out of memory error during inference
Reduce the number of layers offloaded to the GPU using --n-gpu-layers <N>, where <N> is a lower value such as 24.
Slow inference speed
Ensure that flash attention is enabled with --flash-attn and that the CUDA toolkit is up to date.
Model fails to load
Check the integrity of the downloaded model file and try re-downloading it.
Alternative runtimes
Alternative runtimes include LM Studio and llama.cpp. LM Studio offers a more user-friendly interface and is suitable for those who prefer a graphical environment. llama.cpp provides more fine-grained control over model execution and is ideal for advanced users or those needing specific optimizations. Jan is another option for running models, but it may require additional configuration steps compared to Ollama.
Other models that run great on RTX 4090
FAQ (20)
What GPU do I need to run Phi-3.5 MoE?
To run Phi-3.5 MoE, you need a GPU with at least 24.1 GB of VRAM, such as an NVIDIA RTX 3090 or A6000.
Is Phi-3.5 MoE good for coding?
Phi-3.5 MoE is well-suited for coding tasks due to its strong reasoning capabilities and large context length of 131,072 tokens.
Phi-3.5 MoE vs Llama 3.1 8B?
Phi-3.5 MoE has 41.9 billion parameters compared to Llama 3.1 8B's 8 billion, offering more sophisticated reasoning and context handling but requiring significantly more VRAM.
Can I run Phi-3.5 MoE on a Mac?
Yes, you can run Phi-3.5 MoE on a Mac with a compatible GPU that has at least 24.1 GB of VRAM, such as an eGPU setup.
How much VRAM does Phi-3.5 MoE need?
Phi-3.5 MoE requires 24.1 GB of VRAM, which is consistent across different quantization levels.
Is Phi-3.5 MoE censored?
Phi-3.5 MoE is not inherently censored, but its responses may be influenced by the training data and any filters applied during deployment.
Is Phi-3.5 MoE commercial-use allowed?
Yes, Phi-3.5 MoE is licensed under the MIT License, allowing for commercial use without additional restrictions.
Phi-3.5 MoE context length?
Phi-3.5 MoE has a context length of 131,072 tokens, which is significantly larger than many other models, enabling it to handle longer and more complex inputs.
Want personalized recommendations for your exact setup? Detect my hardware →