Can RTX 5080 run Phi-3.5 MoE?
Yes — runs locally
~0 tok/sec · Cannot run — insufficient VRAM
The verdict
The RTX 5080 (16 GB VRAM) handles Phi-3.5 MoE comfortably using the Q4_K_M quantization, which fits in 24.1 GB. Expected throughput is around 0 tokens/second, which feels Cannot run — insufficient VRAM in interactive use. Microsoft MoE — 16 experts of 3.8 B, 6.6 B active per token. Strong reasoning at modest cost.
Setup tutorial: Phi-3.5 MoE on RTX 5080
AI-generated, GPU-specific. Verified commands for your exact hardware.
Run Phi-3.5 MoE on a NVIDIA GeForce RTX 5080 with Q4_K_M quantization. Expect ~17 tok/sec and Grade D performance.
Prerequisites
Before starting, ensure you have at least 23.6GB of free disk space, a compatible operating system (Windows or Linux), and the latest NVIDIA drivers (version 525.60.13 or later) with CUDA 11.8 installed.
Expected performance
With the Q4_K_M quantization, expect ~17 tok/sec and 24.1GB VRAM usage. Given the 16GB VRAM limit, you will have -8.1GB of headroom, which may restrict the practical context window to around 65536 tokens to avoid out-of-memory errors.
1. Install runtimeOllama
pip install ollama
ollama init2. Download the model
Download the Phi-3.5 MoE Q4_K_M quantized model (23.6GB) from Hugging Face.
ollama pull bartowski/Phi-3.5-MoE-instruct-GGUF:Phi-3.5-MoE-instruct-Q4_K_M.gguf3. Run it
ollama run Phi-3.5-MoE-instruct-Q4_K_M.gguf --n-gpu-layers 28 --flash-attn --tensor-parallel 1
ollama chat Phi-3.5-MoE-instruct-Q4_K_M.gguf4. Optimize for RTX 5080
For optimal performance on the NVIDIA GeForce RTX 5080 with 16GB VRAM, set --n-gpu-layers to 28 to maximize GPU utilization while keeping within VRAM limits. Enable --flash-attn for faster attention computation and set --tensor-parallel to 1 for single-GPU operation. This configuration should help achieve the target ~17 tok/sec.
Troubleshooting
Out of memory error during inference
Reduce --n-gpu-layers to 24 or lower and decrease the context window to 32768 tokens.
Slow token generation speed
Ensure CUDA is properly installed and update NVIDIA drivers to the latest version.
Model fails to load
Verify the integrity of the downloaded model file and try re-downloading it.
Alternative runtimes
Consider using LM Studio for a more user-friendly interface, llama.cpp for better performance on smaller models, or Jan for advanced features like multi-GPU support. Ollama is recommended for its ease of use and compatibility with large models like Phi-3.5 MoE.
Other models that run great on RTX 5080
FAQ (20)
What GPU do I need to run Phi-3.5 MoE?
To run Phi-3.5 MoE, you need a GPU with at least 24.1 GB of VRAM, such as an NVIDIA RTX 3090 or A6000.
Is Phi-3.5 MoE good for coding?
Phi-3.5 MoE is well-suited for coding tasks due to its strong reasoning capabilities and large context length of 131,072 tokens.
Phi-3.5 MoE vs Llama 3.1 8B?
Phi-3.5 MoE has 41.9 billion parameters compared to Llama 3.1 8B's 8 billion, offering more sophisticated reasoning and context handling but requiring significantly more VRAM.
Can I run Phi-3.5 MoE on a Mac?
Yes, you can run Phi-3.5 MoE on a Mac with a compatible GPU that has at least 24.1 GB of VRAM, such as an eGPU setup.
How much VRAM does Phi-3.5 MoE need?
Phi-3.5 MoE requires 24.1 GB of VRAM, which is consistent across different quantization levels.
Is Phi-3.5 MoE censored?
Phi-3.5 MoE is not inherently censored, but its responses may be influenced by the training data and any filters applied during deployment.
Is Phi-3.5 MoE commercial-use allowed?
Yes, Phi-3.5 MoE is licensed under the MIT License, allowing for commercial use without additional restrictions.
Phi-3.5 MoE context length?
Phi-3.5 MoE has a context length of 131,072 tokens, which is significantly larger than many other models, enabling it to handle longer and more complex inputs.
Want personalized recommendations for your exact setup? Detect my hardware →