Can RTX 4080 run Phi-3.5 MoE?
Yes — runs locally
~0 tok/sec · Cannot run — insufficient VRAM
The verdict
The RTX 4080 (16 GB VRAM) handles Phi-3.5 MoE comfortably using the Q4_K_M quantization, which fits in 24.1 GB. Expected throughput is around 0 tokens/second, which feels Cannot run — insufficient VRAM in interactive use. Microsoft MoE — 16 experts of 3.8 B, 6.6 B active per token. Strong reasoning at modest cost.
How to run it
- 1. Install Ollama or LM Studio.
- 2. Pull the
Q4_K_MGGUF — best balance of quality and speed on 16 GB. - 3. Start chatting. Expect ~0 tok/sec on first-token, faster after warmup.
Other models that run great on RTX 4080
FAQ (20)
What GPU do I need to run Phi-3.5 MoE?
To run Phi-3.5 MoE, you need a GPU with at least 24.1 GB of VRAM, such as an NVIDIA RTX 3090 or A6000.
Is Phi-3.5 MoE good for coding?
Phi-3.5 MoE is well-suited for coding tasks due to its strong reasoning capabilities and large context length of 131,072 tokens.
Phi-3.5 MoE vs Llama 3.1 8B?
Phi-3.5 MoE has 41.9 billion parameters compared to Llama 3.1 8B's 8 billion, offering more sophisticated reasoning and context handling but requiring significantly more VRAM.
Can I run Phi-3.5 MoE on a Mac?
Yes, you can run Phi-3.5 MoE on a Mac with a compatible GPU that has at least 24.1 GB of VRAM, such as an eGPU setup.
How much VRAM does Phi-3.5 MoE need?
Phi-3.5 MoE requires 24.1 GB of VRAM, which is consistent across different quantization levels.
Is Phi-3.5 MoE censored?
Phi-3.5 MoE is not inherently censored, but its responses may be influenced by the training data and any filters applied during deployment.
Is Phi-3.5 MoE commercial-use allowed?
Yes, Phi-3.5 MoE is licensed under the MIT License, allowing for commercial use without additional restrictions.
Phi-3.5 MoE context length?
Phi-3.5 MoE has a context length of 131,072 tokens, which is significantly larger than many other models, enabling it to handle longer and more complex inputs.
Want personalized recommendations for your exact setup? Detect my hardware →