Can I run Phi-3.5 MoE on my device?

Phi-3.5 MoE requires a minimum of 26GB VRAM. Use RunThisModel to check your specific hardware compatibility and find the best quantization for your device.

How much VRAM does Phi-3.5 MoE need?

Phi-3.5 MoE needs 26GB VRAM at minimum (Q4_K_M quantization). Higher quality quantizations need more: Q4_K_M: 26GB.

How do I download Phi-3.5 MoE?

You can download Phi-3.5 MoE in GGUF format from HuggingFace (24GB minimum). Use the RunThisModel iOS app to download and run it directly on your device, or download manually from HuggingFace.

Can Phi-3.5 MoE run on iPhone?

Phi-3.5 MoE at 41.9B parameters is too large for most iPhones. Consider using an iPad with M-series chip or Mac with Apple Silicon.

Microsoft

Phi-3.5 MoE

Name: Phi-3.5 MoE
Author: Microsoft

Microsoft MoE — 16 experts of 3.8 B, 6.6 B active per token. Strong reasoning at modest cost.

41.9B parametersphimoemit128K context26GB - 26GB VRAM

About This Model

Phi-3.5 MoE from Microsoft is a 16-expert Mixture-of-Experts variant of Phi-3.5. Hits MMLU 78.9 with only 6.6 B parameters firing per token, which is remarkable. The 26 GB VRAM bar at Q4 puts it just above the consumer 24 GB sweet spot — comfortable on RTX A6000 or 32 GB+ Apple Silicon.

Check Your Hardware

See which quantizations of Phi-3.5 MoE your hardware can run.

Quantization Options

Quantization	Bits	File Size	VRAM Needed	RAM Needed	Quality
Q4_K_M	4.5	24 GB	26 GB	32 GB	85%

Context window & KV cache

Adds 2.50 GB to VRAM

Long chats and RAG inputs cost real memory. Drag to see how 32K vs 128K context shifts your grade.

Model native max: 128K tokens. KV-cache estimate is approximate (±30 %); real usage depends on attention layout.

How to run Phi-3.5 MoE

Pick a runtime — copy & paste. Commands are pre-filled with this model’s repo.

GUI. Browse → download → chat. MLX on Apple Silicon.

LM Studio home →

1
Open LM Studio
Go to the 🔍 Search tab.
2
Search for
```
bartowski/Phi-3.5-MoE-instruct-GGUF
```
3
Download
Pick the Q4_K_M quant — best balance of size vs. quality.
4
Chat
Hit ▶ Load Model and start chatting. Toggle 'Local Server' to expose an OpenAI-compatible API on :1234.

Community benchmarks

Real tokens/sec reports from people running Phi-3.5 MoE on actual hardware.

No community runs yet for this model. Be the first to submit your numbers.

Download & Run

HuggingFace

View model & download weights

Ollama

One-command install & run

See It In Action

Real model outputs generated via RunThisModel.com — watch responses stream in real time.

Llama 3.3 70B responding...

Outputs generated by real AI models via RunThisModel.com. Generation speed shown is from cloud inference. Local speeds vary by hardware — check your device.

Frequently Asked Questions

How much VRAM do I need to run Phi-3.5 MoE?

Phi-3.5 MoE requires 26GB VRAM minimum with Q4_K_M quantization. For full precision, you need 26GB VRAM.

What is the best quantization for Phi-3.5 MoE?

Q4_K_M offers the best balance of quality and VRAM usage. Q8_0 is near-lossless if you have enough VRAM.