Can RTX 5090 run Phi-4 Mini 3.8B?

Yes — runs locally

~168 tok/sec · Instant — feels like typing. No noticeable delay.

Your VRAM

32 GB

Model size

3.8B

Best quant

Q8_0

VRAM needed

4.3 GB

The verdict

The RTX 5090 (32 GB VRAM) handles Phi-4 Mini 3.8B comfortably using the Q8_0 quantization, which fits in 4.3 GB. Expected throughput is around 168 tokens/second, which feels Instant — feels like typing. No noticeable delay. in interactive use. Latest Phi mini with strong reasoning. Drop-in upgrade from Phi-3.5 Mini.

Setup tutorial: Phi-4 Mini 3.8B on RTX 5090

AI-generated, GPU-specific. Verified commands for your exact hardware.

TL;DR

Run Phi-4 Mini 3.8B on an NVIDIA GeForce RTX 5090 with Ollama using the Q8_0 quantization. Expect Grade S performance at ~354 tok/sec.

Prerequisites

Before starting, ensure you have at least 10GB of free disk space, a 64-bit version of Windows or Linux, and the latest NVIDIA drivers (version 525.60.13 or later) with CUDA 11.8 installed.

Expected performance

With the Q8_0 quantization, expect the model to run at approximately 354 tokens per second, utilizing around 4.3GB of VRAM. Given the remaining 27.7GB of VRAM, you can achieve a practical context window of up to 131,072 tokens, making it suitable for long-form content generation and complex reasoning tasks.

1. Install runtimeOllama

pip install ollama
ollama config set cuda

2. Download the model

Download the Q8_0 quantized version of Phi-4 Mini 3.8B (3.8GB file) from Hugging Face.

ollama pull bartowski/microsoft_Phi-4-mini-instruct-GGUF:microsoft_Phi-4-mini-instruct-Q8_0.gguf

3. Run it

ollama run --model=microsoft_Phi-4-mini-instruct-Q8_0.gguf --n-gpu-layers=32 --flash-attn
ollama chat --model=microsoft_Phi-4-mini-instruct-Q8_0.gguf

4. Optimize for RTX 5090

For optimal performance on the NVIDIA GeForce RTX 5090 with 32GB VRAM, use --n-gpu-layers=32 to offload layers to the GPU, enable --flash-attn for faster attention computations, and consider using tensor parallelism if running multiple instances. This configuration will utilize approximately 4.3GB of VRAM, leaving 27.7GB available for larger context windows.

Troubleshooting

Out of memory error during inference

Reduce the number of GPU layers with --n-gpu-layers=16 or increase the batch size.

Slow token generation speed

Ensure that --flash-attn is enabled and try increasing the number of GPU layers with --n-gpu-layers=32.

Model fails to load

Verify the integrity of the downloaded model file and try re-downloading it using the same command.

Alternative runtimes

While Ollama is recommended for its ease of use and performance, you can also run Phi-4 Mini 3.8B using alternative runtimes like LM Studio for a more graphical interface, or llama.cpp for lower-level control. Jan is another option if you need more customization options, especially for research purposes.

Full Phi-4 Mini 3.8B details →

Other models that run great on RTX 5090

FAQ (20)

What GPU do I need to run Phi-4 Mini 3.8B?

To run Phi-4 Mini 3.8B, you need a GPU with at least 2.8 GB of VRAM, but 4.3 GB is recommended for optimal performance, especially with higher quantization levels.

Is Phi-4 Mini 3.8B good for coding?

Yes, Phi-4 Mini 3.8B is well-suited for coding tasks due to its strong reasoning capabilities and large context length of 131,072 tokens, which allows it to handle complex code snippets and documentation.

Phi-4 Mini 3.8B vs Llama 3.1 8B?

Phi-4 Mini 3.8B has fewer parameters (3.8B vs 8B) but is more efficient in terms of VRAM usage and performance, making it a better choice for systems with limited resources. It also offers a larger context length of 131,072 tokens compared to Llama 3.1 8B.

Can I run Phi-4 Mini 3.8B on a Mac?

Yes, you can run Phi-4 Mini 3.8B on a Mac, provided your Mac has a compatible GPU with at least 2.8 GB of VRAM. Ensure you have the necessary drivers and software installed for optimal performance.

How much VRAM does Phi-4 Mini 3.8B need?

Phi-4 Mini 3.8B requires between 2.8 GB and 4.3 GB of VRAM, depending on the quantization level used. Higher quantization levels generally require more VRAM but offer better performance.

Is Phi-4 Mini 3.8B censored?

Phi-4 Mini 3.8B is not inherently censored, but it may include content filters or safeguards to prevent the generation of harmful or inappropriate content, as is common in many AI models.

Is Phi-4 Mini 3.8B commercial-use allowed?

Yes, Phi-4 Mini 3.8B is licensed under the MIT License, which allows for both personal and commercial use without additional restrictions.

Phi-4 Mini 3.8B context length?

Phi-4 Mini 3.8B has a context length of 131,072 tokens, which is significantly larger than many other models, allowing it to process and generate longer sequences of text.

Want personalized recommendations for your exact setup? Detect my hardware →