MaziyarPanahi / text-generation updated 1 year ago

Llama-3.3-70B-Instruct-GGUF

Name: Llama-3.3-70B-Instruct-GGUF
Author: MaziyarPanahi

MaziyarPanahi/Llama-3.3-70B-Instruct-GGUF contains GGUF format model files for meta-llama/Llama-3.3-70B-Instruct.

Params

—

Context

—

Downloads 30d

179 K

Likes

Licence: unknown Not gated GGUF View on Hugging Face ↗

Download history

daily snapshots · 33 days

▲ 9 K in the last 30 days (4.7%)

188 K179 K

Aug 19Aug 30Sep 10Sep 20

Estimated VRAM at 8K context unless noted. Pick your hardware to see the verdict per quantization.

File	Quant	Size	Est. VRAM	Verdict on RTX 4090 · 24 GB
Llama-3.3-70B-Instruct.Q2_K.gguf	Q2_K	26.4 GB	29.5 GB	❌ Won’t fit
Llama-3.3-70B-Instruct.Q3_K_S.gguf	Q3_K_S	30.9 GB	34.5 GB	❌ Won’t fit
Llama-3.3-70B-Instruct.Q3_K_M.gguf	Q3_K_M	34.3 GB	38.2 GB	❌ Won’t fit
Llama-3.3-70B-Instruct.Q3_K_L.gguf	Q3_K_L	37.1 GB	41.4 GB	❌ Won’t fit
Llama-3.3-70B-Instruct.Q4_K_S.gguf	Q4_K_S	40.3 GB	44.9 GB	❌ Won’t fit
Llama-3.3-70B-Instruct.Q4_K_M.gguf	Q4_K_M	42.5 GB	47.3 GB	❌ Won’t fit
Llama-3.3-70B-Instruct.Q5_K_S.gguf	Q5_K_S	48.7 GB	54.0 GB	❌ Won’t fit
Llama-3.3-70B-Instruct.Q5_K_M.gguf	Q5_K_M	49.9 GB	55.4 GB	❌ Won’t fit

Estimate: file size × 1.1 + KV cache at 8K + 0.5 GB overhead. Not a benchmark — how we calculate this.

copy-paste, exact tags checked against the Hub

~ · ollama · Q4_K_M

$ ollama run llama-3-3-70b-instruct-gguf

# pin the quantization explicitly
$ ollama run llama-3-3-70b-instruct-gguf-q4_k_m

est. VRAM 47.3 GBon RTX 4090 · 24 GBJSON API →

Base model and the most-downloaded derivatives in the catalog.

Compare with any text-generation model