pythia-6.9b
The Pythia Scaling Suite is a collection of models developed to facilitate interpretability research (see paper). It contains two sets of eight models of sizes 70M, 160M, 410M, 1B, 1.4B, 2.8B, 6.9B, and 12B. For each size, there are two models: one trained on the Pile, and one trained on the Pile after the dataset has been globally deduplicated. All 8 model sizes are trained on the exact same data, in the exact same...
Params
7.0 B
Context
2,048
Downloads 30d
757 K
Likes
66
Download history
daily snapshots · 10 days760 K287 K
Sep 11Sep 14Sep 17Sep 20
Can you run it?
Estimated VRAM at 8K context unless noted. Pick your hardware to see the verdict per quantization.
| File | Quant | Size | Est. VRAM | Verdict on RTX 4090 · 24 GB |
|---|---|---|---|---|
| model.safetensors | f16 | 13.8 GB | 16.8 GB | ✅ Runs comfortably |
Estimate: file size × 1.1 + KV cache at 8K + 0.5 GB overhead. Not a benchmark — how we calculate this.
Run it
copy-paste, exact tags checked against the Hub$ curl -s https://aimodelscomparison.com/api/v1/models/pythia-6-9b
{
"hf_id": "EleutherAI/pythia-6.9b",
"params_b": 6.99,
"context_length": 2048,
"license": { "id": "apache-2.0", "commercial": "yes" },
"downloads_30d": 756930,
"vram_estimates": [
{ "quant": "f16", "gb": 16.8 }
],
"updated_at": "2026-09-11T01:00:23Z"
}
Specifications
- Architecture
- GPTNeoXForCausalLM
- Parameters
- 7.0 B
- Tensor type
- F16
- Context length
- 2,048
- Vocabulary
- 50,432
- Layers / heads
- 32 / 32
- Licence
- apache-2.0
- First seen on the Hub
- 2023-02-14
- Training datasets
- EleutherAI/pile
- Added to our catalog
- 2026-09-11
Compare with any text-generation model
Popular comparisons
Appears in