Self-hosting
80B Models on 4GB RAM: The Real Numbers
An 80-billion-parameter model running in 4GB of RAM sounds like a forum scam from 2019. It isn't anymore. Two open-source runtimes do exactly this today: Swiftlet on Apple devices, AirLLM on basically any GPU. What the headlines leave out is the speed column. That column is the whole decision, so this piece is mostly that column.
Short answer
- Yes, it's real. Qwen3-Next-80B-A3B in 4-bit runs at 4.3GB peak RAM and 4.5–5 tok/s on an M5 Mac, per Swiftlet's own benchmark table (checked 2026-10-12). The 42GB of weights stream from SSD; only the dense core sits in memory.
- The catches: an iPhone 17 runs the 35B at about 1 tok/s. A base M1 MacBook does 2.45 tok/s. And AirLLM, which fits a full 70B into 4GB of VRAM with zero quantization, publishes no tok/s figure at all. That silence is data.
- My read: with 16GB+ of Mac RAM and disk to spare, this is genuinely usable today for chat and drafting. Below that, it's a demo. For daily production work, hosted APIs still win on every axis except privacy.
Why 4GB can hold 80B at all
Both projects exploit the same property of sparse MoE models. Qwen3-Next-80B-A3B has 80B parameters total, but each token activates only about 3B of them. Each layer routes every token to 10 of its 512 experts (the 35B model uses 8 of 256). So at any instant you need the dense core, the parts that run on every token, plus whichever experts this token touches.
Swiftlet keeps the dense core, attention, routers, shared experts and embeddings, resident at about 2.5GB for the 80B at 4-bit, repacks the tens of thousands of routed experts into fixed-stride blobs (a .qpack file), and streams experts from SSD as the router asks for them. AirLLM streams whole layers instead of whole experts. Same insight, different granularity. That one design choice is the reason one project gets to talk about tok/s and the other doesn't.
The pattern worth internalizing, stated plainly in Swiftlet's README: decode speed tracks active parameters, not container size. The 80B and the 35B decode in the same ballpark because both activate ~3B per token. The bigger container mostly costs you disk, not time. This inverts twenty years of local-LLM intuition where model size and speed were the same number.
The Swiftlet numbers, straight from the README
Swiftlet's benchmark table, fetched 2026-10-12. All speeds are the project's self-reported figures on an M5 Mac:
| Model | Disk | Peak RAM | Decode speed |
|---|---|---|---|
| Qwen3.6-35B-A3B, 4-bit | 18 GB | 2.6 GB | 7–11 tok/s |
| Qwen3.6-35B-A3B, 8-bit | 34 GB | 7.6 GB | 3.5–4 tok/s |
| Qwen3-Next-80B-A3B, 4-bit | 42 GB | 4.3 GB | 4.5–5 tok/s |
Two low-end anchors from the same README: a base M1 (8-core GPU, 16GB) decodes the 4-bit 35B at about 2.45 tok/s and the 8-bit at about 1.74. Prefill runs at roughly decode speed on that machine, so a long system prompt is genuinely expensive, and 80B on a base M1 is not a plan, it's a threat.
On an iPhone 17 (base model, not Pro), the 35B runs in about 2.5GB of RAM at about 1 tok/s, with the README honestly appending "today". It ships through the Priv AI app on the App Store as an experimental model. One token per second is readable if you are patient in a way no one is. It's a proof of life, not a product.
One tier note for Mac users: the 8-bit 35B is what the README calls the quality tier. Tested head to head, it removes the repetition artifacts the 4-bit build can show on longer writing, at the cost of disk, RAM and roughly half the speed. If you've used heavily quantized models for drafting, you know exactly the failure mode this fixes.
The most honest sentence in either README deserves repeating, because no marketing deck will ever carry it: with only ~3B parameters active per token, these models chat and write like large models but recall facts like small ones. That is the physics of what you're buying. Fluent prose, 3B-grade trivia.
A prediction I'm willing to put on record: Swiftlet says its decode loop is dispatch bound, not IO bound. Meaning the bottleneck is kernel scheduling, not the SSD, which means it's a software problem, and software problems get faster. I expect these numbers to look dated within three months, in the good direction.
AirLLM: 70B in 4GB, and one missing number
AirLLM takes the other road: layer-wise streaming with no quantization at all. Its pitch is a 70B model running on a single 4GB GPU card, unquantized, undistilled, unpruned, and the same playbook scales absurdly: Kimi K3 (2.8T parameters) under 4GB, DeepSeek-V3 (671B) in about 12GB.
The September 2026 addition is Qwen3.8-Flash-Next (125B MoE): 5.95GB of VRAM end to end on one RTX 4090, at the price of about 360GB of checkpoint disk and a 64GB host for the file-mapped n-gram embedding table. AirLLM now also trains at that scale, streaming frozen weights layer by layer and keeping adapters on the GPU: a 125B LoRA under 6GB, a Qwen3.8-27B in about 2GB.
Now the number you cannot find anywhere in that README: tokens per second. AirLLM doesn't publish one. I don't think that's an oversight. Layer streaming moves every layer's weights for every token, and the tax of that shows up in the one metric they'd have to defend. When you read "2.8T model under 4GB", the verb is fits, not runs well. The project's own speed story is a 3x speedup via optional block-wise quantization, which quietly walks back the no-quantization headline.
To be fair to AirLLM: for air-gapped boxes, offline kiosks, boats and anything where "the model must physically be here" is a hard requirement, "it fits and it's correct" beats "it's fast and it isn't there". That's a real niche, and 36.7k stars say it's a bigger niche than cynics like me assume.
Two roads, side by side
| Swiftlet | AirLLM | |
|---|---|---|
| Strategy | 4-bit quantize + stream experts | No quantization + stream layers |
| Hardware | Apple Silicon, macOS 14+ / iOS 17+ | CUDA cards (MPS experimental) |
| Granularity | Per-expert (~3B active) | Per-layer (whole layers per token) |
| Speed story | 4.5–5 tok/s on 80B (M5) | Unpublished |
| Maturity | 657 stars, Aug 2026, v0-era | 36.7k stars, Nov 2023, v4.0 |
They aren't really competitors. Swiftlet bets that on unified-memory Apple hardware with only ~3B active per token, expert streaming can be fast enough to be interactive. AirLLM bets that correctness without quantization is worth any speed, on any GPU you already own. If you have both a Mac and a spare CUDA card, Swiftlet is the one you'd actually use and AirLLM is the one you'd use to prove a point.
Who should actually bother
A 16GB+ Mac with 42GB of SSD to spare: yes, tonight. The 80B at 4.5–5 tok/s is a workable drafting pace, slower than a fast reader but comfortably inside chat territory, and the 4-bit 35B at 7–11 tok/s on the same machine is properly pleasant. Keep the 3B-recall limitation in mind and don't ask it for the capital of anything obscure.
An iPhone or an 8GB Mac: the 1–2.45 tok/s tier is for demonstrating to yourself that it works. If you want a daily-driver local model on Apple gear, smaller dense models via Ollama's MLX default remain the sane answer, and the hardware math for going bigger is in our self-hosting hardware piece.
A 4GB GPU: AirLLM will let you verify the 70B runs. You will then go sign up for an API anyway, and that's fine, the verification was the deliverable.
Production or daily work: be honest about why you're doing this locally. It isn't money. A hosted DeepSeek V4 API or GLM-5.3-Flash costs so little per million tokens that "saving money" stopped being a local-inference argument somewhere in 2025. The real reasons that survive are privacy, offline and regulatory, and those are legitimate. Everyone whose reason is the bill should close this tab and read which free API tiers actually stay free instead. If local is truly the requirement and the model must be frontier-grade, the budget line starts where DGX Station for Windows sits, not where 4GB sits.
The 2.4T question, for context
Alibaba's Qwen3.8-2.4T-A95B, the largest open-weight model released to date, landed in August. Per vLLM's deployment recipes, the datacenter answer for its 2.4T parameters and ~95B active is 24 GPUs across three 8x B300 nodes. Nobody is streaming that from an iPhone SSD this decade; 95B active is roughly 30x the active load Swiftlet handles today. But direction matters more than the current number. ANEMLL already demonstrated a 397B MoE streaming on an iPhone 17 Pro as a proof of concept, per Swiftlet's README credit. The runtimes are getting good faster than the models are getting small, and the next batch of A3B-class models looks likely to be the first that's genuinely daily-usable on ordinary hardware. When a 4GB machine holds a model that recalls like a big one, the local/API calculus changes for good. Not this year.
All Swiftlet and AirLLM figures above are the projects' self-reported numbers from their GitHub READMEs, fetched 2026-10-12. Benchmark speeds are measured on their hardware (M5 Mac, RTX 4090), not independently verified by us; treat them as ceilings until you measure on your own machine. Qwen3.8-2.4T-A95B deployment figures are as reported in vLLM's recipes. Hugging Face was unreachable from our network, so nothing here relies on HF pages. Both projects iterate weekly, so check the READMEs before buying disk.
Sources: Swiftlet README (GitHub) · AirLLM README (GitHub) · vLLM recipes: Qwen3.8-2.4T-A95B · ANEMLL