Running local AI on every size of machine
A $45 board, a phone, a laptop and a workstation can all run AI offline. Which size of model fits yours?
On this page
Somewhere in your home there is probably a chip the size of a fingernail that listens for one word. It has less memory than a single photo takes up, it has never seen the internet, and it runs a small AI model all day. At the other end of the scale, a desktop the size of a shoebox can hold a model with 235 billion parameters. The file for the second model is about seven million times larger than the first.
People ask “can my computer run AI?” as if the answer were yes or no. The answer is almost always yes. The real question is which size of AI, and that depends on one thing more than any other: how much memory the machine has.
This post walks up a ladder of six machines, from a microcontroller to a workstation. Each rung answers the same three questions: what fits, how fast it runs, and what it is good for. The short version: find your machine in the table below, run the model on its row, and expect each step up to buy you a larger model rather than a faster small one.
One rule holds on every machine: the model has to fit in memory
An AI model is a file full of numbers. To run it, the computer loads that file into memory and reads through most of it for every word it writes. If the file is bigger than the memory, the model does not load, or it runs so slowly that nobody would wait for it.
The trick that makes models fit is quantization, which stores each number with fewer bits. Google’s own table for Gemma 4 shows the effect. Its 12B model needs 26.7 GB at full 16-bit precision and 6.7 GB at 4-bit. That is the same model at a quarter of the size, with a small loss in quality. Nearly every model named in this post is a 4-bit version. Our memory-math explainer has the rule of thumb, which is about 0.6 GB per billion parameters.
| Machine | Memory | What fits | Example model | Good for |
|---|---|---|---|---|
| Microcontroller | 256–512 KB | Kilobytes. No language models. | Keyword spotter, under 20 KB | Wake words, gestures, sensor alerts |
| Raspberry Pi 5 | 1–16 GB | 1–2 billion parameters | Gemma 4 E2B, 2.6 GB | Offline voice projects, tinkering |
| Phone | 1–3 GB free for a model | 2–4 billion parameters | Gemma 4 E4B, 3.7 GB | Summaries, rewrites, photo questions |
| Laptop | 16–32 GB | 4–20 billion parameters | Gemma 4 12B, 8 GB | Daily chat, writing, transcription |
| Desktop with a GPU | 16–32 GB on the card | 20–31 billion parameters | Gemma 4 31B, 20 GB | Fast chat, coding help, images |
| Workstation | 96–512 GB | 120 billion and up | gpt-oss-120b, 65 GB | The largest open models |
One note on scope before the climb. CSuite, the app we make, runs on laptops and desktops with macOS, Windows or Linux. The first three rungs use other tools, and we name them as we go.
A microcontroller runs AI, but not the kind you chat with
A microcontroller is a whole computer on one chip, the kind inside a thermostat or a toy. Its memory is counted in kilobytes. The Arduino Nano 33 BLE Sense Rev2, a popular board for learning this, has 256 KB of working memory and 1 MB of storage. Espressif’s ESP32-S3 has 512 KB built in.
What fits: models measured in kilobytes. Google’s LiteRT for Microcontrollers (the current name for TensorFlow Lite Micro) fits its core runtime in 16 KB. Its starter speech example is a model under 20 KB that tells “yes” from “no.”
How fast: fast enough to keep up with a microphone. Espressif’s published benchmark for its WakeNet9 wake-word model on the ESP32-S3 lists 16 KB of internal memory plus 324 KB of add-on memory, and 3 milliseconds of work for each 32-millisecond slice of sound.
Good for: wake words, gesture detection, and sensors that notice when a machine sounds wrong. This field is called TinyML, and Arduino points beginners to Edge Impulse for training models. Not worth it here: any chatbot. The smallest language model in this post is about 10,000 times larger than the Arduino’s memory.
A Raspberry Pi runs a real language model, slowly
A single-board computer is a different animal. The Raspberry Pi 5 has a four-core 2.4 GHz processor, runs Linux, and comes with 1 GB to 16 GB of memory. Memory is also why the price list looks odd in 2026. The 1 GB board is still $45, while the 16 GB board launched at $120 and now lists at $305 after three memory-driven price rises.
What fits: the smallest real language models, at 1 to 2 billion parameters. Gemma 4 E2B is a 2.6 GB file and used about 1.5 GB of memory in Google’s test on a Pi 5.
How fast: 7.6 tokens per second, after a 7.8-second wait for the first word. That is a steady trickle of words, usable for short answers. The next model up, Gemma 4 E4B, drops to 3.2 tokens per second, which tests your patience.
An add-on does not change the size class. The Raspberry Pi AI HAT+ 2 carries its own AI chip and 8 GB of memory. It launched in January at $130 and is listed at $200 today. The models offered for it at launch were all 1 to 1.5 billion parameters. Raspberry Pi is direct about the gap: it says edge models run at 1 to 7 billion parameters while the large cloud models range from 500 billion to 2 trillion.
Good for: an offline voice assistant, a smart-home brain, or learning how this works on a board you can hold in one hand. Speech fits well: whisper.cpp lists its tiny transcription model at about 273 MB of memory. Not worth it here: long documents, or anything above about 4 billion parameters.
Your phone is a faster AI computer than a Raspberry Pi
This is the rung that surprises people. A recent phone has a graphics chip, and that chip runs a small model about seven times faster than a Pi’s processor does.
What fits: models of 2 to 4 billion parameters. Google lists its phone build of Gemma 4 E2B at 1.1 GB of memory and E4B at 2.5 GB. A phone shares its memory with every open app, so that is about the limit.
How fast: 52 tokens per second on a Galaxy S26 Ultra and 56 on an iPhone 17 Pro for E2B, with the first word in 0.3 seconds. The larger E4B runs at 22 to 25, still quick enough for chat.
You may already be using this rung. Apple lists Apple Intelligence on the iPhone 15 Pro and every iPhone 16 or newer. On Android, Gemini Nano runs inside a system service and gives apps on-device summaries, proofreading, rewrites and image descriptions. To pick the model yourself, the free Google AI Edge Gallery app runs Gemma 4 on Android 12 and iOS 17 or later, with no internet needed.
Good for: summarizing a message thread, rewriting a note, asking about a photo, and working on a plane or a train. Not worth it here: hard reasoning. A 2B model is quick and often wrong on multi-step problems. Our piece on why small models matter covers what they do well.
A laptop is where local AI becomes a daily tool
A laptop is the first rung with enough memory for a model you would choose over a cloud chatbot for everyday work. It is also where CSuite starts. Laptops come in three kinds, and they behave differently.
- Integrated graphics. A thin Windows laptop with an Intel Lunar Lake chip ran Gemma 4 E2B at 48 tokens per second and E4B at 25 in Google’s tests. That is phone speed, with more room.
- Apple silicon. A Mac shares one pool of memory between the processor and the graphics chip. The MacBook Air starts at 16 GB and goes to 32 GB. The llama.cpp project’s gpt-oss guide lists a 36 GB M4 Max writing 92 tokens per second with the 20B model.
- A separate graphics card. Here only the card’s own memory counts, and laptop cards carry less of it than desktop ones. Our laptop guide goes through the options.
What fits: on 16 GB, Gemma 4 12B at about 8 GB is the comfortable pick, and it reads images as well as text. On 32 GB, gpt-oss-20b at 14 GB fits with room for a long conversation. Our Gemma 4 guide has the commands.
Good for: chat, drafting, summarizing documents, and transcription. Whisper’s largest model needs about 3.9 GB, so it sits beside a chat model on a 16 GB machine. Not worth it here: the 30B class on a 16 GB laptop. It will not load, and no setting changes that.
On a desktop, the graphics card’s memory is the ceiling
A desktop adds a full-size graphics card, and the card brings its own fast memory. That memory is the number to shop by. Today’s top consumer card, the GeForce RTX 5090, has 32 GB. The RTX 5080 below it has 16 GB.
What fits: on a 16 GB card, gpt-oss-20b, which needs 14.9 GB with a short conversation. On 24 or 32 GB, Gemma 4 31B, which Google lists at 17.5 GB to load.
How fast: very. The same llama.cpp guide lists gpt-oss-20b at 282 tokens per second on an RTX 5090, 205 on a 16 GB RTX 5080, and 162 on a used RTX 3090. Answers arrive faster than you can scroll.
Good for: coding help, long documents, and image generation, which is built for these cards first. Not worth it here: the 120B class. It needs 64 GB, and no consumer card has that. Our Apple silicon vs NVIDIA comparison covers what happens at that wall.
A workstation buys room for the largest open models
The top rung is defined by memory counted in the hundreds of gigabytes. There are two common routes. NVIDIA’s RTX PRO 6000 workstation card has 96 GB. Apple’s Mac Studio with M5 Ultra starts at $5,499 and can be ordered with up to 512 GB.
What fits: gpt-oss-120b at 65 GB, and on the largest Macs, models like Qwen3 235B at 142 GB.
How fast: the llama.cpp guide lists gpt-oss-120b at 196 tokens per second on the RTX PRO 6000 and 80 on a 192 GB M2 Ultra Mac. Both are faster than a small model on a phone.
Good for: the strongest open models, several people sharing one machine, and work that cannot leave the building. Not worth it here: buying one for everyday chat. A 12B model on a laptop already handles that.
A bigger machine buys a bigger model, not a faster small one
Look at the speed chart again. The same small model runs at 56 tokens per second on a phone and 143 on a high-end desktop card. That is under three times the speed from a machine many times the size. Nobody reads at 143 tokens per second, so the extra speed is mostly wasted.
The value of each rung is the model it unlocks. A Pi holds a 2B model. A laptop holds a 12B model that makes fewer mistakes. A desktop card holds a 31B model, and a workstation holds one four to eight times larger again.
So the practical advice is short. Find your machine in the table and run the model on its row tonight. If the answers are good enough, stop there. If they are not, the fix is more memory, and our hardware calculator shows how much the next step needs.
