How on-device AI works—without the fog.
On-device AI means the model does its work on the phone, laptop, or browser in front of you instead of sending every request to a remote model server. That simple architectural choice changes privacy, speed, offline access, cost, and what the model can realistically do.
Try it with Cappy →
The short version
A language model is a large collection of learned numerical patterns. In a cloud product, your prompt travels to a server that runs those calculations. In an on-device product, the model files are downloaded and the calculations happen locally. A browser can use WebGPU to ask the graphics processor to do this efficiently, with WebAssembly as a compatibility fallback. The first model download can be large; later sessions can reuse cached files.
What can stay on your device
Chat text, saved memories, task lists, and voice transcripts can all remain in browser storage when the product is designed for it. Local does not automatically mean private: analytics scripts, cloud sync, remote fonts, crash reporting, or web search can still create network requests. The useful question is not only ‘Does it use local AI?’ but ‘Which feature sends which data, and when?’
Where local models trade power for control
Smaller models fit consumer hardware and can respond without per-message API fees, but they generally know less and reason less reliably than the largest hosted models. They may also be slower on older devices. A good local-first product is honest about those boundaries and lets people opt into a cloud model only when a task genuinely needs it.
How cappy.pet approaches it
Cappy starts in on-device mode. Its chat history, tiny tasks, and user-controlled memories are stored locally. Optional web search and external model providers are separate choices. That makes the boundary visible: ordinary private reflection can stay local, while fresh web facts or a stronger hosted model can be enabled deliberately.
Quick answers
Does on-device AI work without internet?
Yes, after the model and application files have been downloaded and cached. Features that need fresh information, such as web search or live weather, still require a connection.
Is every on-device AI app automatically private?
No. Local inference keeps the model calculation on the device, but analytics, sync, remote search, fonts, or cloud memory features can still send data. Check each feature's data path.
Related guides
Local AI vs Cloud AI: Privacy and QualityCompare local and cloud AI across privacy, capability, speed, offline access, cost, and the situations where each is a better fit.AI Chat Privacy Checklist: 12 QuestionsUse this practical checklist to evaluate an AI chat app's storage, model providers, memory, deletion, exports, search, and voice handling.User-Controlled AI Memory ControlsUnderstand AI memory, the difference between chat history and saved memory, and the controls a trustworthy assistant should provide.