AI use cases Local AI Run AI models locally
Run AI models locally
Run open-weight language models on your own computer or server for privacy, offline use and predictable cost. Workflow and ranked local AI tools.
- Tools compared
- 3
- Open source
- 3of 3
- Reviewed
- 16 Oct
For: Developers · Privacy-focused teams
What the score and shapes mean
Score = public maintenance, adoption and docs signals, as a percentile of 52 tools. Nobody here tested these hands-on. Shapes are the separate safety label: No issues found, No dependencies scanned, Caution.
What it does for you
Developers, privacy-minded users and teams with their own hardware.
- Chat with an open model without sending data to a provider
- Run models offline
- Expose a local, OpenAI-style API to your apps
- Serve a model to a team on your own hardware
- Try many open models quickly
More about this job
Open-weight models can run on a laptop, a workstation or a server. Local runners make download-and-run simple, while inference engines target speed and throughput when you serve many users.
Workflow in 5 steps
- Check your RAM or GPU memory.
- Install a local runner and pull a small model first.
- Test on your own tasks, not just generic prompts.
- Move to a faster inference engine if you need to serve several users.
- Keep models and runners updated and check each model's licence.
Full steps
- Check your RAM or GPU memory; it decides which model sizes are realistic.
- Install a local runner and pull a small model first.
- Test on your own tasks, not just generic prompts.
- Move to a faster inference engine if you need to serve several users.
- Keep models and runners updated and check each model's licence.
Prompts to copy
Starting points we wrote, not tested results.
Explain what quantisation does to a model's size, speed and quality, and give a rule of thumb for choosing a level for 16 GB of memory.
2 more prompts
Write a script that sends this prompt to my local model server and prints the reply. Use the HTTP API, not a library.
Compare the two local replies below for the same prompt and tell me which is more accurate and why.
Watch out for
- Local models are usually smaller than hosted frontier models.
- Check each tool's licence before commercial use, and check the licence of any code or text the AI reproduces.
- Hardware cost and your time are real costs.
- CautionOllama2 critical-severity known vulnerabilities in runtime dependencies (OSV)
- Cautionllama.cpp12 critical-severity known vulnerabilities in runtime dependencies (OSV)
- CautionvLLMSkill/agent file instructs piping a remote script into a shell: AGENTS.md
Why these warnings?
- Local models are usually smaller than hosted frontier models. Expect weaker results on hard reasoning and long contexts, and test on your own tasks.
- Check each tool's licence before commercial use, and check the licence of any code or text the AI reproduces.
- Hardware cost and your time are real costs. Local is not always cheaper than a hosted API at low volume.
- Ollama: 2 critical-severity known vulnerabilities in runtime dependencies (OSV)
- llama.cpp: 12 critical-severity known vulnerabilities in runtime dependencies (OSV)
- vLLM: Skill/agent file instructs piping a remote script into a shell: AGENTS.md — curl -LsSf https://astral.sh/uv/install.sh | sh
Related jobs
Sources for tool facts
ollama/ollama · ggml-org/llama.cpp · vllm-project/vllm · Ollama on GitHub. Data fetched 2 Nov 2026. Scores and labels are automated, relative and not a hands-on test.
Get the board every Monday
One email a week: the new top 10, the biggest score changes and every safety label that changed.
Free. Double opt-in: you confirm by email first, unsubscribe in one click.