What
Run open-source LLMs on your computer.
Install
curl -fsSL https://ollama.com/install.sh | sh
Pull a Model
ollama pull llama3.1
ollama run llama3.1
Models
- llama3.1 (8B, 70B, 405B)
- mistral
- qwen2.5
- gemma2
- deepseek-r1
API
import requests
r = requests.post("http://localhost:11434/api/generate",
json={"model": "llama3.1", "prompt": "Hello"})
print(r.json()["response"])
Hardware
- M2 Max: up to 70B
- RTX 4090: up to 70B
- 8GB GPU: 7B max
Use Cases
- Privacy
- Offline
- Free at scale
- Customization