---
title: "Model Backends"
description: "Configure inference backends for Caro"
url: "https://caro.sh/docs/backends"
markdown_url: "https://caro.sh/docs/backends.md"
lang: "en"
type: "website"
image: "https://caro.sh/og/docs/backends.png"
---

# Model Backends

Caro supports multiple inference backends for running language models. Choose the best option for your hardware and use case.

## Available Backends

### MLX (Apple Silicon)

Optimized for Apple M1/M2/M3 chips using Metal Performance Shaders. This is the fastest option for Mac users with Apple Silicon.

- **Platform**: macOS (Apple Silicon only)
- **Performance**: Fastest on supported hardware
- **Memory**: Uses unified memory efficiently

### Ollama

Connect to a local Ollama instance for inference. Great for users who already have Ollama set up.

- **Platform**: Cross-platform
- **Requires**: Ollama running locally
- **Models**: Any Ollama-compatible model

### vLLM

Connect to a vLLM server for high-throughput inference. Best for users with powerful GPU servers.

- **Platform**: Cross-platform (client)
- **Requires**: vLLM server running
- **Performance**: Excellent for batch processing

### CPU Backend (Candle)

Pure CPU inference using the Candle library. Works everywhere but slower than GPU-accelerated options.

- **Platform**: Cross-platform
- **Performance**: Slower but universal
- **No GPU required**

## Recommended Models

Caro works best with coding-focused models:

### Qwen 2.5 Coder (Recommended)

- `qwen2.5-coder:3b` - Fast, good for simple commands
- `qwen2.5-coder:7b` - Balanced performance and quality
- `qwen2.5-coder:14b` - Best quality, needs more RAM

### Other Compatible Models

- `codellama:7b` - Meta's coding model
- `deepseek-coder:6.7b` - DeepSeek's coding model
- `starcoder2:7b` - BigCode's model

## Setting Up Ollama

1. **Install Ollama**:
   ```
   curl -fsSL https://ollama.com/install.sh | sh
   ```
2. **Pull a model**:
   ```
   ollama pull qwen2.5-coder:7b
   ```
3. **Configure Caro**:
   ```
   # In ~/.config/cmdai/config.toml
   default_model = "qwen2.5-coder:7b"
   ```

## Model Caching

Caro caches downloaded models to speed up subsequent runs. The cache is stored in:

- **macOS/Linux**: `~/.cache/cmdai/models/`
- **Windows**: `%LOCALAPPDATA%\cmdai\models\`

### Cache Management

```
# View cache size
du -sh ~/.cache/cmdai/

# Configure max cache size (in config.toml)
cache_max_size_gb = 20
```

When the cache exceeds the limit, Caro automatically removes least-recently-used models.

## Performance Tips

- **Apple Silicon**: Use MLX backend for best performance
- **NVIDIA GPU**: Use Ollama with GPU acceleration
- **Limited RAM**: Use smaller models (3B parameters)
- **Slow internet**: Pre-download models with Ollama