AI • Local • Privacy • Offline

Local AI Reference Guide

System: Debian Trixie (Testing)  |  GPU: RTX 2080 (8GB)  |  RAM: 128GB DDR4

Guide to running AI models locally for privacy, offline use, and experimentation using Ollama and Qwen.

Quick Start: If everything is already installed, just run ollama run qwen2.5:7b for speed or ollama run qwen2.5:32b for coding intelligence.

Step 1: System Prep (Debian Trixie)

Ensure your system allows non-free firmware for NVIDIA drivers.

Fix Sources List
sudo sed -i 's/main$/main contrib non-free non-free-firmware/g' /etc/apt/sources.list
Install NVIDIA Drivers
sudo apt update
sudo apt install -y nvidia-driver firmware-linux-nonfree
sudo reboot
Verify: After reboot, run nvidia-smi. You should see your RTX 2080 listed.

Step 2: Install Ollama

Ollama is the engine that runs the AI models.

Install Script
curl -fsSL https://ollama.com/install.sh | sh

Step 3: Network Configuration

Allow other apps (like your web browser) to talk to Ollama.

Edit Service Config
sudo systemctl edit ollama.service

Paste this into the editor, then save (Ctrl+O, Enter) and exit (Ctrl+X):

Service Override Content
[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"
Environment="OLLAMA_ORIGINS=*"
Restart Ollama
sudo systemctl daemon-reload
sudo systemctl restart ollama

Step 4: Model Selection

Hardware Note:
  • RTX 2080 (8GB): Best for 7B models. Extremely fast response.
  • 128GB RAM: Can run 32B or 72B models on CPU. Slower, but much smarter for coding.
Option A: Fast Chat (GPU Only)
ollama pull qwen2.5:7b
ollama run qwen2.5:7b
Option B: Smart Coding (Hybrid GPU/CPU)
# Downloads the 32B model (uses ~20GB RAM)
ollama pull qwen2.5:32b

# Runs with 20 layers on GPU for speed, rest on CPU
ollama run qwen2.5:32b --num-gpu-layers 20

Step 5: Simple Web Interface

Save this as index.html in your web server folder (e.g., /var/www/html/).

Chat UI Code
<!DOCTYPE html>
<html>
<head>
    <title>Local Qwen Chat</title>
    <style>
        body { font-family: sans-serif; max-width: 800px; margin: 20px auto; padding: 20px; }
        #chat-box { border: 1px solid #ccc; height: 400px; overflow-y: scroll; padding: 10px; margin-bottom: 10px; background: #f9f9f9; }
        .msg { margin: 5px 0; padding: 8px; border-radius: 5px; }
        .user { background: #d1e7dd; text-align: right; }
        .ai { background: #e2e3e5; text-align: left; }
        input { width: 70%; padding: 10px; }
        button { padding: 10px 20px; cursor: pointer; }
    </style>
</head>
<body>
    <h1>Chat with Qwen (Local)</h1>
    <div id="chat-box"></div>
    <input type="text" id="userInput" placeholder="Type here..." onkeypress="handleEnter(event)">
    <button onclick="sendMessage()">Send</button>

    <script>
        async function sendMessage() {
            const input = document.getElementById('userInput');
            const chatBox = document.getElementById('chat-box');
            const text = input.value;
            if (!text) return;

            chatBox.innerHTML += `<div class="msg user">${text}</div>`;
            input.value = '';

            try {
                const modelName = "qwen2.5:7b";
                const response = await fetch('http://localhost:11434/api/generate', {
                    method: 'POST',
                    headers: { 'Content-Type': 'application/json' },
                    body: JSON.stringify({
                        model: modelName,
                        prompt: text,
                        stream: false
                    })
                });
                const data = await response.json();
                chatBox.innerHTML += `<div class="msg ai">${data.response}</div>`;
                chatBox.scrollTop = chatBox.scrollHeight;
            } catch (err) {
                chatBox.innerHTML += `<div class="msg ai" style="color:red">Error: ${err}</div>`;
            }
        }

        function handleEnter(e) {
            if (e.key === 'Enter') sendMessage();
        }
    </script>
</body>
</html>

Summary Checklist