running_local_models v1.0.0

name: "running_local_models" description: "Run language models on your own machine: size a model to the hardware with a quant and context budget, choose between Ollama, LM Studio, llama.cpp, and vLLM, connect a local model to agent tools through its OpenAI compatible API, and state plainly what still leaves the device. Use when someone asks to run an LLM locally, asks if their GPU or Mac can run a model, wants a private or offline setup, or wants local vs cloud cost math."

Running Local Models

Purpose

Get a language model running on hardware the user already owns, at a size and speed that hold up in real use. Cover sizing, runtime choice, agent wiring, and an honest privacy and cost read. Runtime neutral: the right answer depends on the machine, the workload, and how many callers share the model.

Workflow

1. Inventory the hardware before naming a model. Record system RAM, GPU and VRAM if any, Apple Silicon chip and unified memory if a Mac, free disk, and operating system. On Apple Silicon treat RAM as the model pool, since CPU and GPU share it. Ask for the target work too: chat, coding help, document questions, embeddings, or an agent that needs tool calls.

1. Size the model to the hardware, weights first, context second. Use the sizing table in references/hardware-sizing.md and run bin/size_model.py to do the math.

bash
   python3 bin/size_model.py table
   python3 bin/size_model.py estimate --params 8 --quant Q4_K_M --context 4096 --vram 8
   python3 bin/size_model.py estimate --params 32 --quant Q4_K_M --context 8192 --vram 24 --ram 64
   

Default to Q4_K_M. Its effective size is about 4.89 bits per weight, so an 8B model is a 4.92 GB file and fits an 8 GB card with room for a short context, while the Q8_0 file at 8.54 GB does not. Step up to Q5_K_M or Q6_K when memory allows and the task is code, math, or strict extraction. Go below Q4 only when nothing else fits, and test a hard prompt from the real workload before trusting it. For mixture of experts models, size by total parameters, not active parameters.

1. Budget context as a separate cost. Longer context means a larger KV cache. An 8B model at Q4_K_M that fits at 4K context can split across CPU and GPU at 32K. Parallel requests multiply the cache bill. Pick the smallest context that covers the job, and say what you picked and why.

1. Pick the runtime by caller count and comfort with a terminal. Full setup for each runtime lives in references/runtime-comparison.md.

• Ollama for one person who wants the fastest start and a scriptable server. Native API at http://localhost:11434/api, OpenAI compatible API at http://localhost:11434/v1.
• LM Studio for a visual app with a built in model browser and a local server at http://localhost:1234/v1. Good when the user does not want a terminal.
• llama.cpp when hardware is unusual, CPU only, or the user needs direct control over GPU layers, threads, and context. Its server runs at http://localhost:8080/v1.
• vLLM when many users or agents share GPU servers and queueing is a measured problem. Its server runs at http://localhost:8000/v1 and it earns its setup cost through PagedAttention and continuous batching, not on a single laptop.

1. Install, load one model, and verify placement before adding anything else.

bash
   ollama pull llama3.2:3b
   ollama run llama3.2:3b
   ollama ps
   

In Ollama, ollama ps should read 100% GPU when a GPU fit was the plan. A split reading means the model plus context overshot the card: cut context first, then quant, then model size. In LM Studio check the loaded memory readout. In llama.cpp check the server log for offloaded layers. One verified good model beats a library of untested downloads.

1. Connect agent tools through the OpenAI compatible endpoint, which all four runtimes provide. Probe the machine first, then swap base URL, dummy key, and model name per runtime.

bash
   python3 bin/probe_local.py
   python3 bin/probe_local.py --chat --model llama3.2:3b
   

python
   from openai import OpenAI
   client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama")
   resp = client.chat.completions.create(
       model="llama3.2:3b",
       messages=[{"role": "user", "content": "Reply in one short sentence."}],
   )
   print(resp.choices[0].message.content)
   

Test three things in order: the models endpoint returns the loaded model, a plain chat request returns text, and tool calling plus structured output pass a real test if the agent needs them. Tool support depends on the model and its chat template, so a chat pass does not prove a tool pass. Keep the server bound to localhost unless the user asks for network access, and warn plainly that a local server has no login by default.

1. Audit what still leaves the device, using references/privacy-and-cost.md. Walk the whole chain: model downloads, update checks, any cloud model picked in the same app, web search or other calling out tools, cloud embeddings or OCR in a document pipeline, synced folders holding chat history or indexes, and signing in. Then run the offline test: load the model, disconnect networking, chat and call the API. Report what broke, since what broke was calling out.

1. Run the cost math with the user's own numbers, never a quoted market price.

bash
   python3 bin/size_model.py cost --tokens-million-per-month 200 --cloud-price-per-million 0.60 --hardware-cost 800 --watts 180 --hours-per-day 6 --kwh-price 0.15
   

Use token volume from a real month and a blended price from the user's own bill. State the deciding reason in one sentence: privacy, offline use, cash at volume, latency, or control. If no reason holds, say hosted is the better call.

Output Contract

Deliver a setup the user can run, not a tour of options:

• Hardware read: RAM, VRAM or unified memory, disk, and the largest model class that fits well.
• Chosen model and quant with the weights size, the context budget, and the total planning figure from bin/size_model.py.
• Chosen runtime with the reason tied to caller count and terminal comfort, plus the exact install, pull or download, and run commands.
• Verification evidence: placement check result such as ollama ps output, a models endpoint response, and a chat test result.
• Agent wiring: base URL, model name, and the tool calling test result when tools are in scope.
• Privacy statement that lists what stays local and each part that still calls out, with the offline test result.
• Cost statement from the calculator with the user's inputs named, and the one sentence deciding reason.

Operating Rules

• Verify every size, port, and requirement against the runtime docs or the model file listing before stating it. File sizes in this skill trace to the llama.cpp quantize figures for Llama 3.1 and to vendor docs for ports and system floors. Unverifiable claims get cut or labelled as a rough estimate.
• Never size by download size alone. Weights, KV cache at the chosen context, parallel requests, and system headroom all count, and the total decides the fit.
• Never promise that local means private without auditing tools, embeddings, cloud model entries, and sync. The prompt path is only one part of the chain.
• Never expose a local server to a network as a casual step. Binding beyond localhost, a tunnel, or Serve on Network turns a private tool into a reachable API with no login by default.
• Never present speed figures from another machine as the user's result. Give ranges as expectations, then measure on the user's hardware with ollama ps, API eval fields, llama-bench, or the runtime readout.
• Never use total parameters and active parameters interchangeably for mixture of experts models. Memory follows total, speed follows active.
• Ship one working model first. Add a second model, a document pipeline, or a shared server only after the first setup passes its chat and placement checks.