MMohan Vishe
All projects
Open sourceAI securityLLM evaluationLocal AI

Local AI Safety Audit

A free, read-only security check-up for your own local AI setup, an auditor for the text a local model actually reads, and a measured answer to whether the difference matters.

View code on GitHubPythonMIT

Start here

Six weak spots at a glance 1. The app answers anyone who can reach it: Anyone on your Wi-Fi, or online, can use, delete or replace your models and use your ollama.com account. Fix: keep Ollama on 127.0.0.1 (the default).; 2. Older versions have known holes: Published bugs let others crash it, read its memory or, in old versions, take over the computer. Fix: update Ollama to 0.40.8 or later.; 3. A model file carries more than the model: Its template and settings can be outdated, rewritten by an uploader, or tampered with. Fix: get models from the maker or Ollama's library.; 4. The layout can change what the model reads: 32 of 43 library models read tool chats differently; qwen2.5 got 45 fewer of 1,240 tool questions right. Fix: test tool use on the exact model you ship.; 5. Helpers turn the model's words into actions: Text hidden in a page or file can steer the helper into reading or sending your files. Fix: one folder, no secrets in it, approve risky steps.; 6. Your chats are kept as ordinary files: Backups, sync tools and anyone using your account can read your saved conversations. Fix: delete old chats; set OLLAMA_NOHISTORY=1. Six weak spots at a glance What can go wrong in a local AI setup, and the fix that matters most for each. 1 The app answers anyone who can reach it Anyone on your Wi-Fi, or online, can use, delete or replace your models and use your ollama.com account. Fix: keep Ollama on 127.0.0.1 (the default) 2 Older versions have known holes Published bugs let others crash it, read its memory or, in old versions, take over the computer. Fix: update Ollama to 0.40.8 or later 3 A model file carries more than the model Its template and settings can be outdated, rewritten by an uploader, or tampered with. Fix: get models from the maker or Ollama's library 4 The layout can change what the model reads 32 of 43 library models read tool chats differently; qwen2.5 got 45 fewer of 1,240 tool questions right. Fix: test tool use on the exact model you ship 5 Helpers turn the model's words into actions Text hidden in a page or file can steer the helper into reading or sending your files. Fix: one folder, no secrets in it, approve risky steps 6 Your chats are kept as ordinary files Backups, sync tools and anyone using your account can read your saved conversations. Fix: delete old chats; set OLLAMA_NOHISTORY=1 Each one is explained below, with how to check it on your own computer.
The six weak spots of a local AI setup, each with the fix that matters most.

In plain words

A chat model learns to read conversations laid out in one exact way, and its maker publishes the chat template that produces that layout. When you run the model with Ollama, the template has been copied into the model file and often rewritten by Ollama. This project checks whether the text the model finally reads is still the text its maker designed, and whether a difference changes what the model does.

How a chat template travels from its maker to the model The maker's template on Hugging Face is copied into the model file (GGUF), and Ollama then runs either that copy or its own rewritten version; the model reads only the result. Of the 32 Ollama library models whose tool conversations differ from the maker's, 24 differ only because of Ollama's own version, 6 start with an older copy in the model file, and 2 have no template in the file to compare. How a chat template travels from its maker to the model Where the difference first appears, for the 32 of 43 Ollama library models whose tool conversations differ. The maker's template on Hugging Face, written by the maker L0 · Jinja The copy in the file made when the model is packaged as GGUF L1 · GGUF header What Ollama runs often Ollama's own rewritten version L3 · Go template The model reads only this one block of text The reference: the text the model was trained to read. 6 of 32 an older copy of the maker's, then Ollama's 2 of 32 the file has no template to compare 24 of 32 Ollama's own version changes the layout Source: runs/p2-headline.json in the project repository.
The chat template's three copies on its way to the model, and where the differences start for the 32 library models that differ.

What it found

  • In Ollama's library, 32 of 43 popular models read their tool conversations differently from what their maker designed (snapshot of 22 October 2026, Ollama 0.40.7). In 24 of the 32, the difference comes from Ollama's own version of the template. On Hugging Face, 5 of 22 differ.
  • It changes behaviour. On 1,240 tool-calling questions from the Berkeley Function Calling Leaderboard, qwen2.5 answered 83.4% correctly with Ollama's layout and 87.0% with its maker's (exact McNemar p ≈ 4×10⁻⁶). Read by Ollama's own parser on both sides, it was 84.9% against 88.1%.
  • A benchmarking pitfall. Ollama's engine keeps recent prompts in memory, and reusing one changed answers to identical prompts at temperature 0. The test ran with that cache off and checks every reply for reused tokens.
How many popular models read their tool conversations differently Ollama library: 32 of 43 differ (28 setting aside order-only changes); built for tools: 15 of 19; Hugging Face: 5 of 22. Of the library differences, 24 start in Ollama's own template, 6 in an old copy in the model file, 2 where the file has no template. How many popular models read their tool conversations differently Compared with the maker's own template. Snapshot of 22 October 2026, Ollama v0.40.7. Ollama library, all compared models 32 of 43 …setting aside field-order-only changes 28 of 43 Ollama library, models built for tools 15 of 19 Hugging Face downloads 5 of 22 Where the 32 library differences start Ollama's own version of the template 24 of 32 An old copy in the model file, then Ollama's 6 of 32 The model file has no template 2 of 32 Source: runs/p2-headline.json in the project repository.
How many popular models read their tool conversations differently from their maker's template.
The maker's layout wins whichever parser reads the reply Read by the benchmark's parser (BFCL): Ollama's layout 1034 of 1240, maker's layout 1079 of 1240; Read by Ollama's own parser, as in everyday use: Ollama's layout 1053 of 1240, maker's layout 1093 of 1240 The maker's layout wins whichever parser reads the reply Share of 1,240 tool questions qwen2.5 answered correctly. Read by the benchmark's parser (BFCL) +45, clear Ollama's layout 83.4% (1,034 of 1,240) The maker's layout 87.0% (1,079 of 1,240) Read by Ollama's own parser, as in everyday use +40, clear Ollama's layout 84.9% (1,053 of 1,240) The maker's layout 88.1% (1,093 of 1,240) Right: extra correct answers with the maker's layout. "Clear": exact paired test, p below 0.001. Ollama's parser, maker's layout: its saved replies read offline by Ollama's own parser code, which reproduced the live everyday run on all 1,240 questions. Source: runs/h3-mini/qwen2.5/report.json, report-A.json, report-ollama-parser.json.
qwen2.5 on 1,240 tool questions: the maker's layout wins whichever parser reads the replies.

How it works

  • It never runs or downloads a model for the audit. It reads only the header of each model file over HTTP range requests (7.3 MB of a multi-gigabyte file for qwen2.5) and renders twelve harmless test conversations through each copy of the template.
  • Ollama's side is Ollama's own code for 41 of the 45 library files: a small Go helper imports Ollama at the pinned version and runs its template code. The Hugging Face models and three library files use a close Python copy of Ollama's other engine.
  • The difference classifier was checked against hand labels written before it existed: tuned on 24, frozen, then 28 of 28 on a held-out set.
  • The tool test is paired and controlled: same model file, same isolated server, temperature 0, the same parser for both layouts in each comparison, and 60 repeated answers identical.

Code and data on GitHub