Agentic Security Assessment

What it does

You give it a website address and tick a box confirming you are allowed to test it. A Planner agent reads your notes ("web checks only, no port scan") and picks from four open-source scanners: nmap, OWASP ZAP, Nuclei and Nikto. You review the plan and approve it, change it or cancel. An Executor agent runs the approved tools one at a time, and you can stop it after any of them. A Reporter agent rates what the tools found and writes the report in plain language: severity, the evidence the scanner saw, and how to fix it.

The report comes as a web page, PDF, Markdown, JSON and SARIF, the format GitHub code scanning and security dashboards import. Run the same check again later and it shows what changed since last time. Findings carry their OWASP Top 10 category, taken from ZAP's own tags rather than guessed.

It covers the finding and reporting stages of a pen test: reconnaissance, scanning, vulnerability analysis and reporting. It does not exploit anything. That stage can damage a system or expose real data, so it belongs with a person holding a signed agreement.

The AI may choose; code decides what is allowed

Scanners are powerful and models make mistakes, so nothing here depends on the model behaving well.

• The target cannot be changed by the AI. It is fixed when you tick the box, and the tools take no address at all: the scanner service reads it from the saved check.
• Permission is checked three times. The page, the API and the scanner service, before every tool. An unapproved plan is "not authorized" as far as the scanners are concerned.
• The Executor can only call planned tools, once each, within a fixed number of turns. It once stopped after three of four tools and claimed all four had run, so its closing words are no longer trusted: code compares the plan with what ran.
• The Reporter cannot invent a finding. Code copies findings from the scanner output; the model only writes the wording, attached by finding ID. A CVE number no scanner reported throws the whole summary away. Severity comes from the scanner's own rating, never the model.
• Looking, not attacking. ZAP runs passive rules and never submits a form. Nuclei and Nikto run without their intrusive, brute-force, fuzzing and denial-of-service checks, throttled, with a pause and a health check between tools.
• Text from the scanned site is data, not instructions. Page titles and server banners could carry prompt injection, so the agents see short excerpts and have no lever to act on them.

Measured, not assumed

• A test set for the Planner's decisions: 12 situations with a known right answer, including notes in mixed Hindi and English, a request for tools that don't exist, and a prompt-injection attempt. 12 of 12 on each of three models on Groq's free tier.
• Seven quality checks by code on every run, scored 0 to 1 and attached to the Langfuse trace: plan followed, stayed in scope, tools completed, target stayed up, every finding found again in the raw scanner file, AI text grounded in reported IDs, fix advice for every Medium-or-higher finding.
• A full four-tool check of the practice site with a 20B model planning and running and a 120B model reporting: 32 findings in 7 minutes 19 seconds, all seven checks at 1.0, about three quarters of the tokens on the small model. The small model skipped the last tool once; the guard rail reminded it and it ran.
• Automated tests cover the permission gate, address checks, plan review, stop, the retest comparison, SARIF, the scanner parsers and the Executor's guard rails, the last ones with a scripted stand-in for the model.

The evaluation earned its place while building. The Planner test set caught two models adding a noisy scanner when Hinglish notes asked for "web checks only", and one model answering a prompt injection with a flat refusal that would have stopped the run. A full run caught the practice site running out of memory, followed by a scanner reporting "0 findings" on a dead site. Each one is now a rule, a check or a test.

How it is built

Four containers behind one web page: a FastAPI backend that owns permission and storage, Langflow running the three agents, a scanner service wrapping the tools, and ZAP. Everything binds to 127.0.0.1. Each agent can run on its own model, set by one line in the config. The practice target is OWASP Juice Shop, a deliberately vulnerable shop, kept in its own folder so adding another target needs no code change.

Next

A detection score against Juice Shop's own list of known problems, an evaluation set for the Reporter's fix advice, and exploitation of findings in a sandbox behind a second explicit approval, so a report can separate confirmed issues from leads.