National Cyber Warfare Foundation (NCWF)login

Automating authorized penetration tests with AutoPentester, an LLM agent framework


0 user ratings
2026-11-10 13:27:58
milo
Red Team (CNA)
"Automating

AutoPentester is an academic LLM-agent framework that automates penetration testing tasks by orchestrating multiple model-backed agents, intended for authorized assessments and security research.








ToolYasodGinige/AutoPentester — an LLM agent-based framework for automated penetration testing tasks
CategoryLLM-driven security automation / automated pentesting framework
Primary UseRunning agent-driven, conversational penetration tests against lab machines and authorized targets, initiated with prompts like testing a machine by IP address
Safe UseFor authorized penetration tests, CTF/lab environments, and academic security research only; the tool targets a user-supplied IP that the operator must be explicitly authorized to test
Telemetry NoteThe framework writes session and processed logs to the processed_log_files directory and depends on external LLM provider APIs (OPENAI_API_KEY, ANTHROPIC_API_KEY), so target prompts and results may transit third-party clouds — a signal defenders and auditors should note

AutoPentester is a research-driven framework out of YasodGinige/AutoPentester that applies large language model agents to penetration testing automation. The project is not a loose script collection but the reference implementation of a peer-reviewed paper accepted at IEEE TrustCom 2025, with an arXiv preprint (2510.05605) backing it. With 96 stars, a Python codebase, and an MIT license, it sits in the growing category of academic tooling that tries to turn an LLM into an operator that plans, executes, and documents a test rather than merely answering questions about one.


The architecture, as the README's framework diagram suggests, is multi-agent. Configuration lives in pentestgpt/config/agent_config.py, where each named agent is mapped to a provider and model. That mapping is per-agent, which is the interesting design decision: a practitioner can mix, say, an Anthropic model for one reasoning role and an OpenAI model for another, tuning cost and capability per stage of the test. The naming of the pentestgpt package signals lineage with prior conversational pentest assistants, but AutoPentester extends that idea toward an orchestrated, multi-provider agent pipeline rather than a single chat client.


Provider support is unusually broad. The README lists openai, anthropic, gemini, azure, gpt4all, titan, and rag as supported providers. The inclusion of gpt4all matters operationally: it implies locally hosted models are a first-class option, which lets an authorized assessor keep target data off third-party clouds — a genuine compliance consideration when testing environments with data-handling restrictions. Google's Gemini is selected per agent through the same pentestgpt/config/agent_config.py file rather than a command-line flag, so configuration is file-centric rather than parameter-centric.


Setup follows conventional Python practice. The README instructs users to create a virtual environment with python3 -m venv myenv, activate it, then pip3 install -r requirements.txt and pip3 install -e .. The project pins Python 3.12.3 as the reference interpreter, which is a useful hint for dependency compatibility. After installation, invocation is a single command: autopentester --loggin — note the doubled g in the flag spelling, which appears to be intentional in the source rather than a typo in the README. The tool then interactively prompts for the provider API key and the target IP address, and asks whether to resume a previous session.


The retrieval-augmented generation component deserves attention. On first run, the framework takes roughly ten minutes to build a vector database for its rag module before testing begins. That warm-up cost is a one-time investment per environment, and it tells us the agents ground their reasoning in an embedded knowledge base of pentesting technique rather than relying purely on model priors. For a professional evaluating the tool, this is the difference between an LLM hallucinating exploit paths and one retrieving documented methodology — an important distinction in a domain where wrong steps waste engagement time at best.


Credential handling is done through environment variables: export OPENAI_API_KEY="", export ANTHROPIC_API_KEY=..., or export GEMINI_API_KEY=... / export GOOGLE_API_KEY=.... Notably, the OpenAI path also supports endpoint overriding via export OPENAI_BASE_URL="https://your-host/v1", which routes traffic through a self-hosted or proxied gateway — another privacy-preserving lever. The README candidly warns that an OpenAI payment method must be linked, an honest acknowledgment that agent loops consume tokens at a rate that matters to anyone running repeated engagements.


The interaction model is prompt-driven. The README's example task reads like natural language: the user states they want to test a machine at a given target IP, and the agents take it from there. The framework retains session state and offers continuation from previous sessions on startup, which suggests iterative testing where an operator pauses and resumes a multi-step workflow. For an authorized engagement against a lab box or a scoped assessment target, this turns the assessor's role from executing individual commands toward supervising and validating agent-proposed activity.


Transparency is a stated priority. Processed logs land in the processed_log_files directory, and the authors state that the quantitative results in their paper were computed from exactly these log files — meaning the logging format is designed to be analyzable, not just forensic baggage. A Survey_analysis directory ships with the repo, and running analysis.py regenerates the graphs from the paper's user study. This is a rarity among security tooling: the evaluation artifacts are published alongside the code, so a reviewer can audit how the framework was measured rather than taking marketing claims on faith.


From a defensive perspective, the telemetry footprint is worth understanding even if you never run the tool. Every session depends on outbound calls to LLM provider APIs, meaning prompts describing target infrastructure — IP addresses, observed services, agent reasoning — leave the operator's machine toward whichever provider is configured, unless gpt4all or a custom OPENAI_BASE_URL gateway is used. Blue teams assessing the risk of this class of tooling in their environment should treat API egress patterns and the local processed_log_files directory as the observable artifacts of use. The authors also invite issue reports for incorrect, inappropriate, or outdated content, a governance signal that the project maintains is maintained with academic accountability in mind.


Where AutoPentester fits in a professional workflow is the supervised-assistant slot. It is best understood as a research framework for studying how agentic LLMs perform end-to-end testing, and as a productivity aid for authorized assessors who want a persistent, session-aware reasoning partner driving standard tooling. It is not, and should not be treated as, a substitute for scoping, rules-of-engagement documents, or human validation of findings — the same caveat that applies to every agentic security tool currently shipping. The IEEE acceptance and published evaluation make it one of the more citable entries in this space, and the MIT license permits both study and extension.


Practitioners who want to see the framework in motion before committing a target can consult the demonstration video linked from the README, and the paper itself is the authoritative description of the agent design. For teams building their own automation, the cleanest lessons here are the per-agent provider mapping in pentestgpt/config/agent_config.py, the RAG vector store for grounding, and the publishable log pipeline — patterns worth borrowing whether or not you adopt the framework wholesale. As with any tool that accepts a target IP and acts on it, deployment belongs squarely inside authorized lab and engagement boundaries, with the operator accountable for every action the agents take.



Official project repository for YasodGinige/AutoPentester.

Download Tool

Educational analysis for authorized security professionals. Use only in controlled, authorized environments.






Source: OffensiveSec
Source Link: https://www.offsecblog.com/2026/11/automating-authorized-penetration-tests.html


Comments
new comment
Nobody has commented yet. Will you be the first?
 
Forum
Red Team (CNA)



Copyright 2012 through 2026 - National Cyber Warfare Foundation - All rights reserved worldwide.