DeepRed: Partial-Credit Evaluation of LLM Agents in CTFs

This repository is an automated benchmark for evaluating Large Language Models in penetration testing scenarios. It provides a framework for testing the security capabilities of language models by challenging them to solve Capture The Flag (CTF) challenges in realistic environments.

Prerequisites

• Python 3.10+
• Linux or Windows operating system (x86_64 architecture)
• VirtualBox (6.0 or later)
• OpenAI model API compatible API key

Note: MacOS is not currently supported due to VirtualBox compatibility issues. VirtualBox 7.0+ has some support for x86_64 on Apple Silicon, but may not work reliably for all challenges in this benchmark.

Installation

1. Clone the repository:

git clone https://github.com/AISE-TUDelft/DeepRed-LLMAgent.git
cd DeepRed-LLMAgent

2. Install Python dependencies:

pip install -r requirements.txt

3. Set up the challenge OVAs:

Follow the instructions in resources/challenges/README.md to download the challenge and Kali VMs.

Note: The Kali OVA is pre-configured for agent communication. The OVA also has all the required tools pre-installed.

The system will automatically:

• Import a fresh Kali VM for each benchmark run
• Configure it with host-only networking
• Remove it after the benchmark completes

4. Verify VirtualBox is working: On Linux:

vboxmanage --version

On Windows:

VBoxManage.exe --version

Networking

DeepRed uses VirtualBox's built-in host-only networking to connect the attacker (Kali) and defender VMs. This provides:

• Isolated network environment for security testing
• Automatic network configuration
• No need for external tools like dnsmasq or bridge-utils

The system automatically creates a vboxnet0 host-only adapter if it doesn't exist, with the network 192.168.56.0/24.

Configuration

DeepRed uses a YAML configuration file to define target VMs. The default configuration file is core_config.yaml in the project root:

vms:
  - name: vm_name
    ova_path: path/to/vm.ova
    flag: flag{expected_flag}
    difficulty: easy|medium|hard

Usage

Running the Benchmark

To run the benchmark against all configured VMs from the core_config.yaml file:

python3 src/deepred/core/run_benchmark.py --model-id openai/gpt-5.1-codex-max

To run against a specific VM:

python3 src/deepred/core/run_benchmark.py --model-id openai/gpt-5.1-codex-max --vm-path resources/challenges/your_vm.ova --flag "expected_flag"

Run with --help to see all available options:

python3 src/deepred/core/run_benchmark.py --help

Data Labeling & Analysis

The labeling/ directory contains utilities for:

• Automatic labeling: Use LLMs to label benchmark outputs
• Human alignment: Calculate inter-rater agreement (Cohen's Kappa, Krippendorff's Alpha)
• Performance scoring: Analyze and visualize model performance

Run each script with --help to see all available options:

python3 labeling/autolabel.py --help

Development & Testing

Experimenting with more models

To experiment with different LLMs, simply specify the model ID when running the benchmark. The system is designed to be compatible with any OpenAI API compatible model (default is OpenRouter). For example:

python3 src/deepred/core/run_benchmark.py --api-base https://api.openai.com/v1 --model-id openai/gpt-5.1-codex-max

Adding New Challenges

To add new challenges, follow these steps:

1. Obtain the OVA file for the new challenge and place it in the resources/challenges/ directory.
2. Update the vm_config.yaml file to include the new VM configuration, specifying the OVA path, expected flag, and difficulty level.
3. Run the benchmark to evaluate the new challenge. The system will automatically import the new VM and include it in the benchmark runs. The results will be logged in the logs/ directory for analysis.
4. Edit the labeling/labeling_format.csv file to add the new challenge with its corresponding checkpoints for labeling and analysis.
5. Run the labeling and analysis scripts to evaluate the new challenge's results.

Human Alignment

To evaluate human alignment, you can use the human_alignment.py script to calculate inter-rater agreement metrics such as Cohen's Kappa and Krippendorff's Alpha. The logs from the runs used for human alignment are in the logs/ directory. The runs use codenames for the models. The human results should be in the labeling/manual_res.csv file. The script will read the logs and human labels, calculate the agreement metrics, and output the results.

Code Style

This project uses Ruff and Black for linting and code formatting. Configuration is in pyproject.toml.

To check your code:

python3 -m ruff check .

To format your code:

python3 -m black .

Project Structure

DeepRed-LLMAgent/
├── src/deepred/                   # Main package source code
│   ├── core/                      # Core benchmarking logic
│   │   └── run_benchmark.py       # Main benchmark runner script
│   ├── infrastructure/            # VM and network infrastructure
│   │   ├── vbox/                  # VirtualBox session management
│   │   └── networking/            # Network configuration
│   └── metrics/                   # Evaluation metrics
├── labeling/                      # Data labeling and analysis scripts
│   ├── autolabel.py               # Automatic labeling using LLMs
│   ├── human_alignment.py         # Inter-rater agreement analysis
│   ├── score.py                   # Scoring and visualization
│   ├── labeling_format.csv        # Labeling schema and checkpoint definitions
│   ├── manual_res.csv             # Human-labeled results for alignment
│   ├── llm_labels/                # LLM-generated labels organized by summary model / judge model
│   └── summaries/                 # LLM-generated summaries organized by model
├── resources/                     # VM images and prompts
│   ├── challenges/                # Challenge VM OVA files
│   └── prompts/                   # System prompts for LLM agents
├── logs/                          # Benchmark execution logs
├── .github/workflows/             # CI/CD configuration
├── pyproject.toml                 # Project metadata and tool configuration
├── vm_config.yaml                 # VM configuration for benchmarks
└── requirements.txt               # Python package dependencies

Challenge Attribution

All of these challenges are sourced from HackMyVM, a platform providing free CTF challenges for cybersecurity learning and practice. A list of the challenges and their contributors:

Filename	Challenge	Author
sysadmin.ova	Sysadmin	Sublarge
aria.ova	Aria	Sublarge
helpdesk.ova	Helpdesk	MrMidnight
Thirteen.ova	Thirteen	Sublarge
Fuzzz.ova	Fuzzz	20206675
nexus.ova	Nexus	ShellDredd
todd.ova	Todd	ll104567
jan.ova	Jan	sml
whitedoor.ova	Whitedoor	Pylon
quick.ova	Quick	eMVee

Acknowledgements

We would like to acknowledge the creators of the challenges for their contributions to the cybersecurity community. We also thank Artur Iurasov and Ivo Harsani for their work on the initial version of the benchmark framework as part of the CSE2000 Software Project course at TU Delft.

License

This project is licensed under the MIT License - see the LICENSE file for details.

We encourage the use of this benchmark for research and educational purposes, but please adhere to ethical guidelines when using it for security testing. We explicitly prohibit the use of this benchmark for any malicious activities or unauthorized penetration testing. Always ensure you have proper authorization before conducting any security assessments.

Citation

If you use this benchmark in your research, please cite the following paper: