<div align="center">
 <img src="./logo.png" alt="AutoPass Logo" width=25% height=20%>
</div>
<p align="center" >
  <i>AutoPass: Evidence-Guided LLM Agents for Compiler Performance Tuning</i>
</p>



## Introduction

AutoPass is an AI-driven compiler optimization framework that **automatically searches and refines LLVM pass pipelines** using a multi-agent workflow (built on CrewAI) and performance feedback.


It is built on a simple idea: **compiler optimization should be driven by evidence, not guesswork**.
Traditional compiler optimization usually falls into two extremes. (1) Fixed heuristics, such as `-O3` and PGO, which are robust but often conservative.
(2) Full search/autotuning, which can find strong pipelines but is often too expensive for practical use. AutoPass aims for a practical middle ground. It uses LLM-based reasoning, but only inside a constrained, compiler-aware, feedback-driven optimization loop.

- **What this repository provides**: code to compile the required LLVM external pass plugin and run the AutoPass optimization workflow on benchmarks.
- **Paper PDFs**:
  - **Appendix**: (PDF: [`paper/appendix.pdf`](paper/appendix.pdf), download: [`appendix.pdf?raw=1`](paper/appendix.pdf?raw=1))
  - **Case study (full trace, QuickSort)**: [`paper/Qsort-trace.md`](paper/QsortTrace.md)
  - **Failure case study**: [`paper/failure-case.md`](paper/FailureCaseStudy.md)

## Installation

### System requirements (tested)

- **OS**: Ubuntu 24.04 (recommended).  
  - Windows users: we recommend **WSL2 (Ubuntu)** because some scripts invoke `/bin/bash`.
- **Python**: 3.10
- **LLVM toolchain**: LLVM 17.x
  - If compiling the external pass as a plugin, use an LLVM toolchain built with shared libs: `-DBUILD_SHARED_LIBS=ON`.
- **Python deps**: see `requirements.txt` (includes `crewai==1.1.0`).

### Python environment

From the repository root:

```bash
python3.10 -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
```

## Usage

### Build the external LLVM pass (required)
Follow `external/README.md` to build the plugin:

- `external/README.md`

### Configure benchmarks + LLVM paths
The benchmark list is defined by JSON files under `sources/loadbenchs/` (default: `sources/loadbenchs/cbench/benchmarks.json`).

You must replace placeholder paths in the benchmark JSON and scripts (details in `usage.md`):

- **`<project_path>`**: absolute path to this repo (used by benchmark definitions)
- **`<llvm_path>`**: absolute path to your LLVM toolchain
- **`<python_interpreter_path>`**: absolute path to your python interpreter

### Configure LLM access
AutoPass uses CrewAI-based agents. Configure your model provider settings in `sources/utils/agents.py` or via a `.env` file (recommended). See `usage.md` for details.

### Run AutoPass (demo)
From the repository root:

```bash
cd sources
chmod +x -R ./scripts
python ./RunAutoPass.py
```

Outputs:
- **Per-run logs**: `sources/log/log_<timestamp>/`
- **Benchmark artifacts**: created inside each benchmark directory (e.g., `IR/`, `perf_jsons/`, `remarks/`, `features/`)

For a step-by-step walkthrough, see `usage.md`.

## Benchmarks

Benchmarks live under `sources/benchmarks/`. A small demo benchmark is included at:

- `sources/benchmarks/cbench_demo/`

To add more benchmarks:

- **Place code**: under `sources/benchmarks/<suite_name>/...`
- **Register them**: update the JSON under `sources/loadbenchs/` (e.g., `sources/loadbenchs/cbench/benchmarks.json`)

Suggested benchmark sources are listed in `benchmarks_list.md`.

### Input/Output of Benchmarks

Some benchmarks require input data at runtime. Put the data in a stable location and set the JSON field `runflags` accordingly (used by `sources/scripts/perf_.sh`).

## Results

AutoPass produces:

- **Optimization process logs** (including agent interactions): `sources/log/log_<timestamp>/`
- **Performance JSONs**: `perf_jsons/` under each benchmark directory (e.g., the demo includes `sources/benchmarks/cbench_demo/automotive_qsort1/perf_jsons/`)

### Cost Analysis (per optimization session)

AutoPass uses a small set of LLM agent calls per benchmark session. Table below summarizes the token consumption and estimated API cost for a complete optimization run (including 3 iterative rounds for the Reasoning/Evaluation agents):

| **Agent Role** | **Session Input (Tokens)** | **Session Output (Tokens)** | **Est. Cost (Session)** |
|---|---:|---:|---:|
| **Score** | 2.4k | 0.8k | $0.001 |
| **Analysis** | 28.2k | 0.5k | $0.008 |
| **Reasoning*** | 74.5k (24.8k × 3) | 25.1k (8.36k × 3) | **$0.030** |
| **Evaluation*** | 78.7k (24.8k × 3) | 8.7k (2.90k × 3) | $0.027 |
| **Total** | **~184k** | **~35k** | **~$0.066** |

*Values reflect a cumulative sum over 3 iterative rounds.

To minimize expenses, we employ a hybrid strategy: the lightweight DeepSeek-V3.2 *non-thinking* model handles the Score, Analysis, and Evaluation agents, while the more computationally intensive DeepSeek-V3.2 *thinking* model is used only for the Reasoning agent. The total cost per optimization run is approximately **$0.066**, which is substantially lower than traditional autotuning methods that often require hundreds of trial compilations.

More importantly, this cost model aligns well with practical deployment constraints. In realistic software settings—especially on mobile and edge devices—developers rarely have the budget to run large autotuning loops with many compile-and-run iterations on the target device. Profile-guided optimization is cheaper operationally, but it typically relies on a single profiling workflow and therefore offers limited opportunity for iterative diagnosis and correction. By contrast, AutoPass uses only a small number of feedback rounds and can support a split workflow where runtime evidence is collected on the target device while the more expensive reasoning is performed off-device. This makes the framework more compatible with practical performance-tuning scenarios, where target-specific feedback is essential but on-device tuning overhead must remain small.


## Core Insights

### 1. Good optimization needs context, not just hotness
The most important optimization targets are not always the functions or loops that look hottest in a profile.

In many cases, the real performance bottleneck depends on:

- the role of a code region in the full algorithm
- how transformations interact with control flow and layout
- how the target hardware responds to those changes

AutoPass is designed around **contextual interpretation** rather than local hotness alone.

---

### 2. LLMs should make evidence-based decisions
LLMs are good at high-level reasoning, but raw generation is not enough for compiler optimization.

Instead of letting the model invent pipelines from intuition alone, AutoPass first collects:

- LLVM IR structure
- compiler optimization remarks
- target architecture information
- runtime feedback from previous rounds

The model then reasons over this evidence before proposing changes.

This turns the LLM from a text generator into an **evidence-guided optimization agent**.

---

### 3. Practical optimization should use constrained editing
In real deployment, unconstrained search is too expensive, and open-ended generation is too unstable.

AutoPass therefore follows a constrained strategy:

- start from the compiler-supported `-O3` pipeline
- edit only valid passes already present in the baseline pipeline
- tune only a small set of bounded parameters
- validate and repair candidate pipelines automatically

This makes the system safer, cheaper, and more realistic for practical use.

---

### 4. Multi-agent design matters because it separates the workflow
The value of AutoPass is not just that it uses multiple agents.  
The real value is that different stages handle different kinds of evidence and decisions.

- **Score Agent**: narrows the context and selects high-impact regions
- **Analysis Agent**: summarizes IR structure and compiler remarks
- **Reasoning Agent**: proposes pass edits and parameter changes
- **Evaluation Agent**: validates results, compares performance, and returns feedback


