SWE-agent, Devin, Codex CLI, Claude Code — how AI writes, understands, and fixes software at repository scale
Core Concept
From Autocomplete to Autonomous Software Engineer
Think of the evolution in three stages. Stage 1 was autocomplete — Copilot-style systems that predict the next few lines inside a single file. Stage 2 was chat-and-edit — you describe a change in English, the model edits one file at a time. Stage 3, where we are now, is the code agent: a system that reads an entire repository, plans a multi-file strategy, writes code, runs tests, reads the error output, fixes failures, and iterates — all autonomously. The human provides a goal (a GitHub issue, a feature spec, a bug report); the agent delivers a pull request.
1The Code Agent Loop
Every code agent follows a variant of the same loop, regardless of vendor:
Issue / Task
natural language
→
Localize
find relevant files
→
Edit
write patches
Validate
PR / merge
←
Test
run suite, lint
←
Debug
read errors
↩ loop continues until tests pass or iteration limit is reached
The critical insight from the SWE-agent paper is that the interface between the LLM and the codebase matters as much as the model itself. SWE-agent's custom Agent-Computer Interface (ACI) provides simplified commands for viewing, searching, and editing files, with guardrails that prevent common mistakes (like forgetting to save). This custom interface boosted performance far more than simply using a better model with a raw shell.
Key lesson: don't just give an LLM a bash shell. Design purpose-built commands that constrain the action space and give clear, concise feedback. This is why every serious code agent (Codex CLI, Claude Code, Aider) builds a custom tool layer on top of the filesystem.
2Codebase Understanding: The Hard Problem
A typical production repository might have 500,000 lines of code across 3,000 files. No LLM context window fits all of that. Code agents solve this with a localization pipeline that narrows from the entire repo to the exact lines that need editing:
Step 1 — Repository Map. Build a lightweight structural index: file tree, class/function signatures, import graphs. SWE-agent generates a "repo map" at startup. Claude Code reads CLAUDE.md and indexes project structure. Devin auto-generates architecture wikis from your codebase.
Step 2 — Semantic Search. Given the issue description, use embedding-based retrieval (or BM25 keyword search) to find the top candidate files. This is essentially RAG applied to source code.
Step 3 — Hierarchical Localization. The Agentless paper showed a clean three-pass approach: first identify relevant files, then narrow to relevant classes/functions within those files, then pinpoint exact edit locations. This hierarchical strategy is cheaper and often more accurate than letting an agent freely explore.
Step 4 — Context Assembly. Combine the localized code snippets with the issue description, relevant test files, and any documentation into a single prompt. This is the "context engineering" that Harrison Chase calls the make-or-break skill of the agent era.
The #1 failure mode of code agents is wrong localization — editing the wrong file or function. On SWE-bench, agents that correctly localize the bug fix it ~80% of the time; agents that mis-localize almost never recover.
3Agent vs. Agentless: A Fundamental Design Choice
The field has split into two philosophies:
Agentic
Free-form tool use (SWE-agent, Devin, Claude Code)
The LLM has a loop with shell access, can explore freely, run tests, backtrack. Flexible but expensive (many LLM calls) and sometimes goes down rabbit holes. Best for complex, open-ended tasks where the fix isn't obvious.
Agentless
Pipeline approach (Agentless, Alibaba Lingma)
A fixed three-phase pipeline: localize → repair → validate. No open-ended tool use. Cheaper ($0.70/issue vs $4+), more predictable, easier to debug. Surprisingly competitive — Agentless matched or beat many agent systems on SWE-bench.
Hybrid
Structured agent (Codex CLI, Aider)
Uses agent loops but within guardrails: constrained edit formats (diff blocks, search-and-replace), mandatory test verification, structured output. Balances flexibility with reliability.
IDE-integrated
Cursor, Gemini Code Assist, GitHub Copilot
Agent runs inside the IDE with access to the editor state, language server (types, references, errors), and terminal. Tight integration enables features like "Finish Changes" (Gemini) and multi-file refactors with type checking.
For botlearn.ai: the agent vs. agentless distinction maps directly to product positioning. Agentless is great for well-defined bug fixes (support tickets). Agentic is needed for open-ended feature development. Your customers likely need both.
4How Code Agents Handle Edits
One of the trickiest implementation details is how the agent specifies code changes. Raw "rewrite the whole file" doesn't scale — it wastes tokens and introduces regressions. Modern approaches:
Search-and-replace blocks (Aider, Claude Code): the model outputs an old_string/new_string pair. The tool finds the exact match and replaces it. This is precise and diff-friendly, but the model must reproduce the old code exactly (whitespace and all).
Unified diff format (Codex CLI): the model outputs standard unified diffs. Familiar to developers, easy to review, and git-compatible. But LLMs sometimes produce malformed diffs.
Full-file rewrite with validation (Devin): Devin writes entire files in its cloud IDE, then runs linting and tests immediately. The trade-off is higher token cost but fewer edit-application failures.
AST-aware edits (research frontier): Instead of text diffs, operate on the Abstract Syntax Tree. Guarantees syntactically valid code but requires language-specific tooling.
Papers to Know
Landmark & Recent Research
NeurIPS 2024
SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, Ofir Press — Princeton University
SWE-agent demonstrated that how an LLM interacts with a codebase matters as much as which model you use. The team designed a custom Agent-Computer Interface (ACI) with purpose-built commands for file navigation, search, and editing — replacing the raw Linux shell. With GPT-4, SWE-agent achieved 12.5% on SWE-bench (the leading score at the time) and 87.7% on HumanEvalFix. The key insight: guardrailed, feedback-rich interfaces dramatically reduce agent errors compared to open-ended shell access.
Why it matters: Established the principle that tool design is a first-class concern in agent engineering. Every subsequent code agent (Codex CLI, Claude Code, Aider) adopted purpose-built command interfaces inspired by this work. The paper also popularized SWE-bench as the standard benchmark.
arXiv 2405.15793
arXiv preprint 2024
Agentless: Demystifying LLM-based Software Engineering Agents
Chunqiu Steven Xia, Yinlin Deng, Soren Dunn, Lingming Zhang — UIUC
Agentless challenges the assumption that you need complex agent loops for software engineering. Instead, it uses a simple three-phase pipeline: (1) hierarchical localization (file → class → line), (2) patch generation using the localized context, and (3) patch validation via test execution. Despite its simplicity, Agentless achieved 32% on SWE-bench Lite at a cost of just $0.70 per issue — matching or beating most agent-based systems at a fraction of the cost. OpenAI adopted Agentless as their go-to approach for showcasing GPT-4o and o1 coding performance.
Why it matters: Proved that agentic complexity isn't always necessary. For well-defined bugs, a structured pipeline can match agents at 5-10x lower cost. This has major implications for production deployment, where cost-per-task drives adoption.
arXiv 2407.01489
arXiv preprint 2025
CodeAgent: Enhancing Code Generation with Tool-Integrated Agent Systems for Real-World Repo-level Coding Challenges
Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, Zhi Jin — Peking University
CodeAgent integrates five specialized tools (code search, code navigation, test execution, web search, and documentation lookup) into a single agent framework for repo-level code generation. The paper focuses on the challenge that real-world coding requires more than just an LLM — it requires understanding project structure, dependencies, APIs, and conventions. CodeAgent outperformed base models and retrieval-augmented baselines on repo-level benchmarks, demonstrating that tool diversity (not just a better model) is key to real-world code generation.
Why it matters: Showed that the "tool portfolio" of a code agent — what tools it can call and how it orchestrates them — is a primary differentiator. This mirrors the MCP ecosystem (Day 03): the agents that win will be the ones with the richest, most reliable tool integrations.
arXiv 2401.07339
Community Pulse
What the Experts Are Saying
Karpathy has argued that coding agents crossed from "unreliable" to "functional" in December 2025, flipping his personal ratio from 80% writing code himself to 80% delegating to agents. He's coined the term "agentic engineering" as the successor to "vibe coding" — emphasizing that the new core skills are intent specification, task decomposition, and fast code review, not line-by-line coding.
Source: Multiple X posts and interviews, early 2026. Karpathy describes the shift as "programming becoming unrecognizable since December."
Willison has started an "Agentic Engineering Patterns" documentation project, collecting and cataloging best practices for working with coding agents. At NICAR 2026, he presented a three-hour workshop demonstrating Claude Code and Codex CLI for data analysis, web scraping, and data cleaning tasks — emphasizing that coding agents are now practical tools for non-software-engineers.
Source: simonwillison.net/tags/ai-assisted-programming and NICAR 2026 workshop materials.
Chase has emphasized that "when agents mess up, they mess up because they don't have the right context." He argues that context engineering — bringing the right information in the right format to the LLM at the right time — is the new moat for coding agents. LangChain released "Deep Agents" in March 2026, built on LangGraph, with virtual filesystem access and code execution capabilities.
Source: Sequoia Capital "Training Data" podcast, and LangChain March 2026 newsletter.
Expert Questions
Questions That Signal Deep Understanding
Q1
"Agentless achieves competitive SWE-bench scores at $0.70/issue versus $4+ for agentic systems. At what task complexity threshold does the agentic approach start to justify its 5-10x cost premium — and how would you measure that threshold?"
Q2
"SWE-agent showed that ACI design matters more than model choice. If you were designing an ACI for a polyglot monorepo (Python + TypeScript + Go), what commands would you add beyond the standard file/search/edit set?"
Q3
"Code agents that rely on search-and-replace edits need the model to reproduce existing code exactly. How does this constraint interact with model temperature settings, and what happens when the target code contains unusual formatting or non-ASCII characters?"
Q4
"Karpathy says agentic engineering skills are 'intent specification, task decomposition, and fast review.' But current SWE-bench doesn't test any of these human-agent interaction skills. What would a benchmark for agentic engineering look like?"
Q5
"Devin auto-generates architecture wikis from codebases. Claude Code uses CLAUDE.md context files. Codex CLI uses codex.md. Are we converging on a standard for 'agent-readable project documentation,' and should it be a new file format or an extension of existing formats like README?"