When the system operator changes from a human to an LLM Agent, design principles need fundamental rethinking. Humans need GUIs and documentation. Agents need semantically clear tools and constraints that throw errors.
The most common lie in ML research is ’the results looked great last time.’ What code was used last time? What data version? What parameters? Nobody can say. I used filesystem transactions (temp directory → atomic rename) to create immutable snapshots of every iteration, turning ’last time’s results’ from a memory into a queryable fact.
AI-generated code must be executed — otherwise it’s just text. But execution means risk. I didn’t choose container isolation or RestrictedPython. Instead I designed a three-layer defense: reject dangerous structures at compile time via AST, replace the entire builtins at runtime, and enforce OS-level resource limits as a backstop. Each layer handles a different class of risk. Overlapping but not redundant.
LangChain, LangGraph, CrewAI, PydanticAI — no shortage of AI orchestration frameworks. I evaluated all of them and built my own. Not NIH syndrome. When you need failure-mode-driven mutation strategies, phase-aware multi-model routing with different temperatures, and adaptive evolution based on trajectory analysis, the abstraction layers of general-purpose frameworks become obstacles to route around.
Letting AI drive research workflows doesn’t mean letting AI decide how the system runs. I made a key separation: AI is just the operator, the execution engine is the law. This decision came from a failure.
Most systems are designed to run first, then audited as an afterthought. I inverted the order — first define what questions the system must answer when things go wrong, then work backwards to what each layer must record. This inversion reshaped the entire architecture.