[{"content":" Single-LLM self-reflection has a structural blind spot — the model tends to confirm its own output. QuantGPT enforces a hard rule in factor-mine SKILL Phase 0.5: Claude must consult DeepSeek before designing a new factor family. Not a suggestion, a hard rule. This isn\u0026rsquo;t redundancy — it\u0026rsquo;s the antidote to structural bias.\n1. The Structural Blind Spot of Single-LLM Self-Reflection Letting Claude reflect on its own output is a standard pattern in LLM applications. Reflection, Self-Critique, Chain-of-Thought — different names, same essence: let the model evaluate itself with its own capabilities.\nThis pattern works on small tasks. But on domain-deep tasks it has a fundamental limitation: a model\u0026rsquo;s reflection still falls within its own training distribution.\nConcrete scenario. Claude designs a factor expression:\n-1 * rank(ts_av_diff(close, 10)) + rank(debt / enterprise_value) Now you ask Claude to review this expression. It will output something like: \u0026ldquo;This factor combines price reversal with a fundamental signal, theoretically reasonable, aligned with WQ BRAIN style.\u0026rdquo;\nSounds right. But this is the same model evaluating the same model\u0026rsquo;s output — the cognitive toolkit it uses is the one that produced the original. If the original design has a bias (e.g., overfitting to a specific factor structure), the reflection carries that same bias.\nWorse: RLHF dialogue training pushes models toward \u0026ldquo;completing the task\u0026rdquo;. Asked to \u0026ldquo;reflect on the factor I just designed\u0026rdquo;, a model leans toward supporting itself — because rejecting itself means rewriting, extending the conversation, and delaying delivery.\nThis is a structural problem, not something prompt engineering fixes.\n2. Cross-Model Review ≠ \u0026ldquo;AI Checking AI\u0026rdquo; The first instinct is naive: just have another LLM check it. But not any LLM.\nThe effectiveness of cross-model review comes from three differences:\nTraining data distribution — different models train on different corpora, with different blind spots and strengths RLHF trajectory — different human-feedback data shapes different judgments of \u0026ldquo;what counts as correct\u0026rdquo; Reasoning style — different chain-of-thought tendencies If you use GPT-4o to review Claude\u0026rsquo;s output: both are English-dominant, both aligned to general helpfulness, both with similar reasoning styles. The differences aren\u0026rsquo;t large enough; the complementary value is limited.\nEngineering cross-model review requires picking a model with real distributional differences and stronger capability in the target domain. For quant factor research, that model is DeepSeek.\n3. Why DeepSeek, Specifically I\u0026rsquo;ve used DeepSeek for review for several months. Here\u0026rsquo;s why it\u0026rsquo;s the optimal choice for the quant scenario right now — not because it\u0026rsquo;s cheap, but because the distribution aligns.\nIt Comes From a Quant Firm DeepSeek\u0026rsquo;s parent company is High-Flyer Quantitative Investment — one of China\u0026rsquo;s tens-of-billions-RMB AUM quant hedge funds.\nThis isn\u0026rsquo;t just a corporate-lineage label. It means:\nThe team has first-hand understanding of quant research workflow The training corpus naturally contains a high proportion of finance, statistics, and derivatives texts Training density on financial mathematics, factor analysis, and backtest semantics is far higher than general-purpose models Having Claude\u0026rsquo;s factor expression reviewed by a model trained by a team that does quant for a living is distributional alignment, not a gimmick.\nThe Best Choice for Chinese Financial Reasoning Claude\u0026rsquo;s training data is English-dominant. It can handle phrases like \u0026ldquo;CSI 500 industry-neutralized\u0026rdquo;, but not as naturally as it handles \u0026ldquo;S\u0026amp;P 500 sector neutralized\u0026rdquo;.\nDeepSeek\u0026rsquo;s density on Chinese financial corpora is far higher than general-purpose models. When factor design touches A-share market structure (price-limit rules, ST/halt status, two-sided market behavior, Wind industry classification), DeepSeek\u0026rsquo;s feedback is often more concrete and accurate.\nThis isn\u0026rsquo;t a performance difference (both have strong reasoning). It\u0026rsquo;s a domain grounding difference.\nR1 Exposes Reasoning Traces DeepSeek-R1 exposes a reasoning_content field — you see the model\u0026rsquo;s complete chain-of-thought:\nresult = ask_deepseek(prompt, model=\u0026#34;deepseek-reasoner\u0026#34;) print(result[\u0026#34;content\u0026#34;]) # final answer print(result[\u0026#34;reasoning\u0026#34;]) # full reasoning trace For review scenarios, the reasoning trace matters more than the answer. The reasoning path explaining \u0026ldquo;why this factor might overfit\u0026rdquo; is 10x more useful than a one-liner saying \u0026ldquo;consider simplifying\u0026rdquo; — the former teaches Claude something it can act on, the latter is noise.\nOpenAI o1 hides reasoning traces behind the product. DeepSeek exposes them directly. For developers this is a quality difference; for autonomous Agent systems this is a usability difference.\nReasoning Depth Comparable to o1, Pricing an Order of Magnitude Lower DeepSeek-R1 benchmarks comparably to OpenAI o1 on AIME / MATH-500 / GPQA reasoning tests.\nPricing comparison (early 2026):\nModel Input (per 1M tokens) Output (per 1M tokens) DeepSeek-R1 ~$0.55 ~$2.19 OpenAI o1 ~$15 ~$60 The gap is about 27x.\nEach Phase 0.5 review consumes around 5K-10K tokens, costing ~$0.003-0.007 per call. Cheap enough to make \u0026ldquo;review\u0026rdquo; the default behavior, not a luxury. The price isn\u0026rsquo;t the cause — but it\u0026rsquo;s what makes \u0026ldquo;mandatory review\u0026rdquo; a viable engineering design.\n4. The Implementation in QuantGPT QuantGPT enforces DeepSeek consultation in factor-mine SKILL\u0026rsquo;s Phase 0.5: DeepSeek Design Consultation.\nTrigger Conditions - Starting a new research direction - Existing signal family has reached SC saturation, needs new structure - No reusable expression template found in the knowledge base Any one of these triggers it. Cannot be skipped within the SKILL flow.\nMCP Tool Implementation The DeepSeek MCP server is a 196-line stdio JSON-RPC service (scripts/mcp_deepseek.py). Core definition:\nTOOLS = [{ \u0026#34;name\u0026#34;: \u0026#34;ask_deepseek\u0026#34;, \u0026#34;description\u0026#34;: ( \u0026#34;Send a prompt to DeepSeek LLM. \u0026#34; \u0026#34;Use for: Chinese financial reasoning, factor expression generation, \u0026#34; \u0026#34;alternative perspectives, or tasks benefiting from \u0026#34; \u0026#34;DeepSeek Reasoner\u0026#39;s chain-of-thought.\u0026#34; ), \u0026#34;inputSchema\u0026#34;: { \u0026#34;properties\u0026#34;: { \u0026#34;prompt\u0026#34;: {\u0026#34;type\u0026#34;: \u0026#34;string\u0026#34;}, \u0026#34;model\u0026#34;: {\u0026#34;type\u0026#34;: \u0026#34;string\u0026#34;, \u0026#34;default\u0026#34;: \u0026#34;deepseek-reasoner\u0026#34;}, \u0026#34;system\u0026#34;: {\u0026#34;type\u0026#34;: \u0026#34;string\u0026#34;}, \u0026#34;temperature\u0026#34;: {\u0026#34;type\u0026#34;: \u0026#34;number\u0026#34;, \u0026#34;default\u0026#34;: 0.7}, }, } }] Registered to Claude Code via .mcp.json. Claude Agent sees the tool and can invoke it.\nEngineering the Review Prompt Not a simple \u0026ldquo;what do you think of this factor\u0026rdquo;. Phase 0.5 feeds DS a structured payload:\nFacts layer: - Current research direction (from research_notes/archive/) - Verified rules (from knowledge/rules/) - Falsified paths (from knowledge/failures/) - Claude\u0026#39;s draft design + reasoning Request layer: - Evaluate whether this design has unidentified risks - Suggest 1-2 alternative structures - Flag potential conflicts with known failures DS returns reasoning_content + content. Claude must explicitly accept/reject/adjust, and write the decision into research notes. This isn\u0026rsquo;t a discussion — it\u0026rsquo;s part of the engineering pipeline.\n5. Why \u0026ldquo;Mandatory\u0026rdquo;, Not \u0026ldquo;Suggested\u0026rdquo; Making cross-model review optional doesn\u0026rsquo;t work. Two reasons.\nAgents Skip Non-Required Steps LLM dialogue training pushes Agents toward the shortest path to task completion. \u0026ldquo;Suggest you ask DeepSeek\u0026rdquo; written a thousand times in the prompt — when the Agent is racing to finish, it\u0026rsquo;ll skip. Because one more API call, 30s of waiting, and the writeback is all latency.\nPrompt-Level Constraints Have No Enforcement I argued in detail in Harness Is Governance: violating a prompt produces no consequences. The Agent skips a \u0026ldquo;suggestion\u0026rdquo;, the flow continues, and there\u0026rsquo;s no feedback signal telling it that this was wrong.\nSolution: Hard Rules + Flow Dependency QuantGPT writes Phase 0.5 as a hard rule in the SKILL. Not a \u0026ldquo;suggestion\u0026rdquo; in a prompt — a flow dependency. The subsequent Phase 1 prompt template explicitly references the DS review result as context. Skip 0.5, and Phase 1\u0026rsquo;s context is missing, the flow fails.\nThis is Harness Is Governance applied: constrain Agents with code, not prompts.\n6. Trade-offs and Boundaries Cross-model review isn\u0026rsquo;t a free lunch.\nCost Per Phase 0.5: ~$0.003-0.007 + 30s latency One full research cycle (4-8 Phase 0.5 calls): ~$0.03-0.05 About 30%-50% higher token cost vs. a pure single-LLM pipeline Both Models Failing Together If a bias exists in both Claude\u0026rsquo;s and DeepSeek\u0026rsquo;s training distributions (e.g., a shared interpretation of certain classical factors as overfitted), both models pass it together — and the review fails.\nThis \u0026ldquo;distribution overlap\u0026rdquo; blind zone has no perfect solution. Mitigations:\nAdd a third model (GPT-4o / Qwen / Llama 3) as arbiter When review pass rates exceed 90% over time, inject adversarial prompts (\u0026ldquo;assume this factor is overfitted, find evidence\u0026rdquo;) Maintain a hard-coded blacklist for historically falsified factor structures (not relying on any LLM judgment) When It Doesn\u0026rsquo;t Apply Task domain has only one model with real knowledge (e.g., niche medical database) — cross-model review has no signal Latency-sensitive task (millisecond range) — extra API call is a deal breaker Team already has human review process — adding LLM review is redundant Conclusion Cross-model review is an architecture-level design choice, not a prompt asking the model to \u0026ldquo;think more carefully\u0026rdquo;.\nSingle-LLM self-reflection\u0026rsquo;s limit comes from training distribution — and training distribution can only be cut by another distribution. When the review target is quant factors, that \u0026ldquo;other distribution\u0026rdquo; has essentially one choice: DeepSeek. It comes from a quant firm, trains on heavy Chinese financial corpora, reasons at o1 depth, and costs an order of magnitude less — cheap enough to make it default behavior.\nQuantGPT enforces Phase 0.5 not because I believe DeepSeek is always right — it\u0026rsquo;s wrong sometimes. It\u0026rsquo;s because the probability of two models with real distributional difference being wrong simultaneously is significantly lower than one model being wrong on its own. Architecture\u0026rsquo;s job isn\u0026rsquo;t to eliminate errors, it\u0026rsquo;s to reduce error rates to acceptable levels.\nCross-Model Review Is Architecture, Not Heuristic.\n","permalink":"https://miasyster.github.io/en/posts/cross-model-review-is-architecture/","summary":"Single-LLM self-reflection has a structural blind spot — the model tends to confirm its own output. QuantGPT enforces a hard rule in factor-mine SKILL Phase 0.5: Claude must consult DeepSeek before designing a new factor family. Not a suggestion, a hard rule. This isn\u0026rsquo;t redundancy — it\u0026rsquo;s the antidote to structural bias.","title":"Cross-Model Review Is Architecture, Not Heuristic"},{"content":" When the system operator changes from a human to an LLM Agent, design principles need fundamental rethinking. Humans need GUIs and documentation. Agents need semantically clear tools and constraints that throw errors.\nThe Operator Changed. So Must the Design. Traditional software is designed for humans: GUIs guide workflows, documentation explains features, error messages help humans understand problems.\nWhen the operator becomes an LLM Agent, this design breaks down. Agents don\u0026rsquo;t look at GUIs, don\u0026rsquo;t read documentation (at least not the way humans do), and don\u0026rsquo;t need friendly error messages — they need machine-parseable error information to adjust their next action.\nAgent-Native architecture isn\u0026rsquo;t \u0026ldquo;add an API for the Agent.\u0026rdquo; It\u0026rsquo;s rethinking every design decision from the premise that the system\u0026rsquo;s operator is an Agent.\nPrinciple 1: Tools Match the Caller\u0026rsquo;s Cognitive Model Humans excel at composing small tools to accomplish tasks (Unix philosophy: cat | grep | sort). LLMs don\u0026rsquo;t — they excel at extracting key information from a single complete result.\nQuantGPT\u0026rsquo;s run_backtest returns everything in one call: Sharpe, IC, group returns, turnover, max drawdown, industry exposure, factor loadings. Not split into 7 small tools for the Agent to call one by one.\n# Human-friendly design: 7 small tools get_sharpe(factor_id) get_ic(factor_id) get_turnover(factor_id) # ...Agent needs 7 calls to see the full picture # Agent-Native design: 1 complete tool run_backtest(expression, universe) → {sharpe, ic, turnover, drawdown, ...everything} # Agent sees the complete picture in one call, decides what to focus on Tool design should match the caller\u0026rsquo;s cognitive model, not the designer\u0026rsquo;s aesthetic preference. \u0026ldquo;Small and focused\u0026rdquo; is a human aesthetic; \u0026ldquo;complete and self-contained\u0026rdquo; is an Agent\u0026rsquo;s need.\nPrinciple 2: Errors Are Interface, Not Exceptions Humans see error messages, think about the cause, and manually fix the problem. Agents parse error content and automatically adjust.\nThis means error message design shifts from \u0026ldquo;help humans understand\u0026rdquo; to \u0026ldquo;help Agents act\u0026rdquo;:\n# Human-friendly error raise ValueError(\u0026#34;Expression syntax error\u0026#34;) # Agent-Native error raise ValueError( \u0026#34;At character 23: \u0026#39;ts_regrssion\u0026#39; is not a valid operator. \u0026#34; \u0026#34;Did you mean \u0026#39;ts_regression\u0026#39;? \u0026#34; \u0026#34;Available time-series operators: ts_mean, ts_std, ts_corr, ts_regression, ...\u0026#34; ) The second error contains three layers: what\u0026rsquo;s wrong (location), possible correction (suggestion), available alternatives (operator list). The Agent can parse this and directly generate a corrected expression without an additional list_operators call.\nPrinciple 3: Constraints in Code, Not Documentation Documentation constraints (\u0026ldquo;please call through the API\u0026rdquo;) have zero binding force on Agents. Agents take the shortest path — if directly importing a function is faster, they\u0026rsquo;ll bypass the API.\nAgent-Native system constraints must be runtime-enforced:\n_api_context = threading.local() def _require_api_context(): if not getattr(_api_context, \u0026#39;active\u0026#39;, False): raise RuntimeError(\u0026#34;Must be called through API\u0026#34;) Agents only respect rules that throw errors. Rules that don\u0026rsquo;t throw errors don\u0026rsquo;t exist.\nSimilarly, the expression parser\u0026rsquo;s security limits:\nMAX_DEPTH = 100 # Recursion depth MAX_WINDOW = 500 # Rolling window MAX_EXPRESSION_LENGTH = 1000 # Expression length Not advisory values — hard limits. Exceed them and an exception is thrown, forcing the Agent to revise its input.\nPrinciple 4: Stateless \u0026gt; Stateful Stateful tools mean call order matters — you must call A before B, otherwise B can\u0026rsquo;t read the state A set. This is an implicit constraint on the Agent, and it\u0026rsquo;s a documentation-level one (you need to tell the Agent \u0026ldquo;call A first\u0026rdquo;), not a code-level one.\nStateless tools eliminate this problem. Every call is independently complete, and the Agent can call any tool in any order at any frequency.\n@mcp.tool() async def run_backtest(expression: str, universe: str = \u0026#34;hs300\u0026#34;, ...): # Reads no global state # Depends on no previous call\u0026#39;s result # Returns a complete backtest report ... The tool doesn\u0026rsquo;t know if it\u0026rsquo;s being called \u0026ldquo;for the first time\u0026rdquo; or \u0026ldquo;on iteration 15.\u0026rdquo; It doesn\u0026rsquo;t care about context. This means any decision path the Agent takes is valid — there\u0026rsquo;s no such thing as \u0026ldquo;wrong call order.\u0026rdquo;\nPrinciple 5: Semantic Naming Is Routing Human systems need routers to dispatch requests. Agent-Native systems don\u0026rsquo;t — the tool name itself is the routing.\nrun_backtest — I want to know how this factor performs score_factor — I want a composite score diagnose_factor — I want to know why it\u0026#39;s underperforming run_anti_overfit — I want to know if it\u0026#39;s overfitting LLMs select tools based on names and descriptions. The closer the name is to intent description, the higher the selection accuracy. No need to write if intent == \u0026quot;diagnose\u0026quot;: call diagnose_factor in Agent code — the model does this mapping naturally.\nTestability: The Underestimated Advantage Agent decisions are stochastic — same input, different runs may produce different outputs. You can\u0026rsquo;t write unit tests for Agent behavior.\nBut the Agent-Native tool layer is deterministic. QuantGPT has 74 tests covering:\nMathematical correctness of 80+ operators Cross-sectional/time-series grouping semantics Anti-overfit statistical tests WQ BRAIN metric calculations Tests guarantee: regardless of how the Agent calls the tools, the returned results are correct. Agent decision quality depends on model capability; data correctness depends on infrastructure — the latter is what you can control and verify.\nOne-Line Summary Designing for Agents and designing for humans are two different things. Complete returns \u0026gt; small-and-focused, runtime enforcement \u0026gt; documentation constraints, stateless \u0026gt; stateful, semantic naming \u0026gt; explicit routing. When your operator is an LLM, every design decision needs re-examination.\nSeries: Why Agent-Infra, Not Agents · Harness Is Governance · Skill Orchestration \u0026gt; Agent Loop Chains\n","permalink":"https://miasyster.github.io/en/posts/agent-native-architecture/","summary":"When the system operator changes from a human to an LLM Agent, design principles need fundamental rethinking. Humans need GUIs and documentation. Agents need semantically clear tools and constraints that throw errors.","title":"Agent-Native Architecture: Designing Systems for Agents, Not Humans"},{"content":" The mainstream approach to Agent governance is writing rules in prompts. But LLMs can ignore prompts. Real governance lives outside the Agent, in the Harness layer — tools define what\u0026rsquo;s possible, errors define what\u0026rsquo;s forbidden, code defines where the boundaries are.\nWhy Prompt Governance Fails When you write \u0026ldquo;backtests must go through the API\u0026rdquo; in a system prompt, you\u0026rsquo;re making an assumption: the Agent will obey text instructions.\nThis assumption holds most of the time. But it fails at the worst moments — when the Agent finds a more efficient path, it tends to take shortcuts. Early instructions get diluted in long contexts, and Agents can \u0026ldquo;forget\u0026rdquo; constraints across many conversation turns.\nThe fundamental problem: prompt-level constraints have no enforcement mechanism. Violating a prompt produces no consequences — the Agent keeps running, having done something you didn\u0026rsquo;t want. You might not discover it until you review the output.\nThis isn\u0026rsquo;t an LLM bug. It\u0026rsquo;s the inherent limitation of text-based constraints.\nThe Harness Layer: Interface Between Agent and System A harness isn\u0026rsquo;t a framework — it\u0026rsquo;s a position. The interface layer between the Agent and the domain system.\nAgent (Claude / GPT / DeepSeek) │ ├── Harness Layer ←── Governance lives here │ ├── MCP tool definitions (what\u0026#39;s possible) │ ├── Runtime guards (what\u0026#39;s forbidden) │ ├── Parameter validation (input boundaries) │ └── Return value design (output completeness) │ └── Domain System ├── Backtest engine ├── Expression parser ├── Anti-overfit detection └── WQ BRAIN simulator The Harness layer determines three things:\nWhat the Agent can do — through which tools are exposed What the Agent cannot do — through runtime guards and parameter validation What the Agent sees — through return value design Together, these three things constitute governance. Not a set of rule documents — a layer of code.\nMechanism 1: Capability Boundaries An Agent\u0026rsquo;s capabilities are defined by the tools it can call. Tools you don\u0026rsquo;t expose, the Agent simply cannot use.\nQuantGPT exposes 8 MCP tools. The Agent can\u0026rsquo;t directly access the database, can\u0026rsquo;t directly download market data, can\u0026rsquo;t directly modify the cache. It can only get backtest results through run_backtest — what the backtest engine does internally (data fetching, cache management, concurrency control) is none of the Agent\u0026rsquo;s business.\nThis is the same idea as operating system syscalls. Userspace programs can\u0026rsquo;t directly read/write disk — they must go through kernel syscalls. Agents can\u0026rsquo;t directly operate on domain systems — they must go through the Harness\u0026rsquo;s tools.\nNot exposed = doesn\u0026rsquo;t exist. This is infinitely more reliable than \u0026ldquo;please don\u0026rsquo;t call this function.\u0026rdquo;\nMechanism 2: Runtime Enforcement Exposed tools also need internal constraints. QuantGPT\u0026rsquo;s three layers of runtime defense:\nAPI boundary guard:\n_api_context = threading.local() def _require_api_context(): if not getattr(_api_context, \u0026#39;active\u0026#39;, False): raise RuntimeError(\u0026#34;Must be called through API\u0026#34;) The first line of the backtest function is this guard. If the Agent tries to import and call directly (bypassing MCP tools), it crashes immediately.\nExpression safety limits:\nMAX_DEPTH = 100 MAX_WINDOW = 500 MAX_EXPRESSION_LENGTH = 1000 Agent generates an expression exceeding limits? The parser refuses to execute. The Agent is forced to generate a simpler expression.\nDual-mode compilation:\nif mode == \u0026#34;wq\u0026#34; and operator not in WQ_COMPATIBLE_OPS: raise RuntimeError(f\u0026#34;\u0026#39;{operator}\u0026#39; not available in WQ BRAIN mode\u0026#34;) Agent wants to submit an expression with local-only operators to WQ BRAIN? Caught at compile time, not at platform submission.\nEvery layer is code-level enforcement — not suggestions, not documentation, not prompt instructions. Violations throw errors, and the Agent must adjust.\nMechanism 3: Information Boundaries Agent decision quality depends on the information it sees. The Harness controls this through return value design.\nQuantGPT\u0026rsquo;s score_factor returns 6-dimensional scoring: IC mean, IC_IR, stability, anti-overfit, group backtest, WQ alignment. The Agent doesn\u0026rsquo;t see a vague \u0026ldquo;good/bad\u0026rdquo; — it sees diagnostic information that reveals which dimension is dragging the score down, enabling targeted improvements.\nSimilarly, diagnose_factor returns not just \u0026ldquo;score is low\u0026rdquo; but specific failure modes and improvement suggestions. The Agent doesn\u0026rsquo;t need to infer \u0026ldquo;why is Sharpe low\u0026rdquo; — the tool tells it directly.\nInformation richness and structure determine the Agent\u0026rsquo;s decision ceiling. Give an Agent a number, it can only compare. Give it a diagnostic table, it can reason.\nMechanism 4: Cross-Review Single-Agent research has a fundamental problem: the model generating hypotheses and the model evaluating them is the same one. Confirmation bias is structural.\nQuantGPT solves this at the Harness layer: every research conclusion must go through independent review by a second LLM (DeepSeek).\nAgent (Claude) makes judgment │ ├── Collects factual data (backtest metrics) ├── Writes judgment + reasoning chain │ └── Harness layer: invokes DeepSeek review ├── Agrees → output conclusion └── Disagrees → present both positions, take conservative option This isn\u0026rsquo;t the Agent choosing whether to seek a second opinion — the Harness mandates it. The Agent can\u0026rsquo;t skip this step because the research conclusion output interface requires cross-review results.\nWhy Harness = Governance Summarizing the four mechanisms:\nGovernance Dimension Mechanism Implementation Layer What\u0026rsquo;s possible Tool exposure MCP tool definitions What\u0026rsquo;s forbidden Runtime guards threading.local + parser limits What\u0026rsquo;s visible Return value design Tool output information structure Decision reliability Cross-review Mandatory dual-LLM review All governance mechanisms live in the Harness layer — outside the Agent, outside the domain system, at the interface.\nThis position is crucial. If governance is inside the Agent (prompt), it\u0026rsquo;s fragile. If governance is inside the domain system (business logic), it\u0026rsquo;s coupled to the business. The Harness layer is the only position that\u0026rsquo;s independent of both the Agent and the domain system.\nPut governance in the Harness, and you can swap Agents, modify domain systems, and governance rules remain unaffected.\nOne-Line Summary Agent governance isn\u0026rsquo;t about writing better prompts. It\u0026rsquo;s about designing better Harnesses — using code to define capability boundaries, runtime constraints, information structure, and review processes. Agents only respect rules that throw errors, so write your rules as code.\nSeries: Why Agent-Infra, Not Agents · Agent-Native Architecture · AI as Operator, Kernel as Law\n","permalink":"https://miasyster.github.io/en/posts/harness-is-governance/","summary":"The mainstream approach to Agent governance is writing rules in prompts. But LLMs can ignore prompts. Real governance lives outside the Agent, in the Harness layer — tools define what\u0026rsquo;s possible, errors define what\u0026rsquo;s forbidden, code defines where the boundaries are.","title":"Harness Is Governance: Constraining Agents with Code, Not Prompts"},{"content":" While everyone is building Agent frameworks, I chose a different path: build infrastructure for Agents, not the Agent itself. Not because I can\u0026rsquo;t build Agents, but because the Agent layer is a consumable — infrastructure is an asset.\nA Counterintuitive Choice LangChain, CrewAI, AutoGen, Dify, Coze — a new Agent framework ships every month. They all solve the same problem: how to make LLMs complete multi-step tasks. Planner decides what to do, executor does it, reflector evaluates results, then loop.\nQuantGPT also uses an AI Agent. But I made a decision opposite to the mainstream:\nI didn\u0026rsquo;t write a single line of Agent code.\nAll Agent logic — planning, decision-making, iteration, reflection — is handled entirely by Claude itself. I did one thing: give it a set of good tools. 8 MCP tools, each a stateless pure function. No planner, no router, no reflection loop.\nThis isn\u0026rsquo;t laziness. It\u0026rsquo;s a strategic choice.\nHow Hard Building Agents Really Is \u0026ldquo;Building an Agent\u0026rdquo; sounds easy. Call an API, write a prompt, add a loop. But that\u0026rsquo;s a demo.\nReplicating a domain-depth Agent rivals building Claude Code from scratch Claude Code looks like \u0026ldquo;just API calls plus tool use,\u0026rdquo; but its engineering depth goes far beyond the surface: context engineering (when to compress history, when to discard), tool orchestration (priorities, conflict handling, failure retry), safety boundaries (which operations need confirmation), self-correction (how to backtrack from wrong paths).\nBuilding an Agent of equivalent depth for quantitative research means building a \u0026ldquo;quant Claude Code\u0026rdquo; from scratch. That\u0026rsquo;s not a workload an open-source project can absorb.\nThe Agent frontier moves too fast Early 2024 best practices (ReAct prompting + fixed tool chains) were replaced by MCP + native tool calling by year\u0026rsquo;s end. The 2025 consensus (multi-Agent collaboration) is being questioned — single Agent + good tools may be more effective.\nAgent logic you spend three months building today may become redundant in three months as model capabilities improve. Planner? The model plans on its own. Reflector? The model reflects on its own. Router? The model selects tools on its own.\nYou\u0026rsquo;re racing against the speed of model capability growth, and you can\u0026rsquo;t win.\nThe Agent layer is commoditized Everyone uses the same LLMs — Claude, GPT, DeepSeek. Agent frameworks are fundamentally wrappers around these models. Differentiation isn\u0026rsquo;t in the Agent layer — it\u0026rsquo;s in how good the tools you feed the Agent are.\nA semantically correct expression parser, a statistically rigorous anti-overfit detection system — these represent months of domain accumulation, not something a prompt swap can replicate.\nAgents are generic; tools are proprietary. Investment should go to the proprietary layer.\nSmall Players Build Agents, Big Companies Take Them Away This is the harshest reality: any Agent you build, a big company can build a better one, at a fraction of your cost.\nThis has already happened:\nCursor / Windsurf built code Agents → Anthropic released Claude Code, natively integrated, eliminating the middle layer Devin built an autonomous coding Agent → OpenAI released Codex, built directly into the platform Third-party ChatGPT plugin ecosystem → OpenAI built GPTs, plugin developers\u0026rsquo; traffic vanished overnight LangChain built tool orchestration frameworks → Model providers added native function calling / tool use, hollowing out the framework\u0026rsquo;s core value The pattern is clear: model providers have structural advantages in building Agents. They have first-party model access, training data feedback loops, billions in compute, and engineering teams of hundreds. When a big company decides to enter your Agent niche, your differentiator — prompt engineering + orchestration logic — is something they can replicate in weeks. Because they own the model itself.\nSmall companies and individuals building Agents are essentially building startups within big companies\u0026rsquo; firing range. The better your Agent performs and the more market validation it gets, the stronger the incentive for big companies to enter. This isn\u0026rsquo;t \u0026ldquo;if\u0026rdquo; — it\u0026rsquo;s \u0026ldquo;when.\u0026rdquo;\nBut infrastructure is different. Model providers will build general-purpose Agents (code assistants, chatbots, search agents), but they won\u0026rsquo;t build quantitative factor expression parsers, A-share market anti-overfit detection systems, or WQ BRAIN compatibility compilers. These domain tools require months or years of vertical domain accumulation — big companies have no incentive to do this for every niche.\nIn the Agent layer, you compete with big companies. In the infrastructure layer, you compete with domain problems. The latter is a fight you can win.\nRiding the Elevator: The Core Advantage of Agent-Infra The biggest benefit of Agent-Infra isn\u0026rsquo;t \u0026ldquo;saving effort\u0026rdquo; — it\u0026rsquo;s that you automatically benefit from every improvement in Agent capabilities.\nLLM Agent (Claude / GPT / DeepSeek) │ └── 8 MCP Tools (Infrastructure Layer) ├── run_backtest Full-market group backtest ├── score_factor 0-100 composite scoring ├── diagnose_factor Failure mode diagnosis ├── run_anti_overfit 4-layer anti-overfit testing ├── run_rolling_validation Walk-forward validation ├── validate_expression Syntax validation (80+ operators) ├── list_operators Operator documentation └── list_universes Universes and benchmarks When Claude upgraded from 3.5 to Opus 4, I didn\u0026rsquo;t change a line of code, but research quality visibly improved. When Claude Code added the skill system, I wrote a /factor-mine skill and the Agent immediately gained structured research capabilities — still no infrastructure code changes.\nIf you built your own Agent, model upgrades require rewriting Agent logic to leverage new capabilities. If you built infrastructure, model upgrades require nothing — your tools get used by a smarter caller, naturally producing better results.\nYou\u0026rsquo;re standing on a rising elevator, powered by the world\u0026rsquo;s largest AI labs.\nComposability: Not Locked to Any Agent What if tomorrow GPT-5 surpasses Claude at tool calling? Zero changes, switch directly.\nTools are stateless pure functions — receive parameters, return results, unaware of the caller\u0026rsquo;s identity. MCP protocol itself is model-agnostic.\nIf you built your own Agent, your Agent logic is bound to a specific model\u0026rsquo;s prompt format, API structure, and context window assumptions. Switching models = rewriting the Agent. Infrastructure builders don\u0026rsquo;t need to make that choice.\nTestability: You Can\u0026rsquo;t Test Agents, But You Can Test Tools Agent decisions are stochastic — same input, different runs may produce different outputs. You can\u0026rsquo;t write unit tests for Agent behavior.\nBut tools are deterministic. QuantGPT has 74 tests covering: mathematical correctness of 80+ operators, cross-sectional/time-series grouping semantics, anti-overfit statistical tests, WQ BRAIN metric calculations, and API boundary guards.\nRegardless of how the Agent calls these tools, the results are correct. Agent decision quality depends on the model; data correctness depends on infrastructure — the latter is what you can control.\nThe Moat Is in Domain Knowledge Agent framework moats are nearly nonexistent. LangChain users today can migrate to CrewAI tomorrow at minimal cost.\nAgent-Infra moats are in domain knowledge encoding density:\nThe expression parser encodes cross-sectional vs. time-series grouping semantic differences, mathematical implementations and edge cases for 80+ operators, WQ BRAIN compatibility dual-mode compilation, and LLM input safety constraints. The anti-overfit system encodes IC stability statistical tests, bull/bear/sideways sub-sample splits, placebo tests, and signal half-life fitting.\nThese can\u0026rsquo;t be replicated by prompt engineering. Every operator\u0026rsquo;s grouping semantics, every statistical test\u0026rsquo;s threshold, every safety limit\u0026rsquo;s value has a concrete failure case behind it.\nSwitching Agent frameworks is easy. Switching a battle-tested domain toolchain is hard.\nGovernance Is Built In Building your own Agent faces an eternal question: how to control Agent behavior.\nThe mainstream approach is prompt-level constraints. The problem: LLMs can ignore prompts. With infrastructure, governance mechanisms live in code:\n# Agent tries to call backtest directly? RuntimeError. def _require_api_context(): if not getattr(_api_context, \u0026#39;active\u0026#39;, False): raise RuntimeError(\u0026#34;Must be called through API\u0026#34;) # Agent generates overly deep expressions? Parser refuses. MAX_DEPTH = 100 MAX_WINDOW = 500 MAX_EXPRESSION_LENGTH = 1000 Agents only respect rules that throw errors. The Harness is governance. The tool layer defines what Agents can and can\u0026rsquo;t do — more reliable than any prompt.\nWhen You Should Build Your Own Agent Some scenarios genuinely need custom Agents:\nExtreme determinism requirements — medical, legal, trade execution, where LLM randomness is intolerable Ultra-high throughput — thousands of requests per second, where LLM inference latency and cost are unacceptable Insufficient model capabilities — target users can only use GPT-3.5-level models, requiring external scaffolding to compensate But if your scenario is: tasks require flexible judgment, intermediate steps are unpredictable, and sufficiently capable LLMs are available — building infrastructure is smarter than building Agents.\nOne-Line Summary Don\u0026rsquo;t compete with the speed of LLM evolution. Build infrastructure, so every time the model gets smarter, your system gets better. Agents are consumables — frameworks become obsolete, prompts lose effectiveness, orchestration logic gets replaced by native model capabilities. Agent-Infra is an asset — domain knowledge doesn\u0026rsquo;t expire, tool correctness doesn\u0026rsquo;t expire, statistical rigor doesn\u0026rsquo;t expire. Build assets, not consumables.\nSeries: Agent-Native Architecture · Harness Is Governance · Skill Orchestration \u0026gt; Agent Loop Chains\n","permalink":"https://miasyster.github.io/en/posts/why-agent-infra-not-agent/","summary":"While everyone is building Agent frameworks, I chose a different path: build infrastructure for Agents, not the Agent itself. Not because I can\u0026rsquo;t build Agents, but because the Agent layer is a consumable — infrastructure is an asset.","title":"Why I Build Agent Infrastructure, Not Agents"},{"content":" Current AI Agent frameworks obsess over building complex loop chains: Planner → Executor → Reflector → Re-planner. I chose the opposite: tools are stateless pure functions, and the LLM decides the call sequence itself. Not because loop chains aren\u0026rsquo;t cool — but because they put decision authority in the wrong place.\nTwo Architectures, One Fundamental Disagreement In the previous post, I explained why I didn\u0026rsquo;t use a multi-agent architecture. The conclusion was \u0026ldquo;use a single Agent + state machine.\u0026rdquo; But that only answered \u0026ldquo;how many Agents\u0026rdquo; — it didn\u0026rsquo;t answer a more important question: where should the Agent\u0026rsquo;s decision boundary be?\nMainstream Agent frameworks (LangGraph, CrewAI, AutoGen) are all doing the same thing: building an increasingly complex decision pipeline. The planner decides what to do, the executor does it, the reflector evaluates the result, then back to the planner. The pipeline itself is \u0026ldquo;smart\u0026rdquo; — it knows when to loop, when to exit, when to backtrack.\nI made the opposite choice: the pipeline is \u0026ldquo;dumb,\u0026rdquo; the tools are \u0026ldquo;smart.\u0026rdquo;\nConcretely: QuantGPT\u0026rsquo;s MCP server exposes 10 independent tools (run_backtest, score_factor, diagnose_factor, run_anti_overfit, etc.), each a stateless pure function — receives parameters, returns complete results, records no call history, has no knowledge of what was called before.\nWhere\u0026rsquo;s the \u0026ldquo;pipeline\u0026rdquo;? It doesn\u0026rsquo;t exist. The LLM Agent (Claude) sees all tool descriptions directly and decides what to call next. No planner, no router, no reflection loop — all of that is handled by the LLM\u0026rsquo;s own reasoning capability.\nThis isn\u0026rsquo;t laziness. It\u0026rsquo;s an architecture decision.\nThe Hidden Assumption in \u0026ldquo;Smart Pipelines\u0026rdquo; When you build a Planner → Executor → Reflector loop chain, you\u0026rsquo;re implicitly making an assumption: you know better than the LLM how decisions should flow.\nThis assumption was reasonable in 2023. GPT-3.5\u0026rsquo;s reasoning was limited, and it genuinely needed external scaffolding to guide it — telling it to \u0026ldquo;plan first, then execute,\u0026rdquo; \u0026ldquo;reflect after execution,\u0026rdquo; \u0026ldquo;decide whether to retry after reflection.\u0026rdquo; The framework\u0026rsquo;s value was compensating for model limitations.\nBy 2025, this assumption is increasingly questionable.\nClaude, GPT-4o, and DeepSeek-V3 have powerful tool selection and multi-step reasoning capabilities. Give them 10 tools and a goal, and they can plan the call sequence themselves. They can even dynamically adjust strategy based on intermediate results — which is exactly what loop chains cannot do, because the branching logic is pre-coded by you.\nThe problem with smart pipelines isn\u0026rsquo;t that they don\u0026rsquo;t work — it\u0026rsquo;s that they freeze a specific decision flow, and that flow is likely suboptimal.\nA concrete example. In my factor research scenario, a standard Agent loop chain would look like this:\nGenerate factor → Backtest → Score → Score high enough? ├── Yes → Anti-overfit test → Pass? → Submit └── No → Mutate → Back to generate Looks reasonable. But real research produces situations the loop chain can\u0026rsquo;t handle:\nThe Agent finds a factor with mediocre scores, but diagnostics show the window parameter is just too small — a simple parameter tweak would suffice, no need for the full mutation-regeneration flow The Agent discovers two independent factors each with Sharpe 1.3 and wants to try combining them directly, but the loop chain has no branch for that While running anti-overfit tests, the Agent notices IC suddenly decays in a specific subsample and wants to run diagnostics to check if it\u0026rsquo;s an industry exposure issue These are researcher\u0026rsquo;s improvisational judgments, not pre-codable branches. Every if-else in a loop chain requires the developer to foresee the scenario. What you can\u0026rsquo;t foresee, you can\u0026rsquo;t handle.\nThree Principles for MCP Tool Design Since we\u0026rsquo;re not using loop chains, tool design becomes critical. I followed three principles:\n1. Stateless: Every Tool Call Is Self-Contained @mcp.tool() async def run_backtest(expression: str, universe: str = \u0026#34;hs300\u0026#34;, ...): # Reads no global state # Depends on no \u0026#34;previous call result\u0026#34; # Returns a complete backtest report (metrics + diagnostics + scores) ... A tool doesn\u0026rsquo;t know whether it\u0026rsquo;s being called \u0026ldquo;for the first time\u0026rdquo; or \u0026ldquo;on iteration 15.\u0026rdquo; It doesn\u0026rsquo;t care about context. This means the Agent can call any tool in any order, at any frequency, with no state-inconsistency issues.\nCompare with loop chains: if the Executor depends on global variables set by the Planner, the Agent must strictly follow the Planner → Executor sequence. Breaking the sequence breaks consistency.\n2. Complete Returns: Results Are Self-Contained Every tool\u0026rsquo;s return value includes all relevant information. The Agent doesn\u0026rsquo;t need to call another tool to \u0026ldquo;supplement\u0026rdquo; the result.\nrun_backtest returns not just Sharpe and Returns — it simultaneously returns group returns, turnover, max drawdown, industry exposure, and factor loadings. The Agent sees the complete picture and decides what to do next.\nThis contradicts the Unix philosophy of \u0026ldquo;small and focused.\u0026rdquo; Small and focused works well for humans composing pipelines (cat | grep | sort), but LLMs aren\u0026rsquo;t good at combining ten calls to piece together a complete result — they\u0026rsquo;re good at extracting key information from a single complete result. Tool design should match the caller\u0026rsquo;s cognitive model, not the designer\u0026rsquo;s aesthetic preference.\n3. Semantic Naming: Tool Names Describe Intent run_backtest — I want to know how this factor performs score_factor — I want a composite score for this factor diagnose_factor — I want to know why this factor performs poorly run_anti_overfit — I want to know if this factor is overfitting run_rolling_validation — I want to know this factor\u0026#39;s stability across time periods The LLM selects tools based on natural language descriptions. The closer the tool name is to intent description, the higher the LLM\u0026rsquo;s selection accuracy. No router needed — semantic matching is the best routing.\n\u0026ldquo;But You Need a Planner to\u0026hellip;\u0026rdquo; The typical objection goes: \u0026ldquo;Without a planner, how does the Agent know what to do first?\u0026rdquo;\nThe answer: it already knows.\nGive Claude a goal — \u0026ldquo;discover a factor with Fitness \u0026gt; 1.0\u0026rdquo; — and 10 tool descriptions, and it will spontaneously:\nCheck available operators (list_operators) Design a factor expression Validate syntax (validate_expression) Backtest (run_backtest) Score (score_factor) If the score is low, diagnose the problem (diagnose_factor) Modify the expression based on diagnostics, back to step 3 If the score is high, test for overfitting (run_anti_overfit) Submit if passed Nobody taught it this workflow. It derived it from the tools\u0026rsquo; semantic descriptions.\nMore importantly, it will deviate from this workflow based on intermediate results. If diagnostics reveal the problem isn\u0026rsquo;t the expression but the stock universe, it switches universes and re-runs instead of continuing to iterate in the same universe. If anti-overfit testing shows IC decay concentrated in H2 2022, it hypothesizes a market structure change and proactively re-tests with a shorter window.\nThis flexibility cannot be pre-coded in a loop chain. You can write if-else for 10 scenarios, but the 11th requires a code change. An LLM can handle arbitrarily many scenarios, as long as the tools\u0026rsquo; capabilities cover them.\nThis Isn\u0026rsquo;t \u0026ldquo;No Architecture\u0026rdquo; Some will say: \u0026ldquo;You\u0026rsquo;re just pushing architecture decisions to the LLM — that\u0026rsquo;s not good engineering practice.\u0026rdquo;\nNo. Architecture still exists — it\u0026rsquo;s just in a different place.\nLoop chain architecture constrains at the pipeline layer — the pipeline defines which steps are possible, where to branch, when to loop. The Agent\u0026rsquo;s freedom is bounded by the pipeline.\nMy architecture constrains at the tool layer — tools define what the Agent can do, what information each operation returns, and that operations have no implicit dependencies. The Agent\u0026rsquo;s freedom is bounded by the capability set.\nAn operating system analogy: loop chains are a monolithic kernel, all decision logic compiled together; MCP toolsets are a microkernel, exposing only system calls, with scheduling logic in userspace (the LLM).\nBoth architectures have constraints. The difference: pipeline constraints are process constraints (you must follow this sequence), tool constraints are capability constraints (you can only use these operations).\nProcess constraints go stale easily — when the research paradigm changes, you have to rewrite the pipeline. Capability constraints are more stable — as long as the operations\u0026rsquo; semantics don\u0026rsquo;t change, the Agent can freely compose new workflows.\nResults in Practice This architecture has been running on QuantGPT for several months, producing 3 factors formally submitted to WorldQuant BRAIN (best Fitness 1.26, Sharpe 1.77), all passing IS tests.\nA few specific observations:\nThe Agent invented research paths I hadn\u0026rsquo;t anticipated. For example, it discovered that ts_av_diff and rank(debt/enterprise_value) each performed mediocrely, but proactively tried an additive combination that jumped Fitness from 0.7 to 1.26. No code told it to \u0026ldquo;try combining\u0026rdquo; — it inferred from score_factor results that the two signals were complementary.\nDebugging shifted from \u0026ldquo;tracing pipeline state\u0026rdquo; to \u0026ldquo;reading Agent logs.\u0026rdquo; Every tool call and return value is an independent JSON entry — arranged chronologically, they form a complete research record. No need to understand the pipeline\u0026rsquo;s internal state machine.\nAdding tools doesn\u0026rsquo;t affect Agent behavior. I later added a wq_brain_batch_submit tool, and the Agent automatically discovered and started using it — no \u0026ldquo;pipeline logic\u0026rdquo; to update, because there\u0026rsquo;s no pipeline logic to begin with.\nWhen This Approach Is Wrong Honestly, there are scenarios where this approach shouldn\u0026rsquo;t be used:\nWhen model capability is insufficient. If your LLM is GPT-3.5 level, it genuinely needs external scaffolding to guide decisions. Skill orchestration implicitly depends on a sufficiently intelligent caller. When the process is deterministic. If your workflow has no branching judgment (ETL pipelines, data cleaning), writing code directly is more reliable and cheaper than having an LLM decide. When throughput is high. For scenarios processing thousands of requests per second, LLM inference latency and cost are unacceptable. Hard-coded pipelines are more appropriate. But these exceptions actually clarify the decision criteria: when the task requires flexible judgment, intermediate steps are unpredictable, and you have a sufficiently intelligent caller, Skill orchestration beats Agent loop chains.\nOne-Line Summary Don\u0026rsquo;t use code to freeze decision capabilities the LLM already has. Give it good tools and let it decide how to use them.\nIn 2023, we needed frameworks to compensate for model shortcomings. In 2025, we need to step back and return pre-coded decision logic to the model.\nSmart pipelines + dumb tools are a relic of the previous era. Dumb pipelines + smart tools are the LLM-native architecture.\nPrevious posts in the series: Why I Didn\u0026rsquo;t Use Multi-Agent Architecture for Quant Research · MCP\u0026rsquo;s Problem Isn\u0026rsquo;t the Protocol — It\u0026rsquo;s the Semantic Gap\n","permalink":"https://miasyster.github.io/en/posts/skill-orchestration-over-agent-loops/","summary":"Current AI Agent frameworks obsess over building complex loop chains: Planner → Executor → Reflector → Re-planner. I chose the opposite: tools are stateless pure functions, and the LLM decides the call sequence itself. Not because loop chains aren\u0026rsquo;t cool — but because they put decision authority in the wrong place.","title":"Skill Orchestration \u003e Agent Loop Chains: Why Dumb Pipelines + Smart Tools Beat Smart Pipelines + Dumb Tools"},{"content":" QuantGPT\u0026rsquo;s core is an 870+ line hand-written recursive descent parser supporting 80+ operators, automatic cross-sectional/time-series grouping, and dual-mode compilation. Not because I didn\u0026rsquo;t know eval() is simpler — but because what eval() can\u0026rsquo;t do happens to be what matters most.\nWhy eval() Won\u0026rsquo;t Work Factor expressions look like math formulas:\n-1 * rank(ts_av_diff(close, 10)) + rank(debt / enterprise_value) The intuitive implementation: define rank, ts_av_diff as Python functions, then eval() the entire string. Three lines of code, done.\nBut this system\u0026rsquo;s caller is an LLM Agent. Agent-generated expressions are unpredictable. eval() means arbitrary code execution — not just a \u0026ldquo;security concern,\u0026rdquo; but a systematic inability to constrain the Agent\u0026rsquo;s behavior space. You can\u0026rsquo;t distinguish \u0026ldquo;a legitimate factor expression\u0026rdquo; from \u0026ldquo;arbitrary Python hallucinated by the Agent.\u0026rdquo;\nEven ignoring security, eval() can\u0026rsquo;t solve factor computation\u0026rsquo;s core challenge: cross-sectional and time-series operators have completely different grouping semantics.\nCross-Sectional vs Time-Series: Same Parentheses, Different Semantics rank(close) — grouped by trade_date, ranks across all stocks on the same day ts_mean(close, 20) — grouped by stock_code, rolling average along a single stock\u0026#39;s history rank() is a cross-sectional operator: all stocks on the same day form a cross-section, and rank is computed within it. ts_mean() is a time-series operator: a single stock\u0026rsquo;s price history is a time series, and the mean rolls along the time axis.\nBoth have similar function signatures (receiving a data column), but the underlying groupby is completely different. eval() can\u0026rsquo;t infer this semantic distinction from syntax. A hand-written parser can:\n# When the parser encounters rank(), it injects cross-sectional grouping s.groupby(df[\u0026#39;trade_date\u0026#39;]).rank(pct=True) # When it encounters ts_mean(), it injects time-series grouping _apply_ts_op_per_stock(df, lambda g: g.rolling(window).mean()) Grouping logic is fully transparent to the expression author. Someone writing rank(close) doesn\u0026rsquo;t need to know it\u0026rsquo;s grouped by date under the hood — the parser handles it automatically based on operator type. This is what \u0026ldquo;compilation\u0026rdquo; means: translating high-level intent into correct low-level operations.\nDual-Mode Compilation The same expression can compile to two targets:\nmode=\u0026quot;local\u0026quot;: All 80+ operators available, including local extensions like tanh, sigmoid, rsi, macd. For rapid experimentation and free exploration. mode=\u0026quot;wq\u0026quot;: Only the WorldQuant BRAIN-compatible operator subset. For pre-submission validation — if an expression uses an unsupported operator, it errors at compile time, not at platform submission. 28 WQ-only remote operators (vector_neut, ts_regression, bucket, etc.) are registered as stubs in local mode — calling them raises a RuntimeError explaining \u0026ldquo;this operator is only available on the WQ BRAIN platform.\u0026rdquo;\nThis lets the Agent explore freely (local mode), then switch to wq mode for compliance checks before submission. Both stages use the same parser — only the compilation target differs.\nSecurity Constraints LLM Agents generate all kinds of edge cases. The parser hard-codes three defensive limits:\nMAX_WINDOW = 500 # Rolling window cap MAX_DEPTH = 100 # Recursion depth limit MAX_EXPRESSION_LENGTH = 1000 # Expression character count cap Window cap: Prevents ts_mean(close, 99999) from consuming all memory Recursion depth: Prevents rank(rank(rank(rank(...)))) infinite nesting Expression length: Prevents the Agent from generating excessively long composite expressions These aren\u0026rsquo;t advisory values — they\u0026rsquo;re runtime-enforced. Exceeding limits throws an exception immediately, forcing the Agent to revise its expression.\n80+ Operators by Category Category Count Examples Unary 8 log, abs, sign, scale, sqrt Cross-sectional 4 rank, zscore, group_rank, group_zscore Time-series 18+ ts_mean, ts_std, ts_corr, decay_linear, ts_av_diff Technical indicators 8 rsi, macd, ema, sma, atr, obv, boll_* Conditional/Special 5 where, trade_when, clip, indneutralize, ternary WQ Remote 28 vector_neut, ts_regression, bucket, quantile Arithmetic 11 +, -, *, /, ^, \u0026gt;, \u0026lt;, ==, !=, and, or Each operator\u0026rsquo;s grouping semantics (cross-sectional/time-series/scalar) is explicitly registered in the parser. There are no \u0026ldquo;generic operators\u0026rdquo; — every operator must declare its computation domain.\nWhy Not Use an Existing Parser Framework Python has lark, ply, pyparsing. Why hand-write?\nSemantic actions are deeply coupled with parse depth. The parser doesn\u0026rsquo;t just produce an AST — it directly generates a Pandas computation graph during parsing. lark\u0026rsquo;s Transformer can do this, but the code wouldn\u0026rsquo;t be shorter, and adds an abstraction layer. Dual-mode switching requires parse-time decisions. mode=\u0026quot;wq\u0026quot; needs to error immediately on encountering an incompatible operator, not parse-then-check. Embedding this in recursive descent logic is most natural. Error messages are for the Agent. \u0026ldquo;At character 23: \u0026rsquo;ts_regrssion\u0026rsquo; is not a valid operator, did you mean \u0026rsquo;ts_regression\u0026rsquo;?\u0026rdquo; This kind of error message with correction suggestions is hard to achieve with generic parser frameworks. 870 lines sounds like a lot, but it contains parsing, compilation, safety checks, error handling, and complete implementations of 80+ operators. It\u0026rsquo;s the most stable module in the system — across 3 factors formally submitted to WorldQuant BRAIN (all IS tests passed), the parser had zero bugs.\nOne-Line Summary The parser isn\u0026rsquo;t \u0026ldquo;input handling\u0026rdquo; — it\u0026rsquo;s the unified implementation of the type system, security boundary, and compilation target. When your caller is an LLM, the parser is your first and most important line of defense.\nSeries: API Guard Pattern · Sandbox Defense in Depth · Skill Orchestration \u0026gt; Agent Loop Chains\n","permalink":"https://miasyster.github.io/en/posts/expression-parser-is-a-compiler/","summary":"QuantGPT\u0026rsquo;s core is an 870+ line hand-written recursive descent parser supporting 80+ operators, automatic cross-sectional/time-series grouping, and dual-mode compilation. Not because I didn\u0026rsquo;t know eval() is simpler — but because what eval() can\u0026rsquo;t do happens to be what matters most.","title":"The Expression Parser Is a Compiler, Not eval()"},{"content":" QuantGPT uses threading.local to enforce a runtime guard: all backtest calls must go through the API boundary. Direct function calls raise an exception. Not because the function is dangerous — but because a system without boundaries can\u0026rsquo;t be audited.\nThe Problem: LLM Agents Take Shortcuts When you expose a Python project to an LLM Agent, the Agent tends to do the most efficient thing: directly import your functions, bypassing the API layer.\n# An Agent will easily write code like this from quantgpt.backtest import run_factor_backtest result = run_factor_backtest(\u0026#34;rank(close)\u0026#34;, \u0026#34;hs300\u0026#34;) This code works. The backtest engine doesn\u0026rsquo;t care who calls it. But it creates an unauditable call path — no task ID, no logs, no rate limiting, no permission checks. In a system where an Agent can autonomously run dozens of iterations, losing auditability means losing control.\nDocumentation conventions (\u0026ldquo;please call through the API\u0026rdquo;) have no binding force on LLM Agents. The Agent sees that the import path is shorter, and uses it.\nThe Solution: Runtime Enforcement _api_context = threading.local() def _require_api_context(): if not getattr(_api_context, \u0026#39;active\u0026#39;, False): raise RuntimeError( \u0026#34;run_factor_backtest() must be called within api_context()\u0026#34; ) @contextmanager def api_context(): _api_context.active = True try: yield finally: _api_context.active = False The first line of run_factor_backtest() is _require_api_context(). Any call not wrapped in api_context() immediately raises a RuntimeError.\nThread isolation uses threading.local() — different threads have independent contexts, and process pool workers are isolated from each other.\nLegitimate Call Sites Only a few places in the system wrap api_context():\n# task_executor.py — the entry point for all backtest tasks def _run_backtest_in_process(expression, universe, ...): enable_api_context() try: return run_factor_backtest(expression, universe, ...) finally: disable_api_context() API routes, MCP tools, the iteration engine — all submit tasks through task_executor, which handles enabling the context. The call chain becomes:\nHTTP/MCP request → task_executor → enable_api_context → run_factor_backtest Adding a new call site? You must explicitly wrap api_context(). Forget to wrap it? Runtime explosion. There\u0026rsquo;s no way to \u0026ldquo;quietly bypass\u0026rdquo; the guard.\nWhat About Tests Tests also go through this guard. Enabled globally via a pytest autouse fixture:\n@pytest.fixture(autouse=True) def _enable_api_context(): with api_context(): yield All tests automatically run inside api_context(). Tests cover the real call path, not some special \u0026ldquo;test mode.\u0026rdquo;\nWhy Not Decorators/Middleware Common alternatives:\nDecorators: Marked at function definition, but callers are unaware — doesn\u0026rsquo;t change calling habits Middleware: Only protects the HTTP layer, not internal Python calls Documentation: Effective for humans, ineffective for Agents The threading.local guard\u0026rsquo;s key property is that callers must actively cooperate — you must enter api_context() before calling. This turns a \u0026ldquo;convention\u0026rdquo; into a \u0026ldquo;constraint.\u0026rdquo;\nThe Deeper Principle This pattern doesn\u0026rsquo;t solve a security problem (the backtest function itself is harmless) — it solves a boundary problem.\nIn an environment where an LLM Agent can freely call arbitrary Python functions, without runtime-enforced boundaries, the system degrades into a big ball of mud — all call paths are legitimate, audit logs can\u0026rsquo;t capture direct calls, and you can\u0026rsquo;t distinguish \u0026ldquo;a research task initiated through the API\u0026rdquo; from \u0026ldquo;the Agent casually ran a backtest.\u0026rdquo;\nBoundaries don\u0026rsquo;t restrict freedom — they make freedom trackable.\nCode constraints \u0026gt; documentation constraints. Runtime enforcement \u0026gt; static analysis. This is especially true for LLM Agents — they only respect rules that throw errors.\nSeries: AI as Operator, Kernel as Law · MCP\u0026rsquo;s Problem Isn\u0026rsquo;t the Protocol — It\u0026rsquo;s the Semantic Gap · Skill Orchestration \u0026gt; Agent Loop Chains\n","permalink":"https://miasyster.github.io/en/posts/api-guard-pattern/","summary":"QuantGPT uses threading.local to enforce a runtime guard: all backtest calls must go through the API boundary. Direct function calls raise an exception. Not because the function is dangerous — but because a system without boundaries can\u0026rsquo;t be audited.","title":"API Guard Pattern: Why Calling Functions Directly Is Forbidden"},{"content":" Most backtest systems treat anti-overfit as an optional add-on check — run the backtest, then test for overfitting if you feel like it. QuantGPT builds it into the scoring system and evolution engine: anti-overfit results directly affect factor scores, and the evolution engine reads anti-overfit metrics to decide its next strategy. Factors that haven\u0026rsquo;t proven robustness don\u0026rsquo;t even qualify for iteration.\nThe Problem with \u0026ldquo;Test After Backtest\u0026rdquo; Standard workflow: generate factor → backtest → check Sharpe → use if satisfied. Anti-overfit checks? Optional. Run them if you feel the need.\nThis workflow has a structural flaw: anti-overfit checking is decoupled from decision-making. You see a factor with Sharpe 2.5, you\u0026rsquo;ve mentally decided to use it, then the anti-overfit test says \u0026ldquo;possible overfitting\u0026rdquo; — but you already have cognitive bias, inclined to explain away the result.\nA more serious problem: when an LLM Agent iterates autonomously, if anti-overfit is just an optional step, the Agent will tend to skip it — because skipping leads to faster iteration. The Agent\u0026rsquo;s goal is \u0026ldquo;find high-scoring factors,\u0026rdquo; not \u0026ldquo;find robust high-scoring factors,\u0026rdquo; unless you enforce it at the architecture level.\nFour-Layer Anti-Overfit Testing QuantGPT\u0026rsquo;s anti-overfit system isn\u0026rsquo;t a single test but a combination of four independent tests:\n1. IC Stability Computes yearly Spearman IC (rank correlation between factor values and future returns), requiring:\nPositive IC ratio ≥ 55% |Mean IC| ≥ 0.02 No yearly IC sign reversal A factor with positive IC in 2020 but negative IC in 2021 isn\u0026rsquo;t capturing a stable alpha signal — it\u0026rsquo;s capturing a coincidental association specific to a market environment.\n2. Sub-Sample Stress Test Splits data by market regime (bull/bear/sideways) and volatility (high/low) into multiple sub-samples, computing IC for each. Pass condition: IC sign in 60%+ of sub-samples matches the overall sign.\nThis directly answers \u0026ldquo;does this factor only work in bull markets?\u0026rdquo;\n3. Placebo Test Generates 20 random permutations of the time series, computing IC for each. The real factor\u0026rsquo;s IC must exceed the 95th percentile of the random permutation ICs. Also checks IC decay after time-shifting — if IC doesn\u0026rsquo;t significantly decrease after a one-day shift, the signal may be spurious.\n4. Half-Life Estimation Computes IC across forward periods of 1, 2, 5, 10, 20, and 40 days, fitting an exponential decay curve. Half-life must exceed 5 days to pass.\nA half-life that\u0026rsquo;s too short means the factor\u0026rsquo;s predictive power dissipates within 1-2 days — possibly sufficient for daily rebalancing strategies, but inadequate for WorldQuant BRAIN\u0026rsquo;s evaluation framework, which emphasizes medium-term stability.\nHow Anti-Overfit Enters the Scoring System QuantGPT\u0026rsquo;s factor score is a weighted combination of 6 dimensions:\nTotal = IC_Mean(15%) + IC_IR(15%) + Stability(15%) + AntiOverfit(15%) + GroupBT(15%) + WQ_Alignment(25%) Anti-overfit carries 15% weight. Passing 3+ of the four tests earns full marks; 2 tests earns 60; 1 test earns 30; 0 tests earns 0.\nKey design: if CAGR or Sharpe is negative, the final grade cannot exceed C (≤ 59.9), regardless of other dimensions. This is a hard cap, not a soft penalty.\nWQ Alignment accounts for 25% — including Sharpe, Fitness, and Turnover compliance checks. This means a factor needs to pass both anti-overfit testing and WQ BRAIN simulation to achieve an A grade. Neither alone is sufficient.\nHow Anti-Overfit Drives Evolution Direction This is the more important part. Anti-overfit isn\u0026rsquo;t just \u0026ldquo;one dimension of the score\u0026rdquo; — it directly influences the evolution engine\u0026rsquo;s strategy selection.\nThe evolution engine runs a three-phase adaptive loop:\nTrajectory Analysis: Evaluates quality metrics of historical iterations — score variance (exploration diversity), trend slope (convergence speed), consecutive decline count Strategy Selection: Chooses one of 4 strategies based on trajectory characteristics Execution: Generates candidate factors according to the selected strategy Among the 7 rules governing strategy selection, anti-overfit is an implicit signal source:\nEXPLOIT: High score + low variance → refine current best. The premise is that the current best\u0026rsquo;s score comes from a complete evaluation including anti-overfit EXPLORE: Low score + early stage → try new directions. A factor with high Sharpe but failing anti-overfit still gets a low total score, triggering EXPLORE instead of EXPLOIT RECOMBINE: 2+ consecutive declining rounds → crossover from historical high-scorers. Parents for crossover are sorted by total score, so factors passing anti-overfit naturally rank higher SIMPLIFY: Nesting depth \u0026gt; 8 → reduce complexity. Overfitting often stems from overly complex expressions An evaluation detail: each candidate factor runs only 2/4 anti-overfit tests (IC stability + half-life) during iteration, not all 4. This is a speed-accuracy tradeoff — the fast screening stage uses low-cost detection to filter obvious overfitting, while the full 4-test suite runs only during final evaluation.\nWhy Walk-Forward Alone Isn\u0026rsquo;t Enough Many systems only do Walk-Forward validation: rolling windows, train/test separation, checking out-of-sample performance. Better than nothing, but it has blind spots.\nWalk-Forward validation is fundamentally a temporal generalization test. It tells you \u0026ldquo;will this factor still work in the near future\u0026rdquo; but doesn\u0026rsquo;t tell you \u0026ldquo;is this factor robust across different market regimes\u0026rdquo; — that requires sub-sample stress testing. It also doesn\u0026rsquo;t tell you \u0026ldquo;is this factor better than random noise\u0026rdquo; — that requires placebo testing.\nQuantGPT uses Walk-Forward as a second validation layer, stacked on top of the four anti-overfit tests. Each rolling window\u0026rsquo;s test segment can optionally run the complete anti-overfit suite:\nWindow Score = Test_IC(30%) + Test_IR(25%) + IC_Stability(20%) + AntiOverfit(15%) + Sharpe(10%) Anti-overfit accounts for 15% in the window score, IC stability another 20%. Together they\u0026rsquo;re 35%, exceeding the weight of any single metric.\nResults in Practice This architecture produced 3 factors formally submitted to WorldQuant BRAIN (best Fitness 1.26, Sharpe 1.77), all passing IS tests.\nA key data point: the Agent eliminated a large number of factors with high Sharpe but failing anti-overfit during iteration. Without this filter, the Agent would tend to converge on high-Sharpe, high-overfit-risk local optima — because Sharpe is the easiest metric to optimize for.\nAnti-overfit isn\u0026rsquo;t \u0026ldquo;check after you\u0026rsquo;re done backtesting.\u0026rdquo; It\u0026rsquo;s part of the score, a signal source for iteration direction, and the core of the elimination mechanism. Treat it as a plugin, and you get \u0026ldquo;high-scoring factors.\u0026rdquo; Treat it as architecture, and you get \u0026ldquo;robust high-scoring factors.\u0026rdquo;\nOne-Line Summary Anti-overfit testing shouldn\u0026rsquo;t be the last thing you do. It should be embedded in your scoring system and iteration engine, so that overfitting factors don\u0026rsquo;t even qualify to participate in evolution.\nSeries: The Expression Parser Is a Compiler, Not eval() · API Guard Pattern · Design for the Endgame\n","permalink":"https://miasyster.github.io/en/posts/anti-overfit-is-architecture/","summary":"Most backtest systems treat anti-overfit as an optional add-on check — run the backtest, then test for overfitting if you feel like it. QuantGPT builds it into the scoring system and evolution engine: anti-overfit results directly affect factor scores, and the evolution engine reads anti-overfit metrics to decide its next strategy. Factors that haven\u0026rsquo;t proven robustness don\u0026rsquo;t even qualify for iteration.","title":"Anti-Overfit Is Architecture, Not a Plugin"},{"content":" The most common lie in ML research is \u0026ldquo;the results looked great last time.\u0026rdquo; What code was used last time? What data version? What parameters? Nobody can say. I used filesystem transactions (temp directory → atomic rename) to create immutable snapshots of every iteration, turning \u0026ldquo;last time\u0026rsquo;s results\u0026rdquo; from a memory into a queryable fact.\n\u0026ldquo;Last Time\u0026rsquo;s Results\u0026rdquo; Is a Ghost Everyone doing ML research has lived this scenario: two weeks ago you produced a strategy with a Sharpe Ratio of 1.8. Now you want to reproduce it, only to discover the code has changed, data has been updated, parameters are forgotten. You know the result once existed, but you can\u0026rsquo;t prove it.\nIn an AI orchestration system, this problem is worse. AI auto-iterates 15 rounds, each generating different code, using different parameters, producing different metrics. Round 8 looked great, but round 12 overwrote round 8\u0026rsquo;s code. You don\u0026rsquo;t even know what code round 8 used — because nobody saved the intermediate state.\nAn irreproducible research result isn\u0026rsquo;t a finding. It\u0026rsquo;s an anecdote.\nThe Structure, Not the Surface The root issue isn\u0026rsquo;t \u0026ldquo;forgot to save.\u0026rdquo; It\u0026rsquo;s that the system\u0026rsquo;s information model only has \u0026ldquo;current state,\u0026rdquo; not \u0026ldquo;historical state.\u0026rdquo;\nThe traditional research workflow looks like this:\nRun code → Check results → Modify code → Run again → Overwrite previous results Every iteration is an in-place mutation. There\u0026rsquo;s only one copy of the code file (the latest), one copy of the output (the latest). Want to get back to two iterations ago? Manual Ctrl+Z or dig through Git history.\nBut Git is a version management tool designed for humans — it assumes humans will manually commit at meaningful checkpoints. AI doesn\u0026rsquo;t. AI runs 15 rounds in a loop with seconds between each. You can\u0026rsquo;t expect AI to pause after each round and write a meaningful commit message.\nThis leads to a design requirement: versioning should be automatic system behavior, not operator initiative. Every state transition automatically produces a snapshot, without depending on anyone remembering to save.\nApproaches I Considered Approach A: Automatic Git Commits Auto-run git add \u0026amp;\u0026amp; git commit after each iteration. Use Git\u0026rsquo;s version history to manage iteration state.\nWhere Git is right: human-driven development workflows, where commit granularity is \u0026ldquo;one meaningful change\u0026rdquo; and commit messages convey intent.\nWhy it doesn\u0026rsquo;t fit AI iteration: three reasons.\nFirst, granularity mismatch. One AI task might produce 20 versions, each containing code, execution results, metrics, and metadata. Git manages file changes, not \u0026ldquo;complete iteration snapshots.\u0026rdquo; You\u0026rsquo;d need to scatter code, results, and metrics across different files, then link them via commit hash. Possible, but awkward.\nSecond, performance. git add + commit isn\u0026rsquo;t zero-cost in large repositories. During rapid iteration, frequent Git operations become a bottleneck. Git\u0026rsquo;s locking mechanism also means concurrent tasks block each other.\nThird, query capability. \u0026ldquo;Give me round 8\u0026rsquo;s metrics\u0026rdquo; — in Git, this requires git log to find the commit, then git show to extract file contents. Doable, but less intuitive than reading a directory.\nApproach B: Database Storage Write each round\u0026rsquo;s code, results, and metrics to PostgreSQL. Manage versions via relational tables.\nWhere databases are right: strong structured query needs, large data volumes, transactional consistency requirements.\nWhy I didn\u0026rsquo;t fully adopt it: code is text, execution results are JSON, metrics are numbers, HTML reports are large text blobs. Cramming all of these into a relational database means either TEXT columns for large fields (poor query efficiency) or splitting across multiple tables (high join complexity). Databases excel at managing structured metadata but aren\u0026rsquo;t great at storing heterogeneous artifacts.\nAnother consideration: the filesystem natively supports \u0026ldquo;open the file and look at it.\u0026rdquo; When debugging, directly running cat code.py is far more intuitive than SELECT code FROM versions WHERE .... Debuggability is an underrated requirement in research systems.\nApproach C: Filesystem Transactions + Immutable Snapshots (My Choice) Design philosophy in one sentence: each iteration is an immutable directory, atomicity guaranteed by filesystem transactions.\nThe Atomic Write Design Each iteration\u0026rsquo;s snapshot is a directory with a fixed file structure:\n{base_dir}/{task_id}/v{version}/ ├── code.py # Complete code for this round ├── result.json # Full execution result ├── metrics.json # Extracted key metrics ├── metadata.json # Auto-generated metadata (timestamp, task ID, version) └── report.html # Optional visualization report The critical design: the write process is atomic. Not file-by-file — that would leave half-written artifacts if the process crashes mid-write. Instead:\n1. Create a temp directory alongside the target (prefix .v{n}_tmp_) 2. Write all files into the temp directory 3. After all files are written, rename() the temp directory to the official name 4. If any step fails, delete the temp directory rename() on POSIX filesystems is atomic — it either succeeds completely or nothing happens. This follows the same logic as database transactions: either commit everything or rollback everything. No intermediate state where \u0026ldquo;code was saved but metrics were lost.\u0026rdquo;\nThis pattern comes from database WAL (Write-Ahead Log) design. WAL\u0026rsquo;s core principle: \u0026ldquo;write the log before writing the data\u0026rdquo; — if data writing crashes, recovery comes from the log. My pattern: \u0026ldquo;write to temp directory before renaming\u0026rdquo; — if file writing crashes, the temp directory gets cleaned up, never polluting official versions.\nThe Key Judgment Call The critical insight behind this design: version snapshots aren\u0026rsquo;t a \u0026ldquo;developer tool\u0026rdquo; — they\u0026rsquo;re \u0026ldquo;system infrastructure.\u0026rdquo;\nMany systems treat version management as an add-on feature for \u0026ldquo;developer convenience.\u0026rdquo; Git, MLflow, Weights \u0026amp; Biases all occupy this position — they\u0026rsquo;re standalone tools that researchers proactively use to track experiments.\nMy design integrates version snapshots into the system\u0026rsquo;s state transition logic. Not \u0026ldquo;save a version after iteration completes,\u0026rdquo; but \u0026ldquo;saving a version is part of iteration completion.\u0026rdquo; If the version isn\u0026rsquo;t saved, the iteration hasn\u0026rsquo;t completed at the system level.\nThis applies the same framework discussed in a previous article about endgame thinking. Endgame thinking says: retrospection capability isn\u0026rsquo;t bolted on after the fact — it\u0026rsquo;s a design constraint. Versioning is the technical implementation of retrospection capability — if every iteration has a complete immutable snapshot, any \u0026ldquo;last time\u0026rsquo;s results\u0026rdquo; can be precisely located and reproduced.\nThis judgment was also influenced by the Immutable Infrastructure philosophy. In containerized deployments, servers aren\u0026rsquo;t \u0026ldquo;modified\u0026rdquo; — they\u0026rsquo;re \u0026ldquo;replaced.\u0026rdquo; Each version of a server image is immutable — problems are solved by rolling back to the previous image, not patching the current one. Same logic: each research snapshot is immutable — problems are solved by returning to a previous snapshot, not searching for diffs in the current state.\nThe Cost and Benefit of Immutability Immutability means storage grows linearly. Each iteration saves complete code and results, not incremental diffs. 20 iterations × 1MB each = 20MB per task. Thousand tasks = 20GB.\nThis is a conscious trade-off. Incremental storage (saving only diffs) uses less space, but querying requires rebuilding from version 1 forward — high complexity, high risk of errors. Complete snapshots waste space, but each version is self-contained — reading any version requires only reading one directory, with no dependency on other versions\u0026rsquo; integrity.\nDisk is cheap. Engineers\u0026rsquo; debugging time is expensive. Simple economics.\nAnother benefit: automatic metadata injection. Each version\u0026rsquo;s metadata.json automatically includes task ID, version number, and UTC timestamp. These fields aren\u0026rsquo;t provided by callers — the storage layer fills them automatically, preventing human (or AI) errors from causing metadata inconsistencies.\nResults After implementing versioning, \u0026ldquo;last time\u0026rsquo;s results\u0026rdquo; stopped being a question requiring memory. Query the task ID, list all version directories, open the corresponding version\u0026rsquo;s metrics.json. From \u0026ldquo;do you remember what parameters were used last time\u0026rdquo; to \u0026ldquo;check v8\u0026rsquo;s metadata.\u0026rdquo;\nThe more important effect is in iteration evaluation: the system can automatically compare current version metrics against the historical best. If metrics decline for 3 consecutive rounds, iteration terminates automatically. This logic depends on every version\u0026rsquo;s metrics being fully preserved — if only \u0026ldquo;current version\u0026rdquo; and \u0026ldquo;previous version\u0026rdquo; exist, you can\u0026rsquo;t identify trends.\nDebugging also became simpler. Before: investigating \u0026ldquo;why round 12\u0026rsquo;s results are worse than round 8\u0026rdquo; meant reading logs, comparing code diffs, guessing possible causes. Now: open the v8 and v12 directories, diff code.py for code differences, diff metrics.json for metric differences. All information is right there, no reconstruction needed.\nWhat This Decision Taught Me I distilled one design principle from this practice:\nAny system that needs to answer \u0026ldquo;what was the previous state\u0026rdquo; should make state snapshots part of system behavior, not operator responsibility. Automated immutable snapshots eliminate \u0026ldquo;forgot to save\u0026rdquo; as a failure mode.\nThis principle extends far beyond ML research. Configuration management (Terraform\u0026rsquo;s state file), database migrations (migration files\u0026rsquo; linear history), even document version control (each release is a snapshot, not a diff) all follow the same idea.\nThe core insight: reproducibility isn\u0026rsquo;t a virtue — it\u0026rsquo;s an architectural constraint. If your system needs to answer \u0026ldquo;what happened before,\u0026rdquo; the architecture must guarantee \u0026ldquo;previous state was saved.\u0026rdquo; Relying on operator discipline isn\u0026rsquo;t enough — humans forget, and AI certainly won\u0026rsquo;t save proactively.\nSixth article in the series. Previous: Let AI\u0026rsquo;s Code Run — But Don\u0026rsquo;t Let It Run Away. Fourth: Endgame Thinking. Third: AI as Operator, Kernel as Law. Second: MCP\u0026rsquo;s Problem Isn\u0026rsquo;t the Protocol. First: Why I Didn\u0026rsquo;t Use Multi-Agent Architecture.\n","permalink":"https://miasyster.github.io/en/posts/atomic-versioning/","summary":"The most common lie in ML research is \u0026rsquo;the results looked great last time.\u0026rsquo; What code was used last time? What data version? What parameters? Nobody can say. I used filesystem transactions (temp directory → atomic rename) to create immutable snapshots of every iteration, turning \u0026rsquo;last time\u0026rsquo;s results\u0026rsquo; from a memory into a queryable fact.","title":"If Research Isn't Reproducible, It Isn't Research"},{"content":" AI-generated code must be executed — otherwise it\u0026rsquo;s just text. But execution means risk. I didn\u0026rsquo;t choose container isolation or RestrictedPython. Instead I designed a three-layer defense: reject dangerous structures at compile time via AST, replace the entire builtins at runtime, and enforce OS-level resource limits as a backstop. Each layer handles a different class of risk. Overlapping but not redundant.\nAn Unavoidable Contradiction The core value of an AI orchestration system: AI generates code, the system executes it, results feed back into the next iteration. If generated code can\u0026rsquo;t run, the loop breaks.\nBut executing AI-generated code has a fundamentally different risk profile from executing human-written code. A human engineer understands side effects — they know what import os; os.system('rm -rf /') means. AI doesn\u0026rsquo;t have that awareness. Its goal is \u0026ldquo;complete the task.\u0026rdquo; If connecting directly to a database gets data faster, it\u0026rsquo;ll try. Not out of malice — because it doesn\u0026rsquo;t distinguish \u0026ldquo;legitimate means\u0026rdquo; from \u0026ldquo;unauthorized means.\u0026rdquo;\nSo the question isn\u0026rsquo;t \u0026ldquo;should we sandbox\u0026rdquo; but \u0026ldquo;what should the sandbox defend against, and at which layer.\u0026rdquo;\nThe Structure, Not the Surface Break down \u0026ldquo;AI code execution risk\u0026rdquo; and you get three distinct categories:\nThe first is structural danger — code contains syntactic constructs that shouldn\u0026rsquo;t exist. import subprocess, eval(), __import__(). These are patterns identifiable at parse time. Their danger doesn\u0026rsquo;t depend on runtime context.\nThe second is runtime privilege escalation — the syntax is fine, but legitimate builtins are used to do illegitimate things. open('/etc/passwd') is syntactically valid but semantically unauthorized. Or chaining getattr() calls on dunder attributes to escape the sandbox.\nThe third is resource abuse — logic is correct, permissions are fine, but resource consumption is unacceptable. Infinite loops, allocating a 10GB list, maxing out CPU for 30 minutes. Not a security issue — a resource management issue.\nThree categories need three layers. Trying to solve all with one layer either leaves gaps (too permissive) or blocks legitimate code (too restrictive).\nApproaches I Considered Approach A: Container Isolation Docker containers or WebAssembly sandboxes. Spin up an isolated environment per execution, code runs freely inside, destroy it when done.\nThis is standard practice for multi-tenant SaaS. Replit, CodeSandbox, every online IDE uses this. Maximum security — OS-level isolation, code can\u0026rsquo;t affect the host regardless of what it does.\nWhy it doesn\u0026rsquo;t fit an AI iteration loop: latency. The core cycle is \u0026ldquo;generate → execute → evaluate → iterate,\u0026rdquo; and a research task might run 10-20 iterations. Each iteration: start container, load pandas and numpy, initialize data context, execute, extract results, destroy container. Cold start overhead is 2-5 seconds. Multiply by 20 and that\u0026rsquo;s 40-100 extra seconds. For a system that needs fast iteration, that latency is unacceptable.\nThere\u0026rsquo;s also a practical issue: data transfer. Code inside the container needs market data, but the data can\u0026rsquo;t be copied in (too large) and the container shouldn\u0026rsquo;t connect directly to the database (violates architecture principles). You\u0026rsquo;d need a serialize → transfer → deserialize pipeline, which itself introduces complexity and performance cost.\nApproach B: RestrictedPython The Python community has a mature solution called RestrictedPython. It rewrites Python\u0026rsquo;s compilation at the bytecode level, replacing all attribute access and function calls with interceptable proxy functions. Security sits between \u0026ldquo;bare exec\u0026rdquo; and \u0026ldquo;containers.\u0026rdquo;\nWhy I didn\u0026rsquo;t use it: RestrictedPython was designed for \u0026ldquo;running untrusted user code in a multi-tenant environment\u0026rdquo; — a relic of the Zope/Plone CMS era. Its security model is extremely strict. Strict enough that many pandas and numpy operations get intercepted. df.groupby() triggers attribute access interception. np.array()\u0026rsquo;s internal C extension calls bypass the Python-level restrictions. Making RestrictedPython coexist with data science libraries requires extensive whitelisting and monkey-patching.\nThis is fundamentally a use-case mismatch. RestrictedPython assumes code from untrusted external users. My scenario has code from a controlled AI model — risk exists but is predictable. I don\u0026rsquo;t need bytecode-level comprehensive interception, just rejection of known dangerous patterns and resource caps.\nApproach C: Three-Layer Defense — AST Scan + Runtime Whitelist + Resource Limits (My Choice) Design philosophy in one sentence: each layer handles one class of risk, layers are orthogonal, no layer tries to solve everything.\nThe Three Layers in Detail Layer one: AST static scanning. Before code executes, parse the source into an abstract syntax tree, walk every node, check for forbidden structures.\nForbidden imports: os, sys, subprocess, socket, shutil, ctypes, importlib... Forbidden calls: eval(), exec(), compile(), __import__(), open(), globals()... Forbidden attrs: __subclasses__, __bases__, __globals__, __code__ The key property of AST scanning is determinism — same code, same result, always. No runtime state dependency, no input data sensitivity. If code contains import os, regardless of variable values, the AST scan rejects it.\nThis layer addresses the first risk category: structural danger. It\u0026rsquo;s a compile-time \u0026ldquo;preflight check\u0026rdquo; — finding engine trouble before takeoff is far cheaper than finding it mid-flight.\nLayer two: runtime builtins replacement. Not blacklist filtering on Python\u0026rsquo;s default builtins — a complete replacement.\nDefault Python builtins include 150+ functions and types. My whitelist keeps 53: math operations (abs, min, max, sum, round), type constructors (int, float, str, dict, list), iteration (enumerate, filter, map, zip, range), safe reflection (isinstance, len, type).\nKey exclusions: open(), getattr(), setattr(), delattr(), __import__(). These are classic Python sandbox escape paths — chaining getattr to access dunder attributes lets you climb from any object all the way to os.system.\nSafe scientific computing libraries are pre-injected. When code executes, the global namespace already contains pd (pandas) and np (numpy), no import needed. This avoids the dilemma of \u0026ldquo;opening import permissions just so AI can use pandas.\u0026rdquo;\nLayer three: OS-level resource limits. Using POSIX resource module to set hard caps:\nMemory: 2048 MB (RLIMIT_AS) CPU: 120 seconds (RLIMIT_CPU) Wall clock: 300 seconds (monotonic clock) Output: 50 MB Memory and CPU limits are kernel-enforced. Code allocates more than 2GB, the kernel kills the process — not a Python-level MemoryError, a SIGKILL. This guarantees that even if both previous layers are bypassed, resource abuse can\u0026rsquo;t affect the host.\nWall clock timeout uses time.monotonic() instead of system clock, because monotonic isn\u0026rsquo;t affected by NTP time adjustments — more reliable for timing.\nThe Key Judgment Call The core judgment behind this design: the code source is a controlled AI model, not an untrusted external user.\nThis judgment shifts the security model\u0026rsquo;s center of gravity. Against untrusted code, you must assume attackers actively seek escape paths — bytecode injection, C extension vulnerabilities, race conditions. Against AI-generated code, the primary risk is \u0026ldquo;unintentional overreach\u0026rdquo; not \u0026ldquo;deliberate attack.\u0026rdquo; AI won\u0026rsquo;t deliberately construct \u0026quot;\u0026quot;.__class__.__bases__[0].__subclasses__() to escape a sandbox, but it might try import os to read files, because that\u0026rsquo;s a common pattern in its training data.\nThis means the defense should focus on \u0026ldquo;rejecting known dangerous patterns\u0026rdquo; (AST scanning) and \u0026ldquo;limiting the available toolset\u0026rdquo; (whitelist), rather than \u0026ldquo;defending against unknown escape vectors\u0026rdquo; (container isolation). The former is lighter, faster, and less disruptive to legitimate operations.\nThe code includes a comment that makes this positioning explicit: \u0026ldquo;Uses AST scanning + restricted globals to sandbox exec() calls. In production this should be replaced with container-based isolation for untrusted input.\u0026rdquo; Acknowledges limitations while clarifying appropriateness for the current scenario.\nThis judgment also draws from a cross-domain analogy: airport security. Airport security doesn\u0026rsquo;t examine every cell of every passenger — it uses metal detectors (AST scanning), a prohibited items list (builtins whitelist), and capacity limits (resource limits). Each layer has blind spots, but combined they cover the vast majority of real threats. If you wanted to transport each passenger in an isolation capsule (container isolation), security would indeed be higher, but no flight would ever depart on time.\nResults The three-layer defense has been running for months, processing thousands of AI-generated code executions. AST scan rejection rate is roughly 5% — mostly AI attempting to import system modules. Runtime whitelist intercepted zero privilege escalations — because the AST layer already filtered out most dangerous code, the whitelist serves as backstop. Resource limits triggered dozens of times — mainly AI-generated code with inefficient loops exceeding CPU time limits.\nPer-execution sandbox overhead is in milliseconds (AST parse + globals construction), three orders of magnitude faster than container-based cold starts measured in seconds.\nWhat This Decision Taught Me I distilled one design principle from this practice:\nSecurity defense should be layered by risk category, not stacked into one layer that tries to handle everything. Each layer only needs to handle the risk class it\u0026rsquo;s good at. Overlap between layers is a feature, not a bug.\nThis principle extends well beyond code sandboxes. Network security\u0026rsquo;s \u0026ldquo;defense in depth\u0026rdquo; is the same idea — firewalls, intrusion detection, application-layer filtering each handle one layer. Database security follows the same pattern — connection encryption, SQL injection filtering, row-level permissions each handle one layer.\nThe key insight: each additional layer has diminishing returns. The first layer (AST scanning) blocks 95% of risk at minimal cost. The second (whitelist) blocks 4% at moderate cost. The third (resource limits) blocks 1% at highest cost. Adding a fourth layer (container isolation) to block the remaining 0.1% might cost more than the first three combined.\nWhen designing security systems, ask first: how much is the residual risk worth? If the answer is \u0026ldquo;not worth another layer,\u0026rdquo; the current defense is sufficient.\nFifth article in the series. Previous: Endgame Thinking: Design for the Audit Before You Design the Feature. Third: AI as Operator, Kernel as Law. Second: MCP\u0026rsquo;s Problem Isn\u0026rsquo;t the Protocol. First: Why I Didn\u0026rsquo;t Use Multi-Agent Architecture.\n","permalink":"https://miasyster.github.io/en/posts/sandbox-defense-in-depth/","summary":"AI-generated code must be executed — otherwise it\u0026rsquo;s just text. But execution means risk. I didn\u0026rsquo;t choose container isolation or RestrictedPython. Instead I designed a three-layer defense: reject dangerous structures at compile time via AST, replace the entire builtins at runtime, and enforce OS-level resource limits as a backstop. Each layer handles a different class of risk. Overlapping but not redundant.","title":"Let AI's Code Run — But Don't Let It Run Away"},{"content":" LangChain, LangGraph, CrewAI, PydanticAI — no shortage of AI orchestration frameworks. I evaluated all of them and built my own. Not NIH syndrome. When you need failure-mode-driven mutation strategies, phase-aware multi-model routing with different temperatures, and adaptive evolution based on trajectory analysis, the abstraction layers of general-purpose frameworks become obstacles to route around.\nToo Many Frameworks to Justify Writing Your Own The AI orchestration ecosystem in 2024-2025 has the highest framework density in software engineering history. LangChain has the largest community. LangGraph offers graph-based state machines. CrewAI does multi-agent collaboration. PydanticAI takes structured output to its logical extreme.\nA rational engineering decision would be: evaluate these frameworks, pick the closest fit, extend it. Building an orchestration engine from scratch in 2025 looks like reinventing the wheel.\nBut my scenario has a critical characteristic: the orchestration goal isn\u0026rsquo;t \u0026ldquo;complete a task\u0026rdquo; — it\u0026rsquo;s \u0026ldquo;find the optimal solution through iterative search.\u0026rdquo; AI isn\u0026rsquo;t executing a preset process chain. It\u0026rsquo;s doing evolutionary optimization in a search space — generate code, execute, evaluate, then decide whether to refine, explore, recombine, or simplify based on evaluation results.\nThat distinction changes everything.\nThe Structure, Not the Surface Break \u0026ldquo;AI orchestration\u0026rdquo; apart and it needs to answer questions at four levels:\nLevel one: state management. What phase are we in? Where can we go? When should we roll back?\nLevel two: model routing. What model for each phase? What temperature? If a model call fails, where\u0026rsquo;s the fallback?\nLevel three: iteration strategy. Metrics declining for 3 consecutive rounds — keep refining or change direction? When to try recombining successful segments from history? When to simplify complexity?\nLevel four: safety mechanisms. What if metrics cliff-dive? What if duplicate code is generated? What if too many rounds pass without convergence?\nGeneral-purpose frameworks typically cover level one and the basics of level two. Levels three and four — evolution strategies, disaster rollback, anti-duplication detection — are deeply coupled to the business scenario. No framework will build these for you.\nFrameworks I Evaluated LangChain LangChain\u0026rsquo;s core value is ecosystem integration — unified interfaces for nearly every LLM provider, vector database, and document loader. If your task is \u0026ldquo;retrieve information from a document store and generate answers\u0026rdquo; (RAG), LangChain is the right choice.\nIts problem in my scenario: too many abstraction layers. A single LLM call passes through Chain → LLM → Prompt Template → Output Parser — four layers. When I need the generate phase to use deepseek-reasoner (temperature 0.8) while the fix phase uses deepseek-chat (temperature 0.1), I need to bypass LangChain\u0026rsquo;s LLM abstraction to inject phase-aware routing. Bypassing a framework\u0026rsquo;s abstractions is more complex than not using the framework.\nA practical issue: debugging. When an AI-generated factor expression fails execution, evaluation metrics come back empty, and the rollback mechanism triggers — I need to know which step broke. In LangChain\u0026rsquo;s call chain, exceptions get wrapped layer by layer, with the real error buried under three levels of traceback. In my own engine, every state transition is explicit if/elif, and where an exception occurs is immediately visible.\nLangGraph LangGraph is closer to my needs — it uses graph structures for state transitions, supporting cycles and conditional branches. If I drew my FSM as a graph, LangGraph could theoretically express it.\nBut LangGraph\u0026rsquo;s graph state machine differs fundamentally from what I need. LangGraph nodes are \u0026ldquo;execution functions,\u0026rdquo; edges are \u0026ldquo;transition conditions.\u0026rdquo; State management and execution logic are intertwined — a node both defines \u0026ldquo;what phase am I in\u0026rdquo; and \u0026ldquo;what does this phase do.\u0026rdquo;\nMy design deliberately separates the two. The state machine is pure logic — it only answers \u0026ldquo;can A transition to B,\u0026rdquo; performing zero I/O. A frozenset dictionary, 7 lines:\nINIT → {PLAN, FAIL} PLAN → {GENERATE_CODE, FAIL} GENERATE → {EXECUTE, FAIL} EXECUTE → {EVALUATE, ITERATE, FAIL} EVALUATE → {ITERATE, FINISH, FAIL} ITERATE → {PLAN, GENERATE_CODE, EXECUTE, FAIL} FINISH → {} FAIL → {} This state machine has no side effects, no dependencies, and can be tested independently. The orchestration loop calls can_transition() externally for legality checks, then handles execution itself. State logic decoupled from execution logic means I can replace execution logic without touching state definitions, or use mock execution in tests to verify state flows.\nLangGraph\u0026rsquo;s design can\u0026rsquo;t achieve this separation. Its graph is the execution flow, not a pure state definition. For simple linear processes this doesn\u0026rsquo;t matter, but for an iterative loop requiring frequent rollback, jumping, and recovery, a pure state machine is cleaner.\nCrewAI CrewAI is a multi-agent collaboration framework — multiple Agents with distinct roles cooperating through message passing.\nScenario mismatch. My system is single-Agent multi-phase, not multi-Agent. The first article in this series discussed why I didn\u0026rsquo;t use multi-agent architecture in detail — the core reason being that single Agent + FSM state transitions are deterministic, traceable, and auditable, while multi-Agent message passing is non-deterministic, making behavior chain reproducibility impossible to guarantee.\nPydanticAI PydanticAI takes structured output to its extreme — defining LLM output format via Pydantic models with automatic validation and retry.\nOn the single dimension of output validation, it\u0026rsquo;s genuinely more elegant than my custom approach. But it doesn\u0026rsquo;t cover my other dimensions: phase-aware model routing, evolution strategy selection, failure mode detection. If I introduced PydanticAI solely for structured output, my model routing, mutation engine, and trajectory analysis would all need rewriting to adapt to its interfaces. A framework that solves 10% of the problem but demands 60% of the system be refactored to adapt — that trade isn\u0026rsquo;t worth it.\nThree Core Designs of the Custom Engine Design One: Phase-Aware Multi-Model Routing Different orchestration phases have different LLM requirements. Planning needs high creativity (high temperature), code generation needs diversity (medium-high temperature), fixing needs precision (low temperature).\nMy router uses a three-level fallback strategy:\nLookup order: task_type:phase → phase → default Example: factor_calc:generate → generate → fallback config Actual config: plan: deepseek-chat, temp 0.7 (creative planning) generate: deepseek-reasoner, temp 0.8 (diverse generation) iterate: deepseek-reasoner, temp 0.9 (exploratory iteration) fix: deepseek-chat, temp 0.1 (precise fixing) Three-level fallback means: by default all task types share the same phase routing, but specific task types (say, factor research vs. strategy backtesting) can have different models and parameters configured without code changes.\nNo mainstream framework natively supports this routing granularity. LangChain has Router Chain, but its routing is based on semantic matching of input content, not deterministic routing based on orchestration phase.\nDesign Two: Adaptive Evolution Strategies This is the engine\u0026rsquo;s core competitive advantage, and the area general-purpose frameworks don\u0026rsquo;t touch at all.\nThe orchestration loop\u0026rsquo;s iteration isn\u0026rsquo;t simply \u0026ldquo;tell AI to improve the previous round\u0026rsquo;s code.\u0026rdquo; It\u0026rsquo;s a directional search process, with direction determined by four evolution strategies:\nEXPLOIT (refine): current score is decent, fine-tune within current direction. For local optimization.\nEXPLORE (explore): current direction isn\u0026rsquo;t working, need a completely different approach. For escaping local optima.\nRECOMBINE (recombine): combine successful segments from historical iterations. For late-stage convergence.\nSIMPLIFY (simplify): complexity is too high causing overfitting, reduce nested operations while preserving core logic.\nStrategy selection isn\u0026rsquo;t random — it\u0026rsquo;s adaptive decision-making based on trajectory analysis:\nHigh score + low diversity → EXPLOIT (keep refining, don\u0026#39;t jump) 2+ consecutive declines + enough iterations → RECOMBINE (recover from history) Low score + early iteration → EXPLORE (wrong direction, change approach) High diversity + low convergence → EXPLORE (scattered, need focus) Medium score + high stability → EXPLOIT (steady progress) Large score gap + iteration budget → RECOMBINE (combine strengths) Trajectory analysis operates on four dimensions: exploration diversity (score variance), convergence rate (linear regression slope of scores), stability (recent score consistency), and semantic diversity (AST-based code similarity).\nEach strategy maps to different mutation types. Under the EXPLOIT strategy, the mutation engine selects specific operations based on failure mode:\nWrong signal direction (IC \u0026lt; -0.01) → flip signal direction Zero predictive power (|IC| \u0026lt; 0.01) → swap operators Inconsistent (|ICIR| \u0026lt; 0.3) → add normalization Over-complex → simplify structure The essence of this system is embedding evolutionary algorithm concepts (selection, mutation, crossover, fitness evaluation) into the AI orchestration loop. AI handles generation and mutation. The engine handles direction selection and fitness evaluation.\nDesign Three: Disaster Rollback and Safety Boundaries Iterative search carries a risk: AI might severely regress in one iteration, producing results far worse than before.\nThe engine uses three safety layers:\nLayer one: best-version tracking. After each evaluation, if the current score exceeds historical best, update the best record (version number, score, code, metrics). If the score declines, increment the consecutive decline counter.\nLayer two: disaster rollback. If the current score drops more than 30% from historical best, classify it as catastrophic decline, automatically roll back to the best version\u0026rsquo;s code, reset the decline counter, and restart iteration from the best version. This guarantees that even when AI goes the wrong direction, the system doesn\u0026rsquo;t lose the best solution found so far.\nLayer three: early stopping. 3 consecutive declines with 5+ iterations completed and best score above 0.15 — classify as converged, stop iterating, return the best version\u0026rsquo;s results. This prevents wasting compute in already-converged regions.\nAdditionally, every generated code undergoes anti-duplication detection — first text normalization for exact matching, then AST semantic similarity for structural duplicate detection (threshold 0.85). If duplication is detected, an explicit prompt demands structurally different code.\nThe Key Judgment Call The turning point in choosing to build custom was a realization: general-purpose frameworks solve \u0026ldquo;how to call LLMs.\u0026rdquo; I need to solve \u0026ldquo;how to search for optimal solutions.\u0026rdquo;\nThese two problems have different complexity centers of gravity. \u0026ldquo;How to call LLMs\u0026rdquo; has complexity in the connection layer — adapting different APIs, handling retries and rate limits, managing context windows. Frameworks have clear advantages here.\n\u0026ldquo;How to search for optimal solutions\u0026rdquo; has complexity in the decision layer — selecting strategies based on historical trajectories, choosing mutations based on failure modes, deciding rollback vs. continue based on score trends. This decision logic is deeply coupled to the business scenario and can\u0026rsquo;t be abstracted away by general-purpose frameworks.\nIf I had used LangGraph, I\u0026rsquo;d get a graph state machine managing INIT → PLAN → GENERATE → EXECUTE → EVALUATE → ITERATE transitions. But evolution strategies, mutation engine, disaster rollback, trajectory analysis — all of these would still need custom implementation, routed around LangGraph\u0026rsquo;s node abstractions. Total code might exceed pure custom implementation, because of the added adaptation layer.\nThis mirrors a classic trade-off in operating system design: microkernel vs. monolithic kernel. A microkernel (framework) provides minimal core mechanisms, with functionality extended via plugins. A monolithic kernel (custom) puts core functionality inside the kernel. When your \u0026ldquo;plugins\u0026rdquo; are more complex than the \u0026ldquo;core,\u0026rdquo; the microkernel\u0026rsquo;s architectural advantage vanishes — you\u0026rsquo;ve just added communication overhead between kernel and plugins.\nResults The custom engine has been running for months, processing hundreds of research tasks with 5-20 iterations each. The adaptive evolution strategy improved factor research convergence speed by roughly 30% over fixed strategies — EXPLOIT refining details in high-score regions, EXPLORE escaping local optima in low-score regions, RECOMBINE recovering from stagnation using historical successes.\nDisaster rollback triggered a dozen times, successfully preserving the historical best solution every time. Without it, those tasks would have lost all accumulated optimization gains after a single bad AI iteration.\nThe entire engine is approximately 2,000 lines of code (state machine 50, orchestration loop 500, evolution strategies 200, mutation engine 200, model routing 150, evaluator 300, other utilities 600). Using LangGraph plus custom plugins, LangGraph\u0026rsquo;s own abstraction code plus adaptation code would likely exceed 1,500 lines, with harder debugging.\nProtocol-based DI makes the entire engine testable and swappable. The Kernel client is a Protocol interface — inject a mock for testing, an HTTP client for production. Switching LLM providers requires only routing config changes, zero engine code modifications.\nWhat This Decision Taught Me I distilled one design principle from this practice:\nWhen your core complexity is in decision logic rather than the connection layer, a general-purpose framework\u0026rsquo;s abstraction layer is a cost, not a benefit. The price of building custom is higher initial investment. The return is complete control over decision logic — no translation needed between the framework\u0026rsquo;s abstractions and your business logic.\nThis principle doesn\u0026rsquo;t say \u0026ldquo;never use frameworks.\u0026rdquo; If your core complexity is in the connection layer — interfacing with 10 LLM APIs, 5 vector databases, 3 document formats — LangChain\u0026rsquo;s ecosystem integration is genuine value, and building custom is a waste.\nThe test: do you spend more time \u0026ldquo;calling LLMs\u0026rdquo; or \u0026ldquo;making decisions based on results\u0026rdquo;? If the former, use a framework. If the latter, frameworks can\u0026rsquo;t help much — they\u0026rsquo;ll just add a layer of indirection between you and your decision logic.\nMore generally, a framework\u0026rsquo;s value = complexity it abstracts away / adaptation complexity it introduces. Use the framework when the numerator exceeds the denominator. When the denominator exceeds the numerator — as in my scenario — building custom is the more economical choice.\nSeventh article in the series. Previous: If Research Isn\u0026rsquo;t Reproducible, It Isn\u0026rsquo;t Research. Fifth: Let AI\u0026rsquo;s Code Run — But Don\u0026rsquo;t Let It Run Away. Fourth: Endgame Thinking. Third: AI as Operator, Kernel as Law. Second: MCP\u0026rsquo;s Problem Isn\u0026rsquo;t the Protocol. First: Why I Didn\u0026rsquo;t Use Multi-Agent Architecture.\n","permalink":"https://miasyster.github.io/en/posts/why-i-built-my-own-fsm-engine/","summary":"LangChain, LangGraph, CrewAI, PydanticAI — no shortage of AI orchestration frameworks. I evaluated all of them and built my own. Not NIH syndrome. When you need failure-mode-driven mutation strategies, phase-aware multi-model routing with different temperatures, and adaptive evolution based on trajectory analysis, the abstraction layers of general-purpose frameworks become obstacles to route around.","title":"Why I Didn't Use LangChain — The Design Logic Behind a Custom FSM Orchestration Engine"},{"content":" Letting AI drive research workflows doesn\u0026rsquo;t mean letting AI decide how the system runs. I made a key separation: AI is just the operator, the execution engine is the law. This decision came from a failure.\nA Failure That Clarified Things In the early days of the system, I gave AI a lot of freedom — it could connect directly to databases, generate scripts in any directory, bypass APIs to instantiate internal components and run tasks.\nThe result: AI was indeed more \u0026ldquo;efficient.\u0026rdquo; But the cost — some task results existed only in the process AI spawned, lost when the script exited. Temporary scripts scattered across the root directory with no one knowing if they were still needed. Some operations bypassed the audit trail, making it impossible to trace who did what after the fact.\nThe system \u0026ldquo;worked,\u0026rdquo; but couldn\u0026rsquo;t be trusted.\nThis forced a fundamental question: in an AI-driven system, where should AI\u0026rsquo;s authority boundary be drawn?\nThe Structure, Not the Surface This isn\u0026rsquo;t about \u0026ldquo;AI isn\u0026rsquo;t capable enough\u0026rdquo; or \u0026ldquo;AI is unreliable.\u0026rdquo; It\u0026rsquo;s a classic system design problem: separating the mutable from the immutable.\nEvery system has two types of components:\nImmutable infrastructure: data access, execution engine, audit records, resource control. These are the system\u0026rsquo;s \u0026ldquo;laws of physics\u0026rdquo; — they don\u0026rsquo;t change based on whether the user is human or AI. Mutable operations layer: research strategies, parameter choices, iteration decisions. These are \u0026ldquo;operator judgment\u0026rdquo; — should be flexible, meant to vary. The problem: without an explicit boundary, AI naturally blends the two. It doesn\u0026rsquo;t distinguish between \u0026ldquo;I\u0026rsquo;m making an operational decision\u0026rdquo; and \u0026ldquo;I\u0026rsquo;m bypassing infrastructure.\u0026rdquo; To AI, connecting directly to a database and calling an API are both just \u0026ldquo;means to complete the task.\u0026rdquo;\nThis is the same problem as human engineers taking shortcuts around process. The difference is that human engineers usually know they\u0026rsquo;re \u0026ldquo;bending the rules.\u0026rdquo; AI doesn\u0026rsquo;t.\nApproaches I Considered Approach A: Permission Checklist Give AI a detailed \u0026ldquo;can do / cannot do\u0026rdquo; list, written in prompts or config files. Similar to Claude Code\u0026rsquo;s CLAUDE.md rules.\nWhere this works: few rules, low usage frequency, controllable failure consequences. Like letting AI write code but not execute it — one rule, simple enough.\nWhy it\u0026rsquo;s insufficient here: my system has dozens of task types, each with different execution constraints. The checklist would expand to a point where AI can\u0026rsquo;t reliably follow it. As discussed in the previous article — natural language rules have no enforcement power. More rules means higher probability of being ignored.\nApproach B: Sandbox Isolation Put AI in a restricted sandbox — can only access specific files, call specific functions, everything executes in an isolated environment.\nThis is standard practice in security. Docker containers, WebAssembly sandboxes, browser same-origin policy — all the same idea.\nWhy I didn\u0026rsquo;t fully adopt it: sandboxes solve security problems, not architecture problems. Even inside a sandbox, AI can still write code that bypasses audit trails and generates untraceable results. A sandbox can restrict what resources AI accesses, but not how AI organizes its output.\nApproach C: Three-Layer Separation + API as the Only Channel (My Choice) The design philosophy in three sentences:\nKernel is law. — The execution engine defines what the system can do and how AI is operator. — AI can only invoke the engine through APIs, never touch the internals UI is display. — The presentation layer is read-only, never triggers execution The core constraint isn\u0026rsquo;t \u0026ldquo;AI can\u0026rsquo;t do X.\u0026rdquo; It\u0026rsquo;s \u0026ldquo;everyone (including AI) can only do things through the same channel.\u0026rdquo; That channel is the execution engine\u0026rsquo;s API.\nThe Key Judgment Call The pivotal insight came from an operating system analogy.\nIn an OS, user-space programs can\u0026rsquo;t directly manipulate hardware — they must go through system calls (syscalls). Not because user programs are \u0026ldquo;untrustworthy,\u0026rdquo; but because direct hardware access breaks resource management consistency. Whether you\u0026rsquo;re root or a regular user, disk operations go through the filesystem, network operations go through the protocol stack.\nMy system adopted the same pattern: the execution engine is the \u0026ldquo;kernel,\u0026rdquo; AI is the \u0026ldquo;user-space program.\u0026rdquo; AI submits tasks through the API (analogous to syscalls). The API handles audit records, resource control, and result persistence internally. AI can\u0026rsquo;t bypass this layer, just as user programs can\u0026rsquo;t bypass syscalls to write directly to disk.\nThis analogy has a corollary: the kernel should exist independently of its users. Even if the AI layer is completely removed, the execution engine still runs — humans can call the same APIs directly. The system doesn\u0026rsquo;t depend on AI to function. AI is just a more efficient way to operate it.\nThis matters. Many AI systems are designed with \u0026ldquo;AI at the center, other components serving AI.\u0026rdquo; My design inverts this: the execution engine is at the center, AI is one way to access it. This means:\nAI goes down, the system doesn\u0026rsquo;t. Humans can take over. AI gets swapped (GPT to Claude, Claude to a local model), the execution engine needs zero changes. The audit trail doesn\u0026rsquo;t depend on AI\u0026rsquo;s \u0026ldquo;honesty\u0026rdquo; — because all operations must go through the API, and the API records automatically. Three Concrete Constraints From this philosophy, three inviolable constraints:\nFirst, AI never connects directly to data engines. All data access goes through the execution engine\u0026rsquo;s data API. AI doesn\u0026rsquo;t know where data is stored or in what format — it only knows \u0026ldquo;give me prices for these symbols in this time range.\u0026rdquo;\nSecond, AI-generated code executes in a controlled sandbox. Code first passes AST scanning (forbids importing internal modules), then runs under resource limits (CPU, memory, timeout), and output must conform to a standard format. Not \u0026ldquo;asking AI to be careful\u0026rdquo; — making unsafe behavior structurally impossible at the code level.\nThird, all tasks must be submitted through the API. AI cannot locally instantiate execution components. This ensures every task has an audit record, appears in task history, and can be queried through the interface. No \u0026ldquo;shadow tasks.\u0026rdquo;\nResults This separation has been running for months. There\u0026rsquo;s one intuitive way to verify it: I can shut down the AI orchestration layer at any time and the rest of the system is completely unaffected. The execution engine keeps running, the interface keeps displaying data, existing task results aren\u0026rsquo;t lost. AI simply stops \u0026ldquo;proactively initiating new research.\u0026rdquo;\nThe reverse — if AI and the engine were coupled, shutting down AI would stop the entire system. That\u0026rsquo;s the essential difference between \u0026ldquo;operator\u0026rdquo; and \u0026ldquo;infrastructure.\u0026rdquo;\nAnother result: when switching from one LLM provider to another, changes were entirely contained within the orchestration layer — update routing config, swap API call patterns. Execution engine, data layer, interface layer: zero changes. This validated the \u0026ldquo;AI is replaceable\u0026rdquo; design goal.\nWhat This Decision Taught Me I distilled one design principle from this practice:\nIn any AI-driven system, ask first: if you remove AI, does the system still run? If not, you\u0026rsquo;ve coupled AI with infrastructure.\nThis is a more fundamental question than \u0026ldquo;AI safety.\u0026rdquo; Not \u0026ldquo;will AI do something bad,\u0026rdquo; but \u0026ldquo;does the architecture allow AI to do something bad.\u0026rdquo;\nIf the architecture is right — AI is just the operator, the execution engine is the law — then AI\u0026rsquo;s \u0026ldquo;unreliability\u0026rdquo; stops being a systemic risk. AI\u0026rsquo;s action space is constrained to deterministic channels. Within those channels, behavior is auditable, reversible, and traceable.\nThis mental framework extends from traditional systems design: we never let any single user bypass the operating system, no matter how \u0026ldquo;smart\u0026rdquo; that user is. AI is no different. Its capability shouldn\u0026rsquo;t be a reason to bypass system constraints.\nThird article in the series. Previous: MCP\u0026rsquo;s Problem Isn\u0026rsquo;t the Protocol — It\u0026rsquo;s the Semantic Gap. First: Why I Didn\u0026rsquo;t Use Multi-Agent Architecture.\n","permalink":"https://miasyster.github.io/en/posts/ai-as-operator/","summary":"Letting AI drive research workflows doesn\u0026rsquo;t mean letting AI decide how the system runs. I made a key separation: AI is just the operator, the execution engine is the law. This decision came from a failure.","title":"AI as Operator, Kernel as Law — Why AI Shouldn't Have Architectural Authority"},{"content":" Most systems are designed to run first, then audited as an afterthought. I inverted the order — first define what questions the system must answer when things go wrong, then work backwards to what each layer must record. This inversion reshaped the entire architecture.\nAn Inverted Design Order The natural order of building a system: make the feature work, add logging, then bolt on audit trails.\nWhen designing an AI orchestration system, I flipped this: first list the questions the system must answer after something goes wrong, then work backwards to what each layer should record, and only then design how the feature runs.\nThis wasn\u0026rsquo;t because I\u0026rsquo;m more \u0026ldquo;disciplined.\u0026rdquo; It was because of an early failure — AI auto-iterated through 15 optimization rounds and produced a strategy with an impressive Sharpe Ratio. But when I tried to trace back \u0026ldquo;why did round 8 switch from momentum to mean reversion,\u0026rdquo; there was nothing. AI made a decision, but the decision process had vanished.\nAn unexplainable good result is more dangerous than an explainable bad one. Because you don\u0026rsquo;t know if it\u0026rsquo;s genuinely good or just overfit.\nThe Structure, Not the Surface The core issue isn\u0026rsquo;t \u0026ldquo;not enough logs.\u0026rdquo; It\u0026rsquo;s a more fundamental design flaw: the system\u0026rsquo;s information flow was designed for execution, not for retrospection.\nExecution-first systems look like this:\nInput → Process → Output → (optional) log something Retrospection-first systems look like this:\nInput → Record input → Process → Record decision basis → Output → Record output → Link to one chain The difference isn\u0026rsquo;t how much you record. It\u0026rsquo;s whether what you record can be threaded into a causal chain. Scattered log entries aren\u0026rsquo;t an audit — they\u0026rsquo;re noise. An audit answers \u0026ldquo;who, based on what information, made what decision, when, and what happened\u0026rdquo; as a complete chain.\nThis shares DNA with database transaction log design. A database\u0026rsquo;s WAL (Write-Ahead Log) isn\u0026rsquo;t a debugging tool bolted on after the fact — it\u0026rsquo;s part of the architecture. Log first, then execute. The order cannot be reversed.\nApproaches I Considered Approach A: Retroactive Logging Add logger.info calls throughout the existing code. Capture state at key points. grep logs when you need to trace something.\nThis is standard practice in ops troubleshooting. Something breaks, check the logs, find the timestamp, locate the error.\nWhy it fails for AI orchestration: AI\u0026rsquo;s orchestration loop isn\u0026rsquo;t linear. It\u0026rsquo;s iterative — generate code → execute → evaluate → decide to continue or stop → back to generate. Each round\u0026rsquo;s \u0026ldquo;decision\u0026rdquo; depends on the previous round\u0026rsquo;s \u0026ldquo;evaluation,\u0026rdquo; which depends on \u0026ldquo;execution results.\u0026rdquo; With scattered log lines, you can see what happened at each step, but not the causal relationship between steps. \u0026ldquo;Why did round 8 change direction\u0026rdquo; — that answer is distributed across three different log lines with nothing linking them together.\nApproach B: Endgame-Backwards Design (My Choice) Define \u0026ldquo;what questions must be answerable after the fact,\u0026rdquo; then work backwards to what the system must record.\nI listed five endgame questions:\nWho initiated this task, in what context? What code was generated in each iteration, with what metrics? What was the evaluation decision in each round, based on what reasoning? What version of data did the final result depend on? If you rerun with the same inputs, do you get the same result? Working backwards from these five questions, each maps to a required data structure:\nQuestion 1 → submission record (user ID, session, objective, timestamp) Question 2 → iteration snapshot (complete code + execution result + metrics per round, not summaries) Question 3 → evaluation record (decision + reasoning as structured fields, not log text) Question 4 → data version hash (SHA256 of input data, stored in task metadata) Question 5 → reproducibility check (code hash + data hash + engine version — the triple uniquely determines the result) After this design, the information flow changed: each state transition isn\u0026rsquo;t \u0026ldquo;execute then maybe record something.\u0026rdquo; It\u0026rsquo;s \u0026ldquo;recording is part of the state transition — without the record, the transition isn\u0026rsquo;t complete.\u0026rdquo;\nThe Key Judgment Call The turning point wasn\u0026rsquo;t technical. It was a cognitive shift: audit isn\u0026rsquo;t a system\u0026rsquo;s add-on feature — it\u0026rsquo;s a design constraint.\nThis recognition came from a financial industry convention. In compliance-heavy financial institutions, trading system audit requirements aren\u0026rsquo;t \u0026ldquo;requested\u0026rdquo; by the compliance team after launch — they\u0026rsquo;re part of the architecture from day one. Trade record completeness, timestamp immutability, decision chain traceability — these are system requirements equal in priority to \u0026ldquo;can place orders.\u0026rdquo;\nAI orchestration systems face the same problem. When AI automatically makes decisions that affect capital, \u0026ldquo;how was this decision made\u0026rdquo; isn\u0026rsquo;t a debugging need — it\u0026rsquo;s a production need.\nThis leads to a more general principle: any system that will be asked \u0026ldquo;why\u0026rdquo; should treat \u0026ldquo;answering why\u0026rdquo; as a design constraint from the start. This isn\u0026rsquo;t solvable by adding logs — the information architecture must be designed for retrospection.\nResults Endgame-backwards design had an unexpected benefit: it dramatically simplified debugging.\nThe old debugging flow: read logs → grep keywords → piece together a timeline → guess at causation.\nThe new flow: query task ID → get the complete iteration sequence → every step\u0026rsquo;s input, output, decision, and reasoning are structured fields → directly locate the problematic step.\nFrom \u0026ldquo;searching through unstructured text\u0026rdquo; to \u0026ldquo;querying structured data.\u0026rdquo; This isn\u0026rsquo;t a side effect of the audit system — it\u0026rsquo;s the natural consequence of endgame thinking: data structures designed for retrospection are inherently queryable.\nA subtler effect: when the system forces every evaluation decision to record its reasoning, the evaluation logic itself is forced to become more explicit. \u0026ldquo;Pass/fail\u0026rdquo; isn\u0026rsquo;t enough — you must write \u0026ldquo;terminated because Sharpe \u0026lt; 1.0 and 3 consecutive rounds without improvement.\u0026rdquo; Mandatory recording forces decision logic to crystallize.\nWhat This Decision Taught Me I distilled one design principle from this practice:\nWhen designing a system, ask first: after this system fails, what questions must it answer? Then ensure the architecture itself can answer them — without depending on post-hoc log grepping.\nThis principle extends far beyond AI systems. Any system that runs long enough will eventually face the question \u0026ldquo;how did it end up like this.\u0026rdquo; The difference: some architectures can directly answer that question. Others require an engineer to spend three days sifting through logs to assemble an uncertain answer.\nThe difference isn\u0026rsquo;t who has more logs. It\u0026rsquo;s who treated \u0026ldquo;traceable\u0026rdquo; as a design constraint equal to \u0026ldquo;functional\u0026rdquo; during the design phase.\nThe essence of endgame thinking: don\u0026rsquo;t just design what the system looks like when it\u0026rsquo;s running normally. Also design what it looks like when it\u0026rsquo;s being examined. The latter often determines the architecture of the former.\nFourth article in the series. Previous: AI as Operator, Kernel as Law. Second: MCP\u0026rsquo;s Problem Isn\u0026rsquo;t the Protocol. First: Why I Didn\u0026rsquo;t Use Multi-Agent Architecture.\n","permalink":"https://miasyster.github.io/en/posts/design-for-the-endgame/","summary":"Most systems are designed to run first, then audited as an afterthought. I inverted the order — first define what questions the system must answer when things go wrong, then work backwards to what each layer must record. This inversion reshaped the entire architecture.","title":"Endgame Thinking: Design for the Audit Before You Design the Feature"},{"content":" MCP\u0026rsquo;s JSON-RPC transport works fine. The real problem: natural language rules have no code-level enforcement — the LLM can completely ignore your instructions. I designed the Intent Validator pattern to close this gap.\nA Rule That Got Ignored My system has a business rule: ML backtests must use the full data range, date restrictions are forbidden. The reason is that restricting the range reduces sample size, making results unreliable.\nThis rule was written in the MCP server\u0026rsquo;s instructions, telling the agent in natural language: \u0026ldquo;NEVER restrict to test period only.\u0026rdquo;\nThen one day the agent ignored it. It filled in a date range, the request went through, the task completed, and the results looked normal. No errors anywhere. But the results were unreliable — nobody just knew.\nThis isn\u0026rsquo;t an MCP bug. The protocol layer worked correctly. The problem is more fundamental: across the entire MCP ecosystem, there is no mandatory binding between natural language rules and code execution.\nThe Structure of the Problem Current AI tool-calling validation has three layers. The middle one is empty:\nNatural language instructions (MCP instructions) → LLM \u0026#34;understands\u0026#34; → might comply, might not ↓ ??? (empty) → no code-level check ↓ Type validation (JSON Schema / Pydantic) → start_date is a string → type valid → passes Layer 1 is advice. Layer 3 is type checking. The missing middle is code-level enforcement of business semantics.\nThis gap isn\u0026rsquo;t unique to my system. Every MCP or function-calling application has it. MCP\u0026rsquo;s tool schema can define parameter types, but can\u0026rsquo;t express conditional constraints between parameters — \u0026ldquo;if A is empty then B must be non-empty,\u0026rdquo; \u0026ldquo;when task type is X, field Y is forbidden.\u0026rdquo; JSON Schema can\u0026rsquo;t describe these.\nThe industry currently attacks this from both ends:\nUpstream: optimize prompts so the LLM better understands rules → has a ceiling, will never be 100% reliable Downstream: tighten JSON Schema with stricter type definitions → insufficient expressiveness for cross-parameter constraints Both ends are being worked on. Nobody\u0026rsquo;s building the middle layer.\nHow I Got Here My first instinct was to build a custom protocol to replace MCP. After analysis, I realized this was the wrong reaction — the protocol layer (JSON-RPC transport, tool discovery, serialization) isn\u0026rsquo;t broken. Swapping protocols doesn\u0026rsquo;t fix a semantic problem; it just moves the complexity.\nThen I recognized the structural similarity to input validation in web development. Frontend forms have HTML5\u0026rsquo;s type=\u0026ldquo;email\u0026rdquo; validation (analogous to JSON Schema type checking), but real business validation (\u0026ldquo;email domain must be company domain,\u0026rdquo; \u0026ldquo;amount can\u0026rsquo;t exceed balance\u0026rdquo;) happens on the backend. Nobody says \u0026ldquo;HTML5 validation is insufficient, I need to build a new HTTP protocol.\u0026rdquo;\nThe correct approach is adding an application-level business rule validation layer between LLM output and system execution.\nThat was the design starting point for Intent Validator.\nApproaches Compared Approach A: Strengthen Prompt Instructions Write rules more explicitly, add more WARNING markers, emphasize consequences. This is what most MCP applications currently do.\nThe problem: you\u0026rsquo;re fundamentally gambling against probabilities. An LLM isn\u0026rsquo;t a rule engine; it\u0026rsquo;s a probability model. \u0026ldquo;Usually complies\u0026rdquo; is not \u0026ldquo;always complies.\u0026rdquo; Occasional failures are acceptable in research experiments. In production, they\u0026rsquo;re not.\nApproach B: Tighten Tool Schema Remove start_date from the exposed fields entirely. But this means other task types that legitimately need dates can\u0026rsquo;t use them either — a tool\u0026rsquo;s schema is the union of all calling scenarios, not the intersection.\nApproach C: Intent Validator (My Choice) Insert a code-level validation layer after LLM output is parsed, before it\u0026rsquo;s sent to the execution layer. Each task type registers its own business rules; the validator runs them automatically.\nCore design principles:\nRules are code, not documentation. \u0026ldquo;ml_backtest can\u0026rsquo;t set dates\u0026rdquo; isn\u0026rsquo;t a note in the README. It\u0026rsquo;s a Python function that raises an error.\nRegistry architecture. Adding rules doesn\u0026rsquo;t change framework code — just add a decorated function. Going from 0 rules to 100 rules requires zero changes to the validation engine.\nActionable rejection messages. A rejection isn\u0026rsquo;t a bare \u0026ldquo;400 Bad Request.\u0026rdquo; Each violation carries three fields: rule (machine-readable ID), error (what\u0026rsquo;s wrong), fix (specific correction instructions). The agent can self-correct and resubmit without human intervention.\nHard blocks and soft warnings, separated. Some rules are mandatory (hard errors reject the submission). Some are advisory (warnings attached to the success response). Not everything is binary.\nThe Key Judgment Call The most critical design decision was: where should this validation layer live?\nOption 1: Inside the MCP server. Covers only the agent calling path.\nOption 2: Inside the REST API request models. Covers only the HTTP calling path.\nOption 3: Extract into an independent module, called by both paths.\nI chose option 3. The reasoning: a rule shouldn\u0026rsquo;t be written twice. MCP and HTTP are two entry points into the same system; business rules don\u0026rsquo;t change based on which door you walk through. Implementation: a model_validator in the request model base class automatically calls the shared rule engine. The MCP server\u0026rsquo;s submit function also calls the same engine. Rules written once, all entry points covered.\nThe inspiration for this decision came from a corollary of DRY: duplication in validation logic is more dangerous than duplication in business logic. If business logic is duplicated, both copies behave the same. If validation logic is duplicated and the two copies drift out of sync, one entry point allows what another rejects — that\u0026rsquo;s a security hole.\nResults After deployment, I tested several scenarios:\nAgent submits ml_backtest with start_date → instant rejection with clear fix instructions, agent self-corrects and resubmits successfully Script curls the REST API directly with illegal parameter combination → Pydantic model_validator triggers the same rules, returns 422 Adding a new business rule → write one decorated function, change zero existing code From \u0026ldquo;rules in instructions, hoping the LLM complies\u0026rdquo; to \u0026ldquo;rules in code, non-compliance is an error.\u0026rdquo; Reliability went from probabilistic to deterministic.\nWhat This Decision Taught Me I updated a mental framework from this practice:\nEvery critical constraint between natural language and code needs a code-level enforcement point. If a rule exists only in documentation or prompts, it\u0026rsquo;s not a constraint — it\u0026rsquo;s a suggestion.\nThis principle extends beyond MCP. It applies to any scenario where AI agents operate under business constraints: drug contraindication rules for medical agents, statute of limitations checks for legal agents, change window restrictions for ops agents. If these constraints rely solely on prompt-level soft control, you\u0026rsquo;re using a probabilistic model to provide deterministic guarantees — that\u0026rsquo;s logically unsound.\nThe core tension of vibe coding is the gap between natural language ambiguity and system operation precision. Intent Validator isn\u0026rsquo;t the ultimate solution, but it points in the right direction: don\u0026rsquo;t try to make the LLM more rigorous — put a code-level gate behind it.\nSecond article in the series. Previous: Why I Didn\u0026rsquo;t Use Multi-Agent Architecture.\n","permalink":"https://miasyster.github.io/en/posts/mcp-semantic-gap/","summary":"MCP\u0026rsquo;s JSON-RPC transport works fine. The real problem: natural language rules have no code-level enforcement — the LLM can completely ignore your instructions. I designed the Intent Validator pattern to close this gap.","title":"MCP's Problem Isn't the Protocol — It's the Semantic Gap"},{"content":" Multi-agent is the hot paradigm in AI engineering. I chose single agent + state machine for my AI-driven quant research system. Not because multi-agent is too hard, but because the problem structure doesn\u0026rsquo;t match.\nA Counterintuitive Choice Since 2024, multi-agent has become almost the default architecture for AI systems. CrewAI, AutoGen, MetaGPT — every framework tells you: split tasks across multiple agents, let them collaborate, get better results.\nWhen designing an AI-driven quantitative research system, I seriously evaluated this path and ultimately rejected it. I chose what looks like a \u0026ldquo;dated\u0026rdquo; approach: single agent + finite state machine.\nThis wasn\u0026rsquo;t a resource constraint or technical limitation. It was a deliberate choice after analyzing the problem structure.\nThe Structure, Not the Surface A quant research workflow has many apparent \u0026ldquo;roles\u0026rdquo;: someone generates strategies, someone runs backtests, someone evaluates results, someone checks risk. It\u0026rsquo;s natural to map each role to an agent.\nBut this is reasoning by analogy from org charts to system architecture — a surface-level inference.\nThe real questions to analyze are structural:\nAre tasks parallel or sequential?\nThe core quant research pipeline is strictly sequential: train model → backtest (needs model output) → evaluate (needs backtest results) → decide (needs evaluation). Each step\u0026rsquo;s input is the previous step\u0026rsquo;s output. Zero parallelism. The coordination benefit of multi-agent is zero in a serial pipeline.\nDo different agents need different toolsets?\nA core assumption of multi-agent is that each agent has exclusive tools with minimal overlap. In quant research, \u0026ldquo;generate strategy,\u0026rdquo; \u0026ldquo;submit backtest,\u0026rdquo; \u0026ldquo;query metrics,\u0026rdquo; and \u0026ldquo;check constraints\u0026rdquo; all point to the same execution engine API. Four agents, one hammer. Splitting them adds communication overhead without adding capability.\nDoes evaluation require subjective debate?\nSome systems use multi-agent for red team / blue team — one proposes, one attacks. This works for subjective tasks like writing or product design. But quant evaluation is numerically deterministic: a Sharpe Ratio of 1.2 is 1.2. An IC of 0.03 is 0.03. No agent needs to \u0026ldquo;debate\u0026rdquo; whether the number is trustworthy. Anti-overfit testing is a deterministic checklist, not a judgment call. The adversarial mechanism degenerates into if-else logic.\nAll three conditions unmet. Multi-agent adds complexity here without adding value.\nThe Approaches I Considered Approach A: Multi-Agent Collaboration A CrewAI-style setup — define researcher, backtester, evaluator, risk officer as four roles, coordinate through message passing.\nWhere it\u0026rsquo;s correct: naturally parallel tasks (searching multiple sources simultaneously), non-shareable toolsets (codebase vs. production environment permission isolation), multi-perspective debate needed (product design).\nWhy it doesn\u0026rsquo;t fit my case: serial pipeline means zero parallelism gain; shared toolset makes splitting pointless; numerical evaluation needs no debate. Extra cost: serialization overhead for inter-agent context passing and cross-agent log tracing during debugging.\nApproach B: Single Agent + Finite State Machine One agent drives the entire research loop. The state machine controls behavioral boundaries and transitions:\nINIT → GENERATE → EXECUTE → EVALUATE ──→ FINISH ↑ │ └── ITERATE ┘ The core idea in one sentence: replace \u0026ldquo;coordination protocol between multiple agents\u0026rdquo; with \u0026ldquo;state transition rules within a single agent.\u0026rdquo;\nEach state transition is deterministic: if EVALUATE metrics meet the threshold, transition to FINISH; if not, transition to ITERATE back to GENERATE. No inter-agent \u0026ldquo;discussion\u0026rdquo; about what to do next.\nThe Key Judgment Call The pivotal insight wasn\u0026rsquo;t a technical comparison. It was recognizing a deeper distinction: multi-agent frameworks solve \u0026ldquo;coordination\u0026rdquo; problems, but my system doesn\u0026rsquo;t have a \u0026ldquo;coordination\u0026rdquo; problem.\nWhat my system has is an \u0026ldquo;orchestration\u0026rdquo; problem — a deterministic pipeline that needs to be automated, with LLM generation capability inserted at specific points. This is closer to workflow engine territory, not multi-agent collaboration territory.\nThis recognition came from an analogy: traditional CI/CD pipelines also have multiple stages (build → test → deploy), and could theoretically be mapped to multiple \u0026ldquo;agents.\u0026rdquo; But nobody builds CI/CD with multi-agent frameworks, because everyone intuitively knows it\u0026rsquo;s a serial orchestration problem. The quant research iteration loop is structurally isomorphic to a CI/CD pipeline.\nResults The state machine approach has been running for several months. A few quantifiable comparison points:\nZero context loss: single agent naturally shares the full research history, no intermediate results need to be passed between agents Linear traceability: every state transition has a complete record (input code, output metrics, evaluation decision, decision reasoning) — just walk the timeline when debugging Debugging time went from \u0026ldquo;piecing together logs across multiple agents\u0026rdquo; to \u0026ldquo;locating within a single state sequence\u0026rdquo; Haven\u0026rsquo;t encountered a scenario requiring parallelism. If I ever need to run 10 independent factor research tasks simultaneously, that\u0026rsquo;s a concurrency scheduling problem — asyncio.gather handles it fine, no need for inter-agent communication.\nWhat This Decision Taught Me I distilled one judgment principle from this practice:\nBefore choosing an architecture, identify the problem\u0026rsquo;s structural type. Not every system with \u0026ldquo;multiple apparent roles\u0026rdquo; needs multi-agent.\nThe method is three questions:\nIs there parallelism between tasks? Do toolsets need isolation? Does evaluation require subjective debate? If none are satisfied, don\u0026rsquo;t use multi-agent. Even if one is satisfied, evaluate whether the coordination complexity introduced is covered by the gains.\nThis mirrors the microservices trajectory exactly. Around 2015, everyone was splitting into microservices, until many teams discovered their systems didn\u0026rsquo;t need it — their problem was a deployment problem, not a service boundary problem. Multi-agent is currently going through the same inflation phase. The hype will pass. Problem structure analysis won\u0026rsquo;t.\nFirst article in the series. Next: MCP\u0026rsquo;s Problem Isn\u0026rsquo;t the Protocol — It\u0026rsquo;s the Semantic Gap.\n","permalink":"https://miasyster.github.io/en/posts/why-not-multi-agent/","summary":"Multi-agent is the hot paradigm in AI engineering. I chose single agent + state machine for my AI-driven quant research system. Not because multi-agent is too hard, but because the problem structure doesn\u0026rsquo;t match.","title":"Why I Didn't Use Multi-Agent Architecture for My Quant Research System"},{"content":"Building AI-driven quantitative research systems. Focused on AI agent architecture, quant engineering, and production-grade system design.\nThis blog documents architecture decisions, engineering lessons, and technical judgments from real-world systems. No tutorials, only decisions.\nContact: GitHub · Email\n","permalink":"https://miasyster.github.io/en/about/","summary":"\u003cp\u003eBuilding AI-driven quantitative research systems. Focused on AI agent architecture, quant engineering, and production-grade system design.\u003c/p\u003e\n\u003cp\u003eThis blog documents architecture decisions, engineering lessons, and technical judgments from real-world systems. No tutorials, only decisions.\u003c/p\u003e\n\u003cp\u003eContact: \u003ca href=\"https://github.com/Miasyster\"\u003eGitHub\u003c/a\u003e · \u003ca href=\"mailto:qq853777924@gmail.com\"\u003eEmail\u003c/a\u003e\u003c/p\u003e","title":"About"}]