Cache-Optimised Framework for Long-Running, Token-Efficient Operations

The system organises context into strict layers to maximise LLM cache hits. Order is non-negotiable.

┌─────────────────────────────────────────────────┐
│ TIER 1 — STATIC (Permanent cache)               │
│ System prompt, tool schemas, project constants  │
│ cache_control: ephemeral                        │
├─────────────────────────────────────────────────┤
│ TIER 2 — DEFAULT MEMORY (Partial cache hit)     │
│ Conversation history, tool outputs, summaries   │
│ Every entry has a unique ID (e.g., [msg_01])    │
│ Append-only unless explicitly purged/compacted  │
├─────────────────────────────────────────────────┤
│ TIER 3 — VOLATILE (No cache)                    │
│ Status telemetry and dynamic warnings           │
│ Rebuilt every cycle                             │
└─────────────────────────────────────────────────┘

Cache invariant: Tier 1 and Tier 2 content is inherently append-only. All passive mutations happen in Tier 3. This ensures the cached prefix grows monotonically. Exception: Deliberate physical deletion via purge or compact_context tools breaks the cache for one cycle but permanently reduces token bloat.

When the AI issues parallel tool calls, responses may complete in non-deterministic order. The system enforces:

• Tier 2 ordering: Parallel tool responses are appended to Tier 2 in completion order, not invocation order.

Everything stays in memory by default. All user messages, AI responses, and tool outputs inherently append to Tier 2.

To manage this accumulating memory seamlessly, every single block in Tier 2 is assigned a unique, visible ID by the system before injection.

Example Tier 2 Stream:

[msg_104] User: Find the auth errors in the logs.
[tool_call_105] AI -> web_search_agent: { ... }
[tool_res_106] System: Sub-agent report ...

The AI explicitly references these IDs to manage its memory footprint.

Memory is managed proactively by the AI using two distinct tools, depending on the severity of the context bloat.

Permanently removes specific IDs from the Tier 2 context window and optionally replaces them with a dense summary.

Input:

{
  "ids": ["msg_104", "tool_call_105", "tool_res_106"],
  "summary": "User requested auth log analysis. Sub-agent found 3 timeout errors related to Redis."
}

Usage Philosophy: The AI should use this tool when a specific sub-task is completed, when large chunks of token-heavy but irrelevant data are present, or when a dead-end path needs to be cleared.

Cache Warning: Purging physically modifies the historical prefix, triggering a mandatory cache miss. Therefore, the AI should only use purge when the token savings significantly outweigh the temporary cost of rebuilding the cache. It is highly recommended, but ultimately at the AI's discretion.

When the session is extremely long and the context window is nearing its absolute hard limit, surgical purges are no longer sufficient. This tool performs a complete state reset without losing vital project knowledge.

The 4-Step Compaction Flow:

1. Initiation: This can be triggered in two ways:

• By the AI: Detects critical token limits or severe context bloat and decides to reset.
• By the User (UI Trigger): The user manually initiates a reset from the interface (e.g., when switching to a completely new topic but needing to retain general context).

In both cases, a resume_instruction is provided (e.g., "We were about to fix the Redis timeout in auth.py. Start by inspecting the retry logic.").

1. Background LLM Call: The system pauses the main orchestrator and makes a separate LLM call. This background process analyses the entire conversation history and generates a dense CompactedState JSON. This JSON captures:

• Project summary & established context.
• Key decisions made so far.
• Completed vs. current tasks.
• Active blockers or errors.
• Affected files/components.

1. History Wipe & Injection: The system permanently wipes the current conversation history (Tier 2). It then injects the generated CompactedState JSON followed by the resume_instruction as the new, ultra-lean Tier 2 baseline.

2. Resumption: The AI resumes its work seamlessly, behaving as if there was no interruption, guided by the fresh state and the new instructions.

The main orchestrator AI does not have direct access to raw web browsing or HTML fetching. Raw web data is notoriously token-heavy. Instead, web interactions are delegated to specialised sub-agents.

Crucial Rule: Sub-agent outputs are not artificially truncated by the orchestrator beyond the underlying model or provider limits. However, every sub-agent must return its result through a predictable structured envelope. This allows the orchestrator to identify findings, actions taken, artifacts, unresolved questions, and warnings without relying entirely on free-form prose. A short agent_commentary field remains available for interpretation, recommendations, and nuance.

Spawns a sub-agent to search the web, read multiple pages, and return a synthesized answer.

Input:

{
  "query": "Latest React 18 concurrent rendering best practices",
  "focus": "Look specifically for performance impacts in large lists."
}

(Note: focus is optional but highly recommended. If omitted, the sub-agent performs a general search while still returning the standard structured result envelope.)

When the AI already has a specific URL, it uses this tool to extract meaning rather than raw markup.

Input:

{
  "url": "https://api.example.com/v2/docs",
  "focus": "Extract the authentication headers required for the /users endpoint."
}

Behaviour: A sub-agent fetches the raw HTML/text, processes it according to the requested focus, and returns a concise structured result directly to the orchestrator's Tier 2. The result contains a factual summary, key findings, source or artifact references when applicable, unresolved questions, warnings, and an optional free-form agent_commentary field.

For complex, multi-step tasks, the AI uses delegate_task to keep its own context clean.

Input:

{
  "tool": "delegate_task",
  "parameters": {
    "goal": "Refactor the authentication module to use async/await",
    "constraints": "Maintain backward compatibility."
  }
}

Delegation Rules:

• The sub-agent runs in total isolation.
• Upon completion, it returns a structured result describing its status, summary, findings, actions taken, artifacts, unresolved questions, and warnings.
• A short agent_commentary field may contain free-form interpretation, recommendations, trade-offs, or other useful nuance.
• The complete structured result is appended directly to the orchestrator's Tier 2.

Standard Sub-Agent Result Envelope:

{
  "status": "completed",
  "summary": "Concise factual description of the result.",
  "findings": [
    {
      "finding": "A concrete finding.",
      "evidence": "Supporting evidence or observation.",
      "confidence": "high"
    }
  ],
  "actions_taken": [],
  "artifacts": [],
  "open_questions": [],
  "warnings": [],
  "agent_commentary": "Short free-form interpretation or recommendation."
}

Fields that are irrelevant to a specific task may be omitted. Task-specific sub-agents may also extend this envelope with fields such as sources, modified_files, tests_run, or recommended_plan, while preserving the common top-level structure.

Tier 3 displays essential telemetry, rebuilt every cycle to keep the AI aware of its limits:

Status: Context: 112,000 / 128,000 (87%) | System Status: Optimal

Before every model invocation, tool execution, or sub-agent call, the orchestrator verifies that the operation can safely fit within the remaining context capacity.

The available operation budget is calculated as:

available_tokens =
    context_limit
    - current_context_tokens
    - reserved_completion_tokens
    - protocol_overhead_tokens
    - safety_margin_tokens

An operation may begin only when its estimated result size fits within this available budget.

• Warning Threshold (80%): Tier 3 warns the AI that context pressure is increasing. The AI should avoid unnecessary context growth and consider using purge.
• High-Risk Threshold (90%): Large tool calls and sub-agent operations must use conservative output estimates and an increased safety margin.
• Critical Threshold (95%): High-volume operations are blocked until the AI uses purge or compact_context.
• Insufficient Operation Budget: An operation is blocked immediately whenever its estimated output cannot safely fit, regardless of the current percentage.

When an operation is blocked, the orchestrator returns a structured notification:

{
  "status": "context_budget_exceeded",
  "pending_operation_id": "op_1842",
  "current_context_tokens": 108000,
  "estimated_result_tokens": 10000,
  "available_operation_tokens": 6000,
  "required_action": "purge_or_compact"
}

After the context has been reduced, the original operation may be resumed using its stable operation ID.

The orchestrator enforces capacity limits only. It must not independently delete, reorder, summarize, or mutate Tier 2 entries. Memory cleanup remains deliberate and must occur through the existing purge or compact_context tools, preserving the architecture's cache stability.