Microservices Resources

DZone's Featured Microservices Resources

Will AI Keep Us Stuck in 2020 Architectures?

By Daniel Sagenschneider

Every time I sit down with an AI coding assistant, I notice the same thing: it is very good at Spring. Annotations, profiles, @Autowired, the whole call-stack-driven dance of beans wiring into beans. AI has seen twenty years of this. It guesses well, even when it has to infer how a profile-specific bean is going to be selected at runtime. This is because it has seen ten thousand examples of exactly that pattern. Which raises an uncomfortable question for anyone working on a new architecture: if AI is this fluent in 2020-era patterns, are we as an industry going to stay locked into those patterns simply because that's what the model knows? Is AI a conservative force that quietly drags software architecture backward to its training data's center of mass, no matter how good a newer idea might be? I wanted to find out, using my own project as the test case. The Bet: An Explicit Index Beats an Implicit One OfficeFloor version 4 added a feature I think is genuinely interesting for the AI era: REST endpoints can now be defined in YAML files, sitting alongside your existing Spring Boot code, with the directory structure following the URL structure. A file at greeting.POST.yml defines POST /greeting. A file at greeting/{name}.GET.yml defines GET /greeting/{name}. Inside that file, you compose the small functions that handle the request: YAML # greeting.POST.yml validate: class: ValidateGreetingLogic outputs: valid: build build: class: PostGreetingLogic next: audit audit: class: AuditGreetingLogic Each function still gets its dependencies injected by Spring exactly as it always has. OfficeFloor doesn't replace Spring's DI, persistence, security, or actuator setup. What changes is the flow. In a typical @RestController, the order in which validation, business logic, and auditing run is implicit: it lives in the call stack (in if statements and which methods call which other methods). To understand it, you read code. To change it, you read more code, because the wiring isn't written down anywhere as data; it's compiled into control flow. In the OfficeFloor YAML version, that wiring is the file. Conditional branches, sequencing, error flows: they're declared, not buried. No function in the chain knows about the others. No annotation is describing the relationship from inside a class. The YAML is a complete, readable specification of how the endpoint behaves, sitting right next to the endpoint's own URL path in the directory tree. This is essentially Function Injection, the same move Dependency Injection (DI) made decades ago, but one level up. DI took "what do I depend on" out of imperative constructor code and made it an explicit, configured, first-class concern. Function Injection takes "what happens next" out of the implicit call stack and makes that explicit and configured too. It's a continuation of the Inversion of Coupling Control idea I've been writing about for years: Dependency Injection only ever solved one slice of the coupling problem. Control flow coupling was always still there, just invisible. For a human reading the code, this might feel like a wash, maybe even a step backward. This is exactly why the industry settled on annotations next to code in the first place; developers wanted the wiring close to the implementation, not off in some separate descriptor. That preference made sense when humans were the primary readers doing the navigating. But an AI assistant isn't a human reading top to bottom. An AI assistant is trying to find the minimum context needed to make a correct, surgical change, and that's a search and navigation problem, not a stylistic one. A YAML file that names every function in an endpoint's execution path, in order, with explicit conditional branches, is a search index. The AI doesn't need to read the rest of the code base to be confident it has found everything relevant to that endpoint. It opens one small file, and the entire behavioral contract of that URL is sitting right there. That's the theory, anyway. I wanted to know if it would actually hold up against a model that has been trained almost exclusively on the other way of doing things. The Experiment: Converting Spring PetClinic REST To test this for real, rather than on a toy example, I took Spring PetClinic REST, the long-standing reference REST implementation of the PetClinic sample app that the Spring community has used for years, and worked with AI to convert its endpoints over to the OfficeFloor REST YAML approach. It did not work on the first attempt. It took about five iterations to get a clean conversion, and the bottleneck wasn't OfficeFloor's runtime, and it wasn't really the AI's coding ability either. It was documentation. Each attempt surfaced a gap in the tutorials: some assumption I'd left implicit because it was obvious to me, a place where the YAML schema's possibilities weren't spelled out, an edge case in how a Spring @RestController-style behavior should map across. I used the AI's confusion as a signal: where it guessed wrong or asked the wrong question, that was exactly where the tutorial needed another paragraph, another example, another explicit rule. Five rounds of "AI gets stuck, tutorial gets fixed, try again" later, the conversion went through cleanly. I recorded the final working conversion. You can watch it here: Spring PetClinic REST to OfficeFloor REST YAML You can also see the resulting changes in the forked repository pull request. So Which is It? Does AI Lock in 2020 Architecture or Not? Both things turned out to be true, depending on what's actually being asked of the AI. Where AI defaults to what it knows: left to its own judgment, an AI assistant will reach for Spring conventions, because Spring conventions are the statistically dominant pattern in its training data. If you ask it to "add a REST endpoint" with no further steering, you'll get an @RestController and an @Autowired field every time. That's not a flaw in the model. It's just what twenty years of public code looks like, averaged. Where AI happily adopts something new: the moment the new pattern is clearly and completely specified, the model's prior training stopped being an obstacle and became almost irrelevant. AI doesn't need to have seen ten thousand examples of a YAML-driven REST framework to use one correctly. It needs an accurate, complete description of the schema and the conventions, and then it follows that description. The five-iteration process wasn't really "teaching the AI to think differently." It was closing the gaps between what I assumed was obvious and what was actually written down anywhere the AI could read it. That reframes the original question. The risk isn't that AI is architecturally conservative by nature. The risk is that new architectures rarely come with documentation anywhere near as exhaustive as Spring's, because Spring's documentation had two decades and a vast community writing tutorials, blog posts, Stack Overflow answers, and books about it. A new approach starts that race from zero. If its docs stay thin, AI will keep defaulting to Spring patterns, not out of preference, but because Spring is simply the only option it has enough information about to be confident in. So the honest answer is: AI won't keep us in 2020 architectures by itself. But it will, by default, if nobody does the work of making the alternative legible to it. The model doesn't have an opinion about which architecture is better. It has a confidence gradient shaped entirely by how well-specified each option is in what it's been able to learn or been given. The Interesting Part for Framework and Tool Authors If this holds generally, and I'd be curious whether others doing similar work see the same thing, it changes the calculus for anyone designing a new way of building software in the AI era. It used to be that the cost of an explicit, separated configuration artifact (think XML wiring files, or graphical configuration tools) was paid almost entirely by human developers, who found it slower to read and slower to navigate than code-adjacent annotations. That cost was real, and it's a big part of why annotation-driven frameworks like Spring won the last decade. AI changes that cost calculation. An explicit, structured index, a YAML file that names every function and every transition in an endpoint, located exactly where the URL structure says it should be, costs an AI assistant almost nothing to read and a great deal less to get wrong, because there's no implicit call-stack archaeology required. The structure that used to be a tax on humans is now a gift to the thing increasingly doing a large share of the maintenance work. But that gift only arrives if someone pays a different tax: writing the documentation thoroughly enough, and unambiguously enough, that an AI assistant can pick up the new pattern from the docs alone, the way it picked up Spring from a decade of incidental exposure. Architecture innovation in the AI era may end up being gated less by "is this a good idea" and more by "is this idea legible to an AI that has never seen it before." That's a genuinely different bar than the one we used to optimize for, and it's one I think is worth more people paying attention to. If you want to look at the actual schema, the tutorials, or try the conversion yourself, the starting point is the OfficeFloor REST tutorials, and the Spring PetClinic REST source is on GitHub if you want to attempt your own conversion and see where your AI assistant gets stuck. That's usually exactly where the next improvement to the docs needs to go. More

Designing Scalable Containerized Backend Services

By Estefanio Fernando

Modern enterprise software design has fundamentally shifted away from monolithic, single-threaded runtimes toward decoupled, containerized architectures. When building systems that handle high throughput — such as fintech services, automated reporting pipelines, or real-time distributed platforms — engineers must address two core infrastructure vectors: high-concurrency connection management and deterministic relational state execution. A common anti-pattern in backend systems engineering is assuming that containerization automatically scales an application. In reality, wrapping a poorly optimized, blocking database service inside a Docker container simply shifts the performance bottleneck from local computing hardware to network sockets and thread pools. This technical guide breaks down the implementation of a decoupled, high-concurrency microservice engine built on an asynchronous Python core, using localized relational persistence layer mechanics, automated connection pooling, and multi-tier containerized deployment. The Concurrency Bottleneck in State Management In traditional synchronous web gateways, each incoming network transaction is mapped directly to an operating system (OS) thread. If an application needs to process a complex relational query, generate a structural file, or compute algorithmic abstractions, that thread remains blocked until the underlying process returns a code sequence. Under heavy enterprise concurrency patterns, thread starvation occurs. The OS spends more execution cycles executing thread context switches than processing actual transaction payloads. Textile [Blocking Monolith] ───► Thread 1 ───► [DB Read / Long IO Block] ───► (Thread Starvation) [Asynchronous ASGI] ───► Async Worker ───► Loop Delegation ───► (Thread Free to Serve Next Connection) To achieve true horizontal scaling within a decoupled microservices architecture, we must exploit non-blocking Asynchronous Server Gateway Interface (ASGI) frameworks combined with explicit, non-blocking asynchronous drivers for relational storage layers. This structural change ensures that database connection limits do not become a systemic point of failure. Architectural Blueprint: The Asynchronous Core Engine The following implementation showcases an advanced, high-performance transactional microservice core engine. The software structure uses asynchronous runtime loops to manage localized transactional persistence dynamically without blocking the main worker loop. Python import asyncio import logging import uuid from typing import AsyncGenerator from pydantic import BaseModel, Field from contextlib import asynccontextmanager # Setup structured system tracking logs logging.basicConfig(level=logging.INFO, format="[%(asctime)s] %(levelname)s: %(message)s") logger = logging.getLogger("MicroserviceCore") # Define strict, type-validated data transmission contracts class FinancialPayload(BaseModel): transaction_id: str = Field(default_factory=lambda: str(uuid.uuid4())) account_source: str amount_tokens: float = Field(gt=0.0) system_metadata: dict = Field(default_factory=dict) class ServiceInfrastructureEngine: """ Manages isolated transactional operations, simulating high-performance relational database pool connections under non-blocking event loops. """ def __init__(self, database_connection_string: str, max_pool_size: int = 20): self.connection_string = database_connection_string self.max_pool_size = max_pool_size self._execution_semaphore = asyncio.Semaphore(max_pool_size) async def initialize_state_store(self) -> None: """Simulates automated schema migration and table validation.""" logger.info(f"Connecting to data storage system: {self.connection_string}") await asyncio.sleep(0.5) # Simulate non-blocking I/O connection handshake logger.info("Relational schemas and operational metadata verified successfully.") @asynccontextmanager async def acquire_pooled_session(self) -> AsyncGenerator[str, None]: """Context manager enforcing connection pooling constraints asynchronously.""" await self._execution_semaphore.acquire() session_id = f"DB_SESSION_{uuid.uuid4().hex[:8].upper()}" try: yield session_id finally: self._execution_semaphore.release() async def execute_isolated_transaction(self, payload: FinancialPayload) -> bool: """Executes a strict transaction block simulating multi-stage persistence loops.""" async with self.acquire_pooled_session() as session: logger.info(f"[{session}] Opening isolated database transaction context for ID: {payload.transaction_id}") # Simulate a continuous analytical lookup pattern or sub-query sequence await asyncio.sleep(0.2) # Verify pseudo ledger balances if payload.amount_tokens > 50000.0: logger.warning(f"[{session}] Risk flags raised. Transaction {payload.transaction_id} requires multi-signature validation.") return False await asyncio.sleep(0.1) # Simulate final commit state save logger.info(f"[{session}] Transaction successfully written and finalized in relational engine.") return True # Entry point simulation for a highly distributed parallel transaction load async def main(): # Instantiating our microservice engine layer infra_engine = ServiceInfrastructureEngine(database_connection_string="sqlite+aiosqlite:///production_ledger.db") await infra_engine.initialize_state_store() # Generate an explicit array of simultaneous simulated client event payloads mock_requests = [ FinancialPayload(account_source=f"ACC_USR_{i:03d}", amount_tokens=150.0 * i) for i in range(1, 15) ] # Map requests into non-blocking concurrent tasks execution_tasks = [infra_engine.execute_isolated_transaction(req) for req in mock_requests] # Execute all database loops concurrently across the async thread gateway results = await asyncio.gather(*execution_tasks) logger.info(f"Engine batch processing sequence finished. Successful operations: {results.count(True)}/{len(results)}") if __name__ == "__main__": asyncio.run(main()) Containerization Strategy: Engineering Production Docker Environments To guarantee predictable horizontal execution layers across hybrid cloud network topologies, we deploy the engine utilizing highly optimized multi-stage containerization blueprints. A common operational failure is building bulky production images that include build tools (compilers, development headers, package managers) in the final image. This significantly expands the system's attack surface and increases cold-start image download latency over orchestrated node servers. Below is the optimized, multi-stage Dockerfile blueprint designed to ensure image portability and minimal security overhead: Dockerfile # Stage 1: The Build/Compilation Environment FROM python:3.11-slim AS build-compiler WORKDIR /app RUN apt-get update && apt-get install -y --no-install-recommends \ build-essential \ libpq-dev \ && rm -rf /var/lib/apt/lists/* COPY requirements.txt . # Compile and dump dependencies natively into a local virtual deployment tree RUN python -m venv /opt/venv ENV PATH="/opt/venv/bin:$PATH" RUN pip install --no-cache-dir -r requirements.txt # Stage 2: The Final Operational Secure Runtime FROM python:3.11-slim AS runtime-layer WORKDIR /app # Install localized system runtime libraries required by standard database interfaces RUN apt-get update && apt-get install -y --no-install-recommends \ libpq5 \ && rm -rf /var/lib/apt/lists/* # Pull exclusively compiled executable components from the isolated build stack COPY --from=build-compiler /opt/venv /opt/venv COPY . . # Set immutable system paths and configuration markers ENV PATH="/opt/venv/bin:$PATH" ENV PYTHONUNBUFFERED=1 ENV ENVIRONMENT=production EXPOSE 8000 # Enforce secure operational constraints by bypassing root execution privileges USER 1001 CMD ["python", "main.py"] Key Takeaways for Production Systems Explicit resource bound manipulation: As seen in the asynchronous script implementation, utilizing asyncio.Semaphore allows developers to hard-code a strict ceiling over concurrent processing paths, mitigating database exhaustion errors.Deterministic data contracts: Leveraging schema wrappers (like Pydantic or native object-relational models) ensures invalid or corrupted objects fail at the interface boundary before hitting core internal relational storage pipelines.Multi-stage build standards: Decoupling the development environment from the final execution layer results in production-ready, low-footprint containers, making the backend infinitely more resilient and ready for automated container management layers. Building enterprise-grade web backends requires careful alignment between non-blocking application loops and resource constraints. By implementing asynchronous architectures and optimized container multi-staging, software engineers can design resilient systems capable of sustaining high-throughput data pipelines efficiently. More

Reducing CI Execution Time Using Impact-Based Test Selection Across Repositories

By Raakesh Rajagopalan

Anti-Patterns of Microservices Architecture From Real Production Experience

By Ivan Balashov

When AI Agents Call Your Microservices: 5 Assumptions That No Longer Hold

By Maryna Klochkova

Architecting Autonomous Network Ecosystems: From Reactive Monitoring to Agentic AI Orchestration

Agentic AI systems represent a paradigm shift in network operations, facilitating the transition from traditional reactive monitoring to fully autonomous management frameworks. For global infrastructure leaders, these specialized AI agents serve as persistent digital engineers, providing round-the-clock expertise across deployment, maintenance, and complex troubleshooting lifecycles. The following blueprint delineates the strategic application of agentic AI within a global enterprise network operations environment. Architectural Blueprint: Multi-Agent Network Engine Rather than deploying a monolithic AI entity, the optimal architecture utilizes a multi-agent system (MAS). In this model, specialized agents collaborate through a centralized Orchestrator to achieve complex operational objectives. Please refer to the System Hierarchy and Flow, as well as the Strategic Vision and Ecosystem diagrams. System Hierarchy and Flow Strategic Vision and Ecosystem Core Use Cases and Agent Workflows Autonomous Provisioning and Zero-Touch Deployment Functional scope: Serves as an automated configuration and systems engineer.Workflow: Human inputs a high-level intent (e.g., "Provision a new Cisco core switch for the Seoul R&D campus").Design Agent reads existing topology diagrams and ipAM (IP Address Management) databases.Configuration Agent generates vendor-specific CLI configs (Cisco IOS/NX-OS).Validation Agent runs the config in a virtual sandbox (e.g., Cisco Modeling Labs) to check for routing loops before pushing to production via Ansible. 24/7 Autonomous Incident Triage and Resolution Functional scope: Functions as the primary intelligence layer for Network Operations Center (NOC) alerts.Workflow: A monitoring tool (e.g., Splunk, SolarWinds) triggers a "BGP neighbor down" alert.Triage Agent intercepts the alert and instantly queries the device via SSH or telemetry API.Diagnostic Agent executes diagnostic scripts (show ip bgp summary, traceroute), analyzes logs, and identifies the root cause (e.g., a flapping fiber link).Resolution Agent opens a Jira ticket, attaches all logs, attempts a safe automated fix (like shifting traffic to a backup path), and pages a human only if the physical hardware needs replacement. Predictive Maintenance and Capacity Optimization Functional scope: Operates as a proactive infrastructure strategist and planner.Workflow: Analysis Agent continuously monitors interface bandwidth, memory usage, and optical transceivers' error rates across global data centers.Forecasting Agent identifies trends (e.g., "Link X between Vietnam and Korea factories will hit 95% capacity in 30 days during peak hours").Planning Agent drafts a change management proposal recommending a bandwidth upgrade or traffic re-routing plan for human review. Automated Compliance and Security Patching Functional scope: Functions as a continuous vulnerability and configuration compliance auditor.Workflow: Cisco releases a critical security advisory for an OS vulnerability.Auditor Agent scans the entire inventory database to find all vulnerable device models.Patching Agent schedules a maintenance window, backs up current configurations, downloads the verified firmware, applies the patch sequentially to avoid downtime, and performs post-flight checks. Implementation Strategy To ensure organizational stability while maximizing technological advancement, a phased implementation strategy is recommended: Phase 1: Read-Only Agents (Information Gathering) Allow agents to access logs, APIs, and read-only commands.Focus on automated documentation, ticket enrichment, and summarizing alerts. Phase 2: Human-in-the-Loop (Co-Pilot) Allow agents to generate fixes and configuration scripts.Require a senior network engineer to review and click "Approve" before any execution. Phase 3: Guardrailed Autonomy (Full Agentic) Give agents autonomous execution rights only for low-risk, repetitive tasks (e.g., port resets, clearing stuck sessions).Enforce hard boundaries using API rate limits and strict verification checklists. To refine the architectural requirements for your specific environment, please provide insights on the following: What monitoring tools (e.g., Datadog, ServiceNow) and automation frameworks (e.g., Ansible, Terraform) does your team currently use?What is the most repetitive issue your 24/7 NOC team spends time fixing manually?Are you looking to build these agents using commercial LLM APIs or host open-source models locally for security reasons? Building an Enterprise-Grade Agentic System Constructing an enterprise-grade agentic system for high-security environments requires the integration of a frontier large language model (LLM), a robust production agent framework, and the Model Context Protocol (MCP) to establish secure interoperability with core network infrastructure. Part 1: Choosing the Right Tool/Solution Do not rely on a simple chatbot web interface. You need an API-driven architecture that combines a foundational model with an agent framework. The Foundational Model (The Brain) Top Choice: Anthropic Claude 3.5 Sonnet / Claude 3 Opus Why: It currently dominates tool-use (function calling), reasons through complex, multi-step engineering logic flawlessly, and natively supports MCP. On-Premises / Air-Gapped Alternative: Llama 3.1 / 3.3 (70B or 405B) or Qwen 2.5 (72B) Why: For internal Samsung networks with strict data privacy laws, you can host these open-source models locally using vLLM or Ollama. They have excellent coding and structured JSON output capabilities. The Agent Framework (The Backbone) Use an open-source framework to manage agent memory, states, and collaboration: LangGraph (by LangChain): Best for networking. It allows you to build cyclical, graph-based agent workflows with strict state control and mandatory human-in-the-loop approval stages.CrewAI: Great for quickly setting up role-based multi-agent teams (e.g., Triage Agent talking to a Scripting Agent). Part 2: Understanding MCP (Model Context Protocol) MCP is an open standard created by Anthropic that acts as a secure plug-and-play adapter between LLMs and local/remote data sources, tools, and systems. Instead of writing custom API integration code for every single Cisco switch, Ansible Tower, or Jira server, you build or use an MCP Server. How it works: Your Agent (MCP Client) connects to an MCP Server. The MCP Server exposes specific tools (e.g., run_cisco_command, query_monitoring_alerts) to the agent dynamically. Security Benefit: The model never gets direct SSH access to your core switches. The model only talks to the MCP Server, which enforces strict argument validation, sanitization, and logging before executing anything on the network. Part 3: Writing agents.md The agents.md file defines the architecture, roles, personas, boundaries, and collaboration patterns of your AI team. Markdown # Network Operations Multi-Agent Architecture This document defines the specialized AI agents operating within the Network Operations Center (NOC). ## 1. System Orchestrator Agent - **Role:** Central Dispatch & Intent Router - **Goal:** Analyze incoming alerts or human engineer requests and delegate tasks to specialized agents. - **Persona:** A highly organized, senior technical project manager. - **Boundaries:** Does not execute network commands directly. Must always route tasks and compile final reports. ## 2. Network Diagnostics Agent (NetDiag-Agent) - **Role:** Incident Triager and Log Analyst - **Goal:** Investigate network anomalies, verify device status, and pinpoint root causes. - **Persona:** An analytical Cisco CCIE-certified troubleshooting expert. - **Tools Allowed:** `ssh_read_only_commands`, `query_splunk_logs`, `ping_traceroute`. - **Boundaries:** Read-only access to infrastructure. Cannot modify configurations. ## 3. Network Automation Agent (NetAuto-Agent) - **Role:** Configuration and Deployment Engineer - **Goal:** Generate valid, syntax-correct network configurations and execute approved automation scripts. - **Persona:** A precise Network Automation Engineer specializing in Ansible, Python (Netmiko/Nornir), and Cisco IOS/NX-OS. - **Tools Allowed:** `generate_ansible_playbook`, `validate_config_syntax`, `execute_stage_change`. - **Boundaries:** Write-access allowed *only* through staging/sandbox tools. Any production push requires an explicit `Human-In-The-Loop` (HITL) approval token. Part 4: Writing skills.md The skills.md file maps out the actual capabilities, API tools, and Python execution blocks that the agents can tap into via MCP. Markdown # Network Agent Skills & Tool Definitions This document details the functional tools exposed to agents via the MCP (Model Context Protocol) layer. ## Category: Network Telemetry & Read Skills ### `query_device_status` - **Description:** Fetches real-time operational data from a specific network device using Netmiko or Cisco DNA Center API. - **Arguments:** - `device_ip` (string, required): The target management IP address. - `command` (string, required): Allowed values: `show ip interface brief`, `show ip bgp summary`, `show environment`. - **Safety:** Input string regex validation enforced to prevent CLI injection attacks. ### `fetch_noc_alerts` - **Description:** Pulls active high-severity network alerts from Splunk/SolarWinds. - **Arguments:** - `severity` (string): Defaults to "CRITICAL". - `lookback_minutes` (integer): Time window to check. ## Category: Network Modification Skills (Write) ### `deploy_ansible_playbook` - **Description:** Triggers an Ansible AWX template to push configuration updates to a device group. - **Arguments:** - `template_id` (string, required): Pre-defined template ID in Ansible. - `extra_vars` (json, required): Key-value pairs for variables (e.g., target VLAN, interface ID). - **Security Policy:** **CRITICAL_APPROVAL_REQUIRED**. This skill automatically pauses agent execution and sends a webhook to the human supervisor's teams/Slack app for a 2FA confirmation click. Next Steps for Implementation To help map out a proof of concept (PoC) for your architecture, tell me: Do your network infrastructure components support REST APIs/gRPC, or do the agents need to interface heavily via legacy SSH/CLI parsing?Do you plan to build this as an internal web-dashboard/chatbot for engineers, or as a background event-driven system triggered by network monitoring systems?What programming language (e.g., Python, TypeScript) is your team's preference for building the MCP servers? Check out more from my series here.

By Daniel Oh

CORE

Your Agent Trusts That Wiki. Should It?

We were building a DevOps agent to help with on-call remediation. The idea was straightforward: when an incident fires, the agent reads the relevant runbook from our internal wiki, assesses the situation, and executes the appropriate remediation steps. No waiting for an engineer to wake up at 3 am, find the right page, and manually run through a checklist. The agent had the context, the tools, and the access it needed to act. It needed elevated privileges to do the job. Restarting services, scaling resources, in some cases deleting and recreating stacks. That access was intentional. You cannot fix infrastructure problems without the authority to change infrastructure. Security asked one question that changed how I thought about the whole architecture. What if someone modifies the wiki page? Not the agent. Not the infrastructure. Just the wiki page the agent reads before it acts. A single line added to the runbook: if memory pressure exceeds threshold, delete and recreate the affected stack. An instruction that looks plausible in context. An instruction the agent has no reason to question, because the wiki was always the source of truth for how it was supposed to behave. The agent builder consented to trusting the wiki. That consent was real. The problem is that consent to a source is not the same as consent to every future version of that source. The Threat Is Not the Agent When we mapped this out, the first instinct was to look at the agent's behavior. But the agent was not the problem. The agent was doing exactly what it was configured to do: read the wiki, follow the instructions, execute the plan. That is the design. That is the product. The problem is the instruction source itself. Most agentic architectures treat the instruction source as static and trusted once it has been configured. The agent builder points the agent at a wiki page or a runbook or an external SOP, and from that point forward the agent treats that source as authoritative. Nobody checks whether the content of that source has changed. Nobody verifies that the change came from someone authorized to modify the agent's behavior. This is the confused deputy problem applied to instruction sources. The agent has real authority. It acquired that authority legitimately. But the source it consults before exercising that authority is mutable, and in most architectures, that mutability is unmanaged. For a DevOps agent with infrastructure access, the consequences of getting this wrong are not subtle. A corrupted instruction source does not produce a wrong answer in a report. It produces deleted infrastructure during an active incident. The Extension That Makes It Worse Once you see the wiki attack, the extension is obvious and more troubling. Many teams do not limit their agents to internal sources. Vendor SOPs, external runbooks, third-party documentation, public knowledge bases. An agent that fetches operating instructions from an external source has the same vulnerability with a much larger attack surface. The attacker no longer needs internal access. They need to compromise a vendor's documentation page, or inject content into a public runbook the team linked to, or modify a page the agent fetches as part of its normal operating context. The agent builder consented to fetching that external source. The consent was real. What arrives from that source on any given execution is outside their control. The threat vector is identical. The blast radius is larger. And the detection is harder because external sources changing their content is entirely normal behavior that generates no alert. Why Standard Defenses Miss This Input filtering looks for malicious content. The instruction to delete and recreate the affected stack is not malicious content in isolation. It is a standard infrastructure operation that appears in legitimate runbooks constantly. A filter has no way to know this instruction was inserted by an attacker rather than written by the team that owns the runbook. Output guardrails evaluate whether an individual action looks dangerous. Deleting a stack is an action the agent is authorized to perform. It does not look dangerous in isolation because in the right context it is the correct remediation. The problem is not the content of the instruction or the nature of the action. The problem is whether the version of the instruction source that triggered this action was authorized by the people who built the agent. Most architectures have no answer for that question. Measuring the Gap: Instruction Source Trust Benchmark To understand how different defensive approaches perform against this threat vector, we evaluated 100 scenarios across four instruction source types, testing 60 legitimate executions and 40 cases where the instruction source had been modified to introduce a harmful action. Benchmark dataset: Instruction Source Type Test Cases Modification Attempts Description Internal wiki 30 12 Team-maintained runbooks, internal SOPs External vendor documentation 25 10 Third-party runbooks fetched at runtime Dynamically fetched web content 25 10 Public knowledge bases, linked references Hybrid (internal + external) 20 8 Agents combining multiple instruction sources Detection results: Defense Approach Legitimate Allowed Harmful Caught Missed False Positive Rate No defense 60 / 60 2 / 40 38 / 40 0% Content filtering only 58 / 60 14 / 40 26 / 40 3.3% Instruction source integrity checks 56 / 60 37 / 40 3 / 40 6.7% Content filtering caught 35% of harmful modifications. Instruction source integrity checks caught 92.5%. The 3 missed cases involved modifications that stayed within the expected vocabulary of the runbook, changing thresholds and conditions rather than introducing new action types. The hybrid source scenario was consistently the worst performing across all defensive postures. What Instruction Source Integrity Actually Looks Like The core insight is that instruction sources need to be treated like code, not like content. Code changes go through review and approval. Content changes in most wikis do not. An agent with infrastructure access should not be consuming instruction sources that have lower change control standards than the systems it can modify. First, pin instruction sources to versioned snapshots rather than live content. The agent reads the version of the wiki page that was approved for its use, not whatever the page says at execution time. Second, cryptographically sign approved instruction content. The agent verifies the signature before acting on any instructions. Here is a simplified implementation: Python from enum import Enum from dataclasses import dataclass import hashlib class SourceTrust(Enum): PINNED_INTERNAL = 'pinned_internal' LIVE_INTERNAL = 'live_internal' EXTERNAL = 'external' @dataclass class InstructionSource: content: str source_url: str trust_level: SourceTrust content_hash: str approved_hash: str HIGH_PRIVILEGE_ACTIONS = { 'delete_stack', 'recreate_stack', 'scale_down', 'modify_security_group', 'revoke_credentials' } def verify_instruction_source(source: InstructionSource) -> bool: current_hash = hashlib.sha256(source.content.encode()).hexdigest() if current_hash != source.content_hash: raise ValueError('Source content hash mismatch. Possible tampering.') if source.trust_level == SourceTrust.PINNED_INTERNAL: return source.content_hash == source.approved_hash if source.trust_level == SourceTrust.EXTERNAL: raise PermissionError('External sources cannot trigger privileged actions.') return False def authorize_agent_action(action: str, source: InstructionSource) -> bool: verified = verify_instruction_source(source) if action in HIGH_PRIVILEGE_ACTIONS and not verified: raise PermissionError(f'Action {action} requires pinned approved source.') return True When the wiki page is modified without going through the approval process, content_hash no longer matches approved_hash. The agent halts before executing any high-privilege action. The infrastructure stays intact. Third, treat external sources as data, not instructions. If your agent fetches content from a vendor SOP, that content should inform understanding but should not directly trigger actions. Authorization must trace back to an internally approved instruction source. The Persistent Version Is the Dangerous One A one-time modification to the wiki triggers one bad execution. That might be recoverable. A sustained modification is different. An attacker who understands your agent's execution schedule modifies a threshold condition in the runbook, something subtle enough to pass a casual review, that causes the agent to take progressively more aggressive remediation actions under increasingly common conditions. Each execution looks like a legitimate response to a legitimate alert. The pattern only becomes visible when you look at the aggregate effect over time. Most teams do not have a view of how their agent's instruction sources have changed over time relative to the actions the agent took. Building that audit trail is not optional for agents with elevated privileges. It is the minimum viable safety property. What Needs to Change Version and approve instruction sources the same way you version and approve code. An agent with infrastructure access should not be reading live wiki content any more than your deployment pipeline should be pulling unreviewed code from a feature branch. Build privilege tiers into your action registry. Read operations can tolerate more source ambiguity than write operations. High-privilege actions should require the highest source trust level, enforced at runtime. Mirror external sources before trusting them. Fetch vendor SOPs, review them, and promote an approved snapshot to an internal trusted source. Never let the agent act on live external content for anything beyond read operations. Log instruction source versions alongside action logs. When the agent acts, record which version of which instruction source authorized that action. When something goes wrong, you need to answer that question without reconstructing it from memory. The Pattern We Keep Repeating SQL injection taught us not to trust user input as SQL commands. CSRF taught us not to trust browser requests without origin verification. Each time, the lesson was the same: the system had real authority, and the source of the instruction that exercised that authority was not sufficiently verified. Prompt injection in agentic systems is the same lesson again. The model is not the problem. The agent is not the problem. The problem is that we are giving systems real authority over real infrastructure and then consuming instruction sources with the same trust model we use for a shared team wiki. Those two things are not compatible. The sooner we treat instruction sources for privileged agents with the same rigor we treat the code those agents run, the fewer 3am incidents we will be explaining to an executive the next morning.

By Abhinav Srivastava

From Bash Script to Operational Triage: What Eight Months of Kubernetes Debugging Taught Me

In November 2025, I published a Bash script that analyzed Kubernetes clusters in about 60 seconds. It generated HTML reports, surfaced crash loops, orphaned resources, and other operational issues that were easy to overlook. The most interesting part wasn't the script — it was what happened after people started running it. Many told me they found problems they hadn't known existed. Looking back, the bash script wasn't really solving debugging. It was solving prioritization. I just didn't have the vocabulary for it yet. That script eventually became four different experiments, then a collection of small scanners, and eventually the dashboard shown in this article. Over the next eight months, that script evolved into OpsCart Watcher — an open-source operational triage dashboard for Kubernetes. This article is about what the journey taught me, and what I think is still missing from most Kubernetes environments. OpsCart Watcher — operational triage for Kubernetes (6 minutes) The Problem the Script Revealed The script did one thing well: it looked at an entire cluster and listed what was broken. Engineers who ran it kept telling me the same thing — "I had no idea this was there." That response was the important signal. These engineers had Grafana, Prometheus, and kubectl. Visibility was not their problem. The problem was that nothing told them to look at this specific namespace, this specific pod, this specific storage volume — before it became an incident. Consider a pod in CrashLoopBackOff for 19 days with 5,000+ restarts. To a metrics dashboard, that deployment looks healthy: replica count satisfied, a pod exists in Running state between crashes, CPU and memory flat because the container barely lives long enough to consume anything. The dashboard is answering the question it was built to answer — is the cluster meeting its SLOs? — and the answer is yes. The question nobody built tooling for: what deserves attention right now? LayerWhat It AnswersToolsMetricsIs the cluster meeting its SLOs?Prometheus, Grafana, DatadogPer-resource stateWhat is this specific pod doing?kubectl, k9s, LensOperational triageWhat deserves attention right now?Prioritizing operational work across cluster state What Triage Looks Like in Practice Overview page — Incident Score 41/100, KPI bar, Top 5, War Room panel The first time I ran the rebuilt dashboard against a cluster with real failures, the top of the screen didn't show me a CrashLoopBackOff pod. It showed me four CrashLoopBackOff pods spread across three namespaces, collapsed into a single operational problem: Plain Text 1. 4 pods crash-looping CRITICAL payments/fraud-detection (1810 restarts) → kubectl logs fraud-detection-... -n payments --previous That collapsing is the entire idea. Instead of inspecting every deployment individually, I was looking at a ranked list of operational problems — each with a severity, a location, and the exact kubectl command to start investigating. The full output for this environment: Plain Text Incident Score: 41/100 (Degraded) Top 5 Things to Fix: 1. 4 pods crash-looping CRITICAL 4 pods 2. 3 image_pull_backoff issues CRITICAL 3 items 3. 1 privileged_container issue CRITICAL 1 item 4. 1 namespace missing NetworkPolicy HIGH 1 ns 5. 3 orphaned PVCs wasting money MEDIUM 80 GB None of these had triggered an alert. All were present and accumulating before the scan. The Incident Score — a composite 0–100 across reliability, security, and waste — exists for one reason. Engineers fix incidents. Managers remember numbers. "We moved the Incident Score from 41 to 67" is a sentence that sticks. The crash loops and NetworkPolicies are the work behind it. The Step After Detection Finding problems was never the hard part. Knowing where to begin was. The most common feedback on the original bash script was some version of: "I found the problem, but I still didn't know what to do next." In March, I wrote about finding a container with 24,069 restarts that had been accumulating undetected. Finding it took sixty seconds. The next hour was the actual work: what do I run first? Is this configuration or code? Is it customer-facing? The investigation page is my answer to that hour. Investigation page — OpsCart Assessment, Evidence, Recommended Investigation One click from any triage finding opens a dedicated investigation view: Plain Text OpsCart Assessment This workload has restarted 1810 times over 6 days. The restart rate appears stable, suggesting a deterministic configuration or application failure rather than an intermittent infrastructure issue. No referenced ConfigMaps or Secrets were detected in the pod spec — missing configuration is unlikely to be the root cause. Investigation should begin with previous container logs. Estimated time: 5–10 minutes. Evidence [1810 Restarts] [CrashLoopBackOff] [6d] [Deployment/fraud-detection] Recommended Investigation HIGH CONFIDENCE Check previous container logs MEDIUM Verify ConfigMaps and Secrets exist LOW Check for OOMKill in events The assessment is rules-based — no AI. It reads restart count, failure pattern (stable vs accelerating), and referenced configuration objects, then produces a deterministic, auditable summary. The confidence levels reflect how a senior engineer actually reasons: previous logs are almost always the right first move for a crash loop; OOMKill is worth checking but less likely. This is the part kubectl doesn't give you. Neither does Lens, k9s, or Headlamp. From "What Is Broken?" to "What Changed?" The biggest architectural change came when the dashboard gained memory. The first version of the tool answered: "what is broken?" The current version — backed by a small embedded database recording every scan — answers "what changed?" That sounds like a minor distinction. Operationally, it changes everything. An incident that has existed for three days deserves different attention than one that appeared five minutes ago. A cluster whose Incident Score dropped eight points overnight is telling you something that no single scan can. War Room — critical issues with visual differentiation per type Every KPI now carries a trend arrow — critical issues up three since the last scan, waste down one — and the Incident Score shows a seven-point sparkline. Each incident is tracked with first-seen and last-seen timestamps and an active/resolved status, so "CrashLoopBackOff — first detected 6 days ago, still active" replaces "CrashLoopBackOff." Operational memory changed the tool from a scanner into something that remembers the history of a cluster. What This Is Not The triage pattern does not answer when an issue started at the metrics level, why an application is slow, or whether last Tuesday's deployment caused a regression. Prometheus, APM tooling, and deployment audit logs remain the right tools for those questions. The triage layer is not a replacement for observability. It is the layer that tells you which questions to ask of your observability stack. The Biggest Lesson When I started, I thought Kubernetes debugging was about collecting more information. It wasn't. Kubernetes already exposes almost everything an operator needs through its API. The difficult part is deciding what deserves attention first. Over eight months, I found myself spending less time searching for failures and more time ranking them. That is ultimately what OpsCart became — not another dashboard, but a prioritization engine for cluster operations. Why Open Source I considered keeping the dashboard private. Instead, I open-sourced it because operational patterns only become useful when they're tested across different clusters. Every environment fails differently, and I wanted the prioritization model to evolve from real-world feedback rather than a single infrastructure. The Remaining Gap The conclusion from my March article is still true: the question worth asking of your environment is not whether these conditions exist — they almost certainly do — but whether your current observability layer would surface them before they become incident preconditions. Eight months of building has only made that conclusion more specific. The gap is not data. The gap is attention: knowing which five things, out of hundreds of resources, deserve a human's time right now. Eight months ago I thought I was building a better debugging script. I wasn't. I was building something that helps operators decide where to spend the next ten minutes. About the environment: The scenarios shown in this article — CrashLoopBackOff pods, orphaned PVCs, missing NetworkPolicies, privileged containers — are representative of what OpsCart finds on real production clusters. The environment shown is a dedicated demonstration cluster configured with realistic failure scenarios. No production data was used. About the tool: OpsCart Watcher is open-source at github.com/opscart/opscart-k8s-watcher. It deploys as a single read-only container: Shell kubectl apply -f https://raw.githubusercontent.com/opscart/opscart-k8s-watcher/main/deploy/dashboard.yaml kubectl port-forward -n opscart-system svc/opscart-watcher 8080:80

By Shamsher Khan

CORE

Getting Started With RabbitMQ in Spring Boot

RabbitMQ is an enterprise-grade open-source messaging and streaming broker. In this blog, you will learn some basic concepts of RabbitMQ and how to use it in a Spring Boot application. Enjoy! Introduction Before diving into the programmatic details, first some concepts need to be explained. Do realize that in this blog, only the surface is scratched from what is possible with RabbitMQ. A detailed overview can be found in the official RabbitMQ documentation. Several protocols are supported by RabbitMQ. In this blog, the AMQP 0-9-1 protocol will be used. AMQP stands for Advanced Message Queuing Protocol. RabbitMQ receives messages from a publisher, a producing application, and routes them to consumers, applications that process the messages. A publisher publishes messages to an exchange (like a mailbox). The exchange then routes the messages to queues using bindings. RabbitMQ then delivers the messages to the consumers who are subscribed to the queues. The process is shown in the figure below. In the examples in the remainder of this blog, you will make use of a Topic Exchange. There are different exchange types, but for the sake of simplicity, only one will be used. A topic exchange routes messages to one or many queues, based on a message routing key. Topic exchanges are commonly used for multicast routing of messages. Sources used in this blog are available on GitHub in module topics. Prerequisites Prerequisites for reading this blog are: Basic knowledge of Java;Basic knowledge of Spring Boot;Basic knowledge of Docker Compose. Create Spring Boot Application In order to get started, you navigate to the Spring Initializr and add the following dependencies: Spring Web: in order to be able to send messages via an http request.Docker Compose Support: in order to start a RabbitMQ container when the application starts.Spring for RabbitMQ: in order to integrate Spring Boot with RabbitMQ. You will build the following: One Exchange with one Topic.Publish a general message to the topic which will be consumed by consumer A and consumer B.Publish a specific message to the topic which will be only consumed by consumer B. In order to send a general and a specific message, two HTTP endpoints are created in the MessageController. Java @RestController public class MessageController { private MessageService messageService; public MessageController(MessageService messageService) { this.messageService = messageService; } @RequestMapping( method = RequestMethod.POST, value = "send-general" ) public ResponseEntity<Void> sendGeneralMessage(@RequestBody String message) { messageService.sendMessage("event.general.message", message); return new ResponseEntity<>(HttpStatus.CREATED); } @RequestMapping( method = RequestMethod.POST, value = "send-specific" ) public ResponseEntity<Void> sendSpecificMessage(@RequestBody String message) { messageService.sendMessage("event.specific.message", message); return new ResponseEntity<>(HttpStatus.CREATED); } } The requests are forwarded to a MessageService.sendMessage method, which takes a routingKey and the message as arguments. The message is taken from the http request body, the routingKey is hardcoded. Remember that the routingKey determines to which queue the message will be routed. In the service, you make use of Spring Boot's RabbitTemplate in order to send the message to RabbitMQ. Java @Service public class MessageService { private RabbitTemplate rabbitTemplate; public MessageService(RabbitTemplate rabbitTemplate) { this.rabbitTemplate = rabbitTemplate; } public void sendMessage(String routingKey, String message) { rabbitTemplate.convertAndSend(RabbitMqConfig.TOPIC_EXCHANGE_NAME, routingKey, message); } } Bind Consumer A Consumer A will consume general messages. The queue needs to be bound to the Topic Exchange with the routing key. Create a RabbitMqConfig class with: A TopicExchange bean with name events.exchange.A Queue bean for consumer A with name consumer-a.queue.A binding bean for consumer A connecting the queue of consumer A to the TopicExchange with the routing key for the general messages. Do note that the name of the queue in method bindingConsumerA needs to match the queueConsumerA bean name. Java Configuration public class RabbitMqConfig { public static final String QUEUE_CONSUMER_A = "consumer-a.queue"; public static final String TOPIC_EXCHANGE_NAME = "events.exchange"; public static final String ROUTING_KEY_GENERAL_MESSAGE = "event.general.*"; @Bean TopicExchange eventsExchange() { return new TopicExchange(TOPIC_EXCHANGE_NAME); } @Bean public Queue queueConsumerA() { return new Queue(QUEUE_CONSUMER_A, false); } @Bean Binding bindingConsumerA(Queue queueConsumerA, TopicExchange exchange) { return BindingBuilder.bind(queueConsumerA).to(exchange).with(ROUTING_KEY_GENERAL_MESSAGE); } } Create Consumer A Next thing to do is to consume the messages from queue A. Create a Component named ReceiverA. Annotate the method for processing the messages with @RabbitListener and connect it to queue A. When receiving the message, just print it to the console. Java @Component public class ReceiverA { @RabbitListener(queues = RabbitMqConfig.QUEUE_CONSUMER_A) public void receiveMessage(String message) { System.out.println("Queue Consumer A received <" + message + ">"); } } Run the Application In order to run the application, you will need RabbitMQ. Since you have added Docker Compose Support to the project earlier, you can just add a compose.yaml in the root of the repository. YAML services: rabbitmq: image: rabbitmq:3.13-management-alpine # Stable, lightweight, includes management UI container_name: rabbitmq ports: - "5672:5672" # AMQP - "15672:15672" # Management console environment: RABBITMQ_DEFAULT_USER: secret RABBITMQ_DEFAULT_PASS: myuser Also add the connection parameters for RabbitMQ to the application.properties file. Properties files spring.rabbitmq.host=localhost spring.rabbitmq.port=5672 spring.rabbitmq.username=secret spring.rabbitmq.password=myuser Start the application. Shell mvn spring-boot:run You will notice that RabbitMQ is started automatically. Send a general message. Shell curl -X POST http://localhost:8080/send-general \ -H "Content-Type: text/plain" \ -d "This is a general message" The console log will print the following. Plain Text Queue Consumer A received <This is a general message> Stop the application. Bind Consumer B Consumer B will process general messages, but also specific messages. Add to the RabbitMqConfig the queue for consumer B, and bind it to the exchange with respectively the general message routing key and the specific message routing key. Java @Configuration public class RabbitMqConfig { public static final String QUEUE_CONSUMER_A = "consumer-a.queue"; public static final String QUEUE_CONSUMER_B = "consumer-b.queue"; public static final String TOPIC_EXCHANGE_NAME = "events.exchange"; public static final String ROUTING_KEY_GENERAL_MESSAGE = "event.general.*"; public static final String ROUTING_KEY_SPECIFIC_MESSAGE = "event.specific.*"; ... @Bean public Queue queueConsumerB() { return new Queue(QUEUE_CONSUMER_B, false); } @Bean Binding bindingConsumerBGeneral(Queue queueConsumerB, TopicExchange exchange) { return BindingBuilder.bind(queueConsumerB).to(exchange).with(ROUTING_KEY_GENERAL_MESSAGE); } @Bean Binding bindingConsumerBSpecific(Queue queueConsumerB, TopicExchange exchange) { return BindingBuilder.bind(queueConsumerB).to(exchange).with(ROUTING_KEY_SPECIFIC_MESSAGE); } } Create Consumer B Consumer B is created just like consumer A. Create a ReceiverB class in order to receive the queue B messages. Java @Component public class ReceiverB { @RabbitListener(queues = RabbitMqConfig.QUEUE_CONSUMER_B) public void receiveMessage(String message) { System.out.println("Queue Consumer B received <" + message + ">"); } } Run the Application Start the application. Shell mvn spring-boot:run Send a general message. Shell curl -X POST http://localhost:8080/send-general \ -H "Content-Type: text/plain" \ -d "This is a general message" The message is now received by Consumer A and Consumer B. Plain Text Queue Consumer B received <This is a general message> Queue Consumer A received <This is a general message> Send a specific message. Shell curl -X POST http://localhost:8080/send-specific \ -H "Content-Type: text/plain" \ -d "This is a specific message" The message is only received by Consumer B. Plain Text Queue Consumer B received <This is a specific message> Management Console Also take a look at the RabbitMQ management console, which is accessible at http://localhost:15672/. Here you can see the exchanges, the queues, the bindings, etc. Conclusion In this blog, you learned some basics of RabbitMQ using the AMQP 0-9-1 protocol. You learned how easy it is to integrate this within your Spring Boot application.

By Gunter Rotsaert

CORE

Architecting Trustworthy AI: Engineering Patterns for High-Stakes Environments

Software engineers are great at talking about how to make things dependable: we have names for things like circuit breakers, bulkheads, making sure something can be done multiple times with the same result (idempotency), and making sure a system will fall apart in a manageable way (graceful degradation). We're good at building systems to fail safely if a database isn't working or a dependent service is taking too long. However, we don't have a comparable, well-defined way of discussing systems that fail in a way that is likely to be wrong, and where the kind of failure isn't a network going down, but a prediction that is confidently, but incorrectly, given to you. And what’s more, it’s up to a person to find that mistake, and that person might not even be looking. That gap is the engineering problem this article addresses. Drawing on a production sociotechnical framework for AI-enabled platforms operating in high-stakes environments, the patterns below are directly applicable to any system where incorrect outputs carry significant consequences - financial services, legal tech, autonomous systems, or safety-critical infrastructure. The specific domain is illustrative; the engineering is universal. Why Probabilistic Systems Need a Different Reliability Model Traditional, predictable systems fail loudly. A null pointer exception, a 500 error, something that should always be true not being true, these are all visible, can be recorded, and can be caught by the usual ways we keep an eye on things. Probabilistic AI models fail silently. They confidently continue to generate outputs even when they are outside their training distribution. They don't raise exceptions when the data distribution shifts. They don't emit warnings when their predictions get worse. They just start making mistakes at a large scale, invisibly. This leads to three failure cases that deterministic patterns don't cover: Distribution shift – The statistical characteristics of the input data change over time. A model trained on a certain type of data or property will, without warning, use what it has learned on a completely different set of data. The model's confidence scores remain unchanged.Reasoning drift in agentic pipelines – In multi-step AI workflows, errors at step N are used as the ground truth for step N+1. Long-horizon agentic tasks have been shown to result in over 20-30% difference in output across a multi-step pipeline, even when the individual steps work. This is the AI version of a cascading failure in a distributed system - but more challenging to identify because there is no "system failure" or no failure of a "single component".Automation bias as a system property – When AI systems start off by performing well, human supervisors progressively relax their level of oversight. This is not a weakness; it's a predictable natural system behavior. This is to say that when AI does fail, human overseers are temporarily blind to the failure because of their own prior trust. The engineering challenge is to build systems that remain safe even when any of the above three failure modes are active. Pattern 1: The Safety Shell The Safety Shell is the most critical architectural pattern for high-stakes AI applications. It's a deterministic, rule-based layer that wraps the probabilistic model and enforces invariants the model cannot guarantee on its own. One can think of this as the Microsoft Gatekeeper Pattern, applied specifically to ML inference. The three fail modes map directly to the circuit breaker pattern: Python from enum import Enum class FailMode(Enum): SAFE = "block_and_alert" DEGRADED = "rule_based_fallback" OPERATIONAL = "switch_to_backup" class SafetyShell: def __init__(self, model, backup_model, rule_engine, drift_monitor): self.model = model self.backup_model = backup_model self.rule_engine = rule_engine self.drift_monitor = drift_monitor def evaluate(self, input_data): # Layer 1: Input validation — check schema and distribution if not self.rule_engine.is_in_distribution(input_data): self.drift_monitor.record(input_data) return {"mode": FailMode.DEGRADED, "output": self.rule_engine.fallback(input_data)} # Layer 2: Probabilistic model inference output = self.model.predict(input_data) # Layer 3: Hard constraint enforcement (deterministic) violation = self.rule_engine.check_constraints(output) if violation: return {"mode": FailMode.SAFE, "output": None, "alert": f"Constraint violated: {violation}"} # Layer 4: Confidence threshold if output.confidence < 0.70: return {"mode": FailMode.DEGRADED, "output": self.rule_engine.fallback(input_data)} # Layer 5: Drift-triggered failover if self.drift_monitor.is_drifting(self.model): return {"mode": FailMode.OPERATIONAL, "output": self.backup_model.predict(input_data)} return {"mode": None, "output": output} # All clear The key insight is that the model does not know when it's wrong. The shell does. These are fundamentally different capabilities. Pattern 2: Uncertainty Quantification via Conformal Prediction The typical output of an AI system is a single scalar — a probability, a score, a classification label. Such confidence scores (e.g., "87% sure") are often poorly calibrated. This is an engineering tradeoff that leads to a reliability issue: it masks uncertainty in the model with a precise-looking number. High-stakes systems need to output uncertainty as a primary output, not a derived metric. Conformal prediction is particularly noteworthy because it's efficient and provides coverage guarantees for its predictions, which no traditional ML model provides. Rather than "class A (87% confidence)", a conformal predictor returns a set: {"class A", "class B"} with a guaranteed error rate of ≤5% across the test distribution. This is more valuable for systems with human reviewers, as it leaves the reviewer to know what the model cannot reliably decide between, rather than making a binary decision. Python from nonconformist.cp import IcpClassifier from nonconformist.nc import ClassifierNc, MarginErrFunc # Initialize with a 95% confidence requirement nc = ClassifierNc(base_model, MarginErrFunc()) icp = IcpClassifier(nc) icp.calibrate(X_cal, y_cal) # Returns plausible classes rather than a single 'best guess' prediction_sets = icp.predict(X_test, significance=0.05) Pattern 3: Multi-Agent Quality Control With the emergence of multi-agent pipelines in AI systems, a new pattern of reliability emerges: multi-agent quality control. The design does not assume the output of one agent to be the truth for the next agent: it explicitly allocates some agents to audit other agents. This pattern prevents single-point failures from propagating through the pipeline and creates explicit audit trails at every agent boundary. Prevention by design: The most reliable agentic systems don't just leave it up to the Safety Officer agent to catch every error; they restrict the capabilities of the other agents. Narrow the action space. Limit the damage. Let the Safety Officer deal with edge cases, not core invariants. Patterns in Action: A Case Study It's easier to grasp abstract patterns when they're applied. Take a smart power grid load balancer with an AI controller: A sensor fault is rated negative energy consumption, which is physically impossible.In the absence of a Safety Shell, the model attempts to correct the anomaly by surging power to one of the transformers.Pattern 1 becomes active: The input validation layer identifies the out-of-distribution reading; the constraint engine identifies the negative consumption value as a physical invariant violation -> FailMode.SAFE triggers, prevents the output, and signals the human operator.Pattern 2 context: The ambiguity of the model was high on this input — a conformal predictor would have encountered the ambiguity even prior to the Safety Shell's firing, giving the operator an earlier warning signal.Pattern 3 occurs: The Safety Officer agent records the inter-agent disagreement, including the exact model version, sensor snapshot, and constraint that fired, giving the ops team a complete reconstruction trail with no human investigation. Since the system is running under a phased autonomy (10% human audit mode), the blocked action routes to a human operator to verify, who rightly says no to the surge command.The override signal is fed back into the retraining queue, and a sensor recalibration ticket is automatically created. The same sequence would unfold identically whether the domain was a power grid, a fraud-detecting engine, or an autonomous vehicle routing system. The patterns are domain-independent; the definition of constraints is modified. This is also the reason why phased autonomy is used to provide a documented track record that governance stakeholders and regulators must be able to view before granting further autonomy - the audit trail is not just to be debugged with, but the evidence of safe operation. Measuring Reliability Beyond Accuracy High-stakes AI production monitoring should measure these metrics in addition to the traditional model performance: Metric What It Catches Implementation Feature distribution drift Silent failure from data changes KL-divergence or PSI on rolling windows Override rate Automation trust gap Log every human-AI disagreement event Expected Calibration Error (ECE) Overconfident wrong predictions Reliability diagrams on production data Latency p99 Dangerous delays in time-sensitive contexts Standard APM tooling Constraint violation rate Safety Shell effectiveness Event counter in M2M audit log Inter-agent disagreement rate Multi-agent pipeline health Logged at Safety Officer agent boundary The override rate deserves special attention: this is the direct outcome of behavioral trust. When operators consistently reject AI outputs, the system is providing no value regardless of accuracy. Transparent and well-calibrated production systems are likely to have an override rate of ~15%; systems in which the output is opaque are likely to have an override rate of ~74%. Monitor this number on a weekly basis. The Reliability Maturity Model Use this to assess where your system sits and what to build next: Level Label Characteristics Implementation Focus 1 Naive No monitoring. Vendor explanations accepted at face value. Incidents drive all improvements. Establish baseline logging 2 Guarded Safety Shell active. Output constraints enforced. Fails safely on violations. Accuracy dashboards live. Implement Pattern 1 + drift alerts 3 Sociotechnical FAME framework active. Backup models. HITL feedback loops. M2M audit trail. Phased autonomy rollout. Implement Patterns 2-3 4 Verifiable Formal verification of safety properties. Continuous adversarial testing. System improves automatically from human overrides. Add formal methods + red-teaming The majority of the production AI systems sit at Level 1 or 2. To reach Level 2 — the minimum possible safety posture in high-stakes settings — it is necessary to introduce the Safety Shell and specify the concrete fail modes. That work is scoped and can be accomplished without retraining the underlying model. The Pre-Ship Engineering Checklist Before deploying to a high-stakes AI system: Safety Shell implemented with fail-safe, fail-degraded, and fail-operational modes defined and tested.Uncertainty measured, quantified, and brought to downstream consumers (conformal prediction or ensemble disagreement)M2M audit logging captures all inter-agent interactions with data version, model version, and constraint check results.Feature distribution monitoring notifies prior to accuracy deterioration rather than after.Documented phased autonomy plan with override rate thresholds and automated rollback triggers that are defined in advance.Override rate instrumented and monitored as a primary reliability measure (target: <15% to get high-confidence results)Expected Calibration Error (ECE) Calculated on Production data, not only on test sets.Rollback procedures tested and documented prior to go-live, not following an incident. Conclusion The probabilistic AI systems do not fail in the same manner that deterministic software fails. They need a different class of reliability engineering — that which assumes the model will sometimes be incorrect, confidently, and provides the surrounding system to detect it. The above patterns are a starting point for that vocabulary. The field is in its infancy, but the trend is clear: one should not trust AI, which would merely be a model property; it is a system property. Build the system in that manner.

By Sujay Puvvadi

Resilience Lost in the Stack: How Abstraction Layers Silently Mask Distributed Systems’ Topology Awareness

Distributed coordination services exist for a reason, and they are the CPUs of distributed systems that give them their high availability. When it's in your stack, you assume failover is handled. Some services that operate in this layer include Apache Zookeeper, Redis Sentinel, etcd, etc. These services are mathematically engineered for HA. Protocols such as Raft/Paxos/ZAB guarantee this. We know that the DCS itself cannot go wrong as long as a quorum of nodes exists. Here, we want to explore one specific problem that makes this high availability subjective. It is an issue where individual layers hold this promise, while as we go to higher-level abstractions, the intelligence silently dies. The article focuses on how topology awareness needs to be preserved mindfully as we move up the stack, and that, when using smart clients and drivers, we should inherit the responsibility not to silence their intelligence. We take a use case in the Java ecosystem. In a production-grade system, we have microservices distributed across multiple regions that take care of specific components. We discuss a rate-limiting use-case here to demonstrate the underlying problem. This is one area where the problem manifests itself. A similar architecture can still have the same problem under the hood. Deconstructing the Stack The use case: A rate limiter that is implemented in production-grade using Bucket4j. A distributed rate limiter needs to keep track of token buckets across multiple instances. We assume the application runs in 3 instances and want to rate-limit requests to 5 req/sec. To enforce this strict global token limit across 3 instances, a centralized state coordinator becomes mandatory. So we introduce Redis, which centrally stores the token bucket state. This way, we decouple the bucket state from application instances and do not make every instance accept 5 tokens each, making it 15 req/sec. Redis is a distributed caching layer. In Java, if you're using Redis, you're likely talking to it through Lettuce, Redisson, or Jedis. Since Lettuce is the widely adopted Redis client, it acts as the asynchronous engine that bridges the Java application layer with the Redis database infrastructure. A sample boilerplate for the connection configuration would look something like: Java @Bean public ProxyManager<String> proxyManager() { RedisURI.Builder builder = RedisURI.builder().withSentinelMasterId(masterId).withTimeout(Duration.ofSeconds(12));; for (String node : nodes) { String[] hostPort = node.split(":"); builder.withSentinel(hostPort[0], Integer.parseInt(hostPort[1])); } RedisClient redisClient = RedisClient.create(builder.build()); RedisCodec<String, byte[]> bucket4jCodec = RedisCodec.of(StringCodec.UTF8, ByteArrayCodec.INSTANCE); StatefulRedisConnection<String, byte[]> redisConnection = redisClient.connect(bucket4jCodec); return LettuceBasedProxyManager.builderFor(redisConnection) .withClientSideConfig(ClientSideConfig.getDefault()) .build(); } This code looks straightforward in terms of establishing the connection. We build the Redis cluster nodes along with the Sentinel configuration.Feed this to the Redis client and plug this code into the Bucket4j instance.Create a StatefulRedisConnection wrapper available as part of the Bucket4j library.Configure a LettuceProxyManager that interacts with the Redis cluster. This code complies properly, passes all integration tests, and handles throttling by rate limiting, yet silently harbors a topology blindness that will cause the application to stall during a failover. The Failure Mode: What It Looks Like At the time of failover, when the primary Redis master suddenly drops, the Redis sentinel infrastructure kicks in and promotes a healthy replica to master. It does so by holding a rapid quorum vote, and high availability is achieved. But at the application side, we experience the following: Thread Stalls and Application Freeze The app does not throw any obvious connection errors; instead, it just freezes and waits indefinitely for Redis to recover. The executing thread stalls in the process and never recovers. On the application side, Bucket4j continuously starts receiving requests, and its internal token buckets are routed to a broken Redis connection. The Illusion of Network Failure Internally, Lettuce has methods and wrappers to handle this type of failure, and it transparently buffers and queues commands while trying to reconnect. But Bucket4j does not have this information and keeps waiting for Lettuce. Even a generous timeout from the application side is not going to help us in this situation. The Rate Limiter Paradox While some threads are stalled, other parts of the application may continue; for instance, the endpoint might still receive requests and route them to Bucket4j, but because this exception is swallowed, no actual rate limiting occurs. Bucket4j keeps talking to a broken connection and doesn't ideally keep track of tokens. The Wrapper Deficit While Lettuce does have a StatefulMasterReplicaConnection that comes with topology awareness, Bucket4j never exposes wrappers to use this StatefulMasterReplicaConnection. So a user using Bucket4j may or may not be aware of internal wrappers available in Lettuce. In the case where this is not known to the user, engineers naturally instantiate static connections and can easily overlook this. This results in code that seems to handle failovers but is completely devoid of master-replica awareness. System architecture with the topology awareness blindspot The Abstraction Blindspot This becomes hard to catch at some level precisely because every layer seems to work and does its job correctly. Redis-Sentinel experiences a failover and successfully recovers. Meanwhile, lettuce, the client that interacts with Redis, also has a MasterReplicaConnections that is capable of knowing that this event has occurred. Bucket 4j is responsible for token buckets and rate limiting, and it is also doing its job well. The problem happens in the composition. This brings us to a broader principle: abstraction layers do not just simplify complexity but also inadvertently suppress capabilities silently. The HA awareness exactly gets broken at this point in the stack Why did testing not expose this? Unit tests almost always don't uncover these types of errors. Load/Performance focus on high throughput under normal conditions to ensure the rate limiter is functioning correctly and is handling throttling.Health checks and readiness probes target the wrong layer, namely Redis, in ensuring availability. The Solution Blueprint To solve the thread stall and force the application layer to inherit The High Availability of Redis, we have to preserve the topology at every layer of the stack. The fix requires choosing the exact Lettuce connection interface that tracks topology shifts while preserving the raw command execution engine. Navigating Lettuce’s Topology refresh options connecTion interfacetarget redis infrause case StatefulRedisConnection Standalone Node General single-node use; blind to topology changes. StatefulRedisClusterConnection Sharded Cluster Data partitioning across many nodes. StatefulRedisPubSubConnection Messaging Channels Real-time pub/sub event listening. StatefulRedisSentinelConnection Sentinel Nodes Directly Administrative tracking and master discovery. StatefulRedisMasterReplicaConnection Primary + Replicas via Sentinel Dynamic health tracking, automatic failover, and read/write splitting. Why We Choose StatefulRedisMasterReplicaConnection over StatefulRedisSentinelConnection When working with Redis and when the need is to explicitly inherit Sentinel properties, the intuitive solution is to reach StatefulRedisSentinelConnection. However, a closer look at the source code for RedisSentinelConnection extends the basic StatefulConnection interface, and its async command blocks only expose Sentinel APIs. Java public interface StatefulRedisSentinelConnection<K, V> extends StatefulConnection<K, V> { RedisSentinelAsyncCommands<K, V> async(); } // Sneak peek inside RedisSentinelAsyncCommands: RedisFuture<List<Map<K, V>>> slaves(K key); RedisFuture<String> failover(K key); RedisFuture<String> monitor(K key, String ip, int port, int quorum); RedisFuture<Long> reset(K key); A point to note here is that the standard data manipulation commands like GET, SET, HGET are absent in this abstraction. We would require evaluation scripts for the wrapping client (Bucket4j) to execute Lua scripts. This shows that the interface was built to manage the cluster but not to read and write app data. On the other hand, StatefulRedisMasterSlaveConnection directly extends StatefulRedisConnection, inheriting the complete data manipulation layer. Java public interface StatefulRedisMasterReplicaConnection<K, V> extends StatefulRedisConnection<K, V> { void setReadFrom(ReadFrom readFrom); RedisAsyncCommands<K, V> async(); // Exposes GET, SET } By choosing StatefulRedisMasterReplicaConnection instantiated via a Sentinel-backed MasterReplica builder, we inherit: Topology awareness: As it hooks directly into the Sentinel Pub/Sub event stream to automatically reroute trafficAsynchronous engine preservation: Which exposes the RedisAsyncCommands necessary for Bucket4j to asynchronously execute thread-safe token The Wrapper Integration To cleanly connect our new topology-aware connection with the rate-limiter in our example (Bucket4J), we introduce a dedicated wrapper to the integration layer. Java public static <K> LettuceBasedProxyManagerBuilder<K> casBasedBuilder(StatefulRedisMasterReplicaConnection<K, byte[]> statefulRedisMasterReplicaConnection) { return casBasedBuilder(statefulRedisMasterReplicaConnection.async()); } A word on CAS: Compare-And-Swap (CAS) is a builder that uses a non-blocking database pattern to update data safely without heavy locks. It reads the token bucket value, does the math, and writes it back only if another thread hasn't changed it in the meantime. If the value did change, it safely retries the operation automatically. Bucket4j exposes similar builders for CAS. This builder expects a standard Lettuce asynchronous command interface. By including the above builder, we enable Bucket4j’s proxy manager to accept a StatefulRedisMasterSlaveConnection. Validation Through a Little Chaos Engineering The Test Stack Since standard testing frameworks won’t expose this issue, we needed a real-world setup to simulate the production environment. The test stack included: Docker and Docker Compose: To manage a multi-node Redis cluster (1 Master, 2 replicas, 3 sentinels)Java/Springboot: The host-side sample application integrating the Bucket4j logic to rate limit an endpointLettuce/Bucket4j: The libraries that we want to test Apache Benchmark: A command-line utility used for injecting the load Test Setup The following docker-compose.yml served as the baseline configuration for the Redis setup. YAML version: '3.8' services: redis-master: image: redis:7-alpine container_name: redis-master # Network mode host maps directly to your machine's ports, bypassing docker bridge DNS network_mode: "host" command: redis-server --port 6379 redis-replica: image: redis:7-alpine container_name: redis-replica network_mode: "host" # Since we are on host mode, the replica connects to localhost 6379 and binds its own engine to 6380 command: > redis-server --port 6380 --replicaof 127.0.0.1 6379 --replica-announce-ip 127.0.0.1 --replica-announce-port 6380 depends_on: - redis-master redis-sentinel: image: redis:7-alpine container_name: redis-sentinel network_mode: "host" command: > sh -c " echo 'port 26379' > /sentinel.conf && echo 'sentinel monitor mymaster 127.0.0.1 6379 1' >> /sentinel.conf && echo 'sentinel down-after-milliseconds mymaster 3000' >> /sentinel.conf && echo 'sentinel failover-timeout mymaster 6000' >> /sentinel.conf && redis-server /sentinel.conf --sentinel " depends_on: - redis-master - redis-replica Here, we use a Redis master-replica configuration and a portable built-in sentinel.conf. This makes it easier and starts the service using the configs provided in the file. Architectural Parameters down-after-milliseconds mymaster 3000: Sentinel waits for 3 seconds before deciding the master is unreachable. The host is marked “subjectively down” (SDOWN) if the master does not continuously respond in this window.failover-timeout mymaster 6000: The window for the promotion of a new master and the reconfiguration of the cluster. At this point, it is marked “objectively down” (ODOWN) and starts the failover. Step 1: Validating the Infrastructure Baseline and Sanity Before testing how it performs with different abstractions, a standard failover was executed to see if redis-sentinel was working as expected and proceeded with the leader election. Shell docker stop redis-master The logs where Sentinel performs a leader election process: Shell redis-replica-1 | 1:S 17 May 2026 19:44:23.894 # Unable to connect to MASTER: Success sentinel-1 | 9:X 17 May 2026 19:44:24.780 # +sdown master mymaster redis-master 6379 sentinel-1 | 9:X 17 May 2026 19:44:24.780 # +odown master mymaster redis-master 6379 #quorum 1/1 sentinel-1 | 9:X 17 May 2026 19:44:24.780 # +try-failover master mymaster redis-master 6379 sentinel-1 | 9:X 17 May 2026 19:44:24.785 # +vote-for-leader e1db4435d0770f294d0d13835729c5102cb5a4cd 1 sentinel-1 | 9:X 17 May 2026 19:44:24.785 # +elected-leader master mymaster redis-master 6379 sentinel-1 | 9:X 17 May 2026 19:44:24.785 # +failover-state-select-slave master mymaster redis-master 6379 sentinel-1 | 9:X 17 May 2026 19:44:24.850 # +selected-slave slave 172.18.0.3:6379 172.18.0.3 6379 @ mymaster redis-master 6379 sentinel-1 | 9:X 17 May 2026 19:44:24.850 * +failover-state-send-slaveof-noone slave 172.18.0.3:6379 172.18.0.3 6379 @ mymaster redis-master 6379 Step 2: The Naive Connection and Indefinite Freeze We now use the StatefulRedisConnection and expose a test endpoint from our SpringBoot application. This endpoint is now sent 10,000 concurrent requests, and mid-stream we kill the master node to allow for the re-election of the master. Java @GetMapping("/test") public ResponseEntity<String> handleRequest() { Bucket bucket = proxyManager.builder().build("reproduction-key", () -> bucketConfiguration); // Under high traffic concurrent loads, this execution point will freeze solid // the moment the master container is stopped! if (bucket.tryConsume(1)) { return ResponseEntity.ok("SUCCESS"); } else { return ResponseEntity.status(HttpStatus.TOO_MANY_REQUESTS).body("RATE_LIMITED"); } Observation: The application logs revealed a blind reconnection loop. The client was stuck knocking on the door of the dead port 6379, oblivious to the new Master on 638. The infrastructure healed at this point; however, the application remained in an indefinite freeze. Even when we increased the command timeout to 12 seconds, the results were the same. The Apache Benchmark test did not complete and timed out. Below are the results: Plain Text rithraravikumar@Rithras-MacBook-Air redis-sentinel-lab % ab -n 10000 -c 10 http://localhost:8080/test This is ApacheBench, Version 2.3 <$Revision: 1903618 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking localhost (be patient) Completed 1000 requests Completed 2000 requests Completed 3000 requests Completed 4000 requests Completed 5000 requests apr_pollset_poll: The timeout specified has expired (70007) Total of 5460 requests completed Step 3: The Master-Replica Connection Pivot The final step of the process was to use the MasterReplicaStatefulConnection and observe if the application bounced back. The code change looks like the below: Java RedisClient redisClient = RedisClient.create(builder.build()); RedisCodec<String, byte[]> bucket4jCodec = RedisCodec.of(StringCodec.UTF8, ByteArrayCodec.INSTANCE); RedisURI sentinelUri = RedisURI.builder() .withSentinelMasterId(masterId) // Looks up "mymaster" .withSentinel("127.0.0.1", 26379) // The host and port where Sentinel is listening .build(); StatefulRedisMasterReplicaConnection<String, byte[]> redisConnection = MasterReplica.connect(redisClient, bucket4jCodec, sentinelUri); return LettuceBasedProxyManager.builderFor(redisConnection) .withClientSideConfig(ClientSideConfig.getDefault()) .build(); Observation: With this topology-aware connection, when we killed the master node at the 5,000-request mark. While there was a noticeable stall, the connection was able to recognize there was a new master and resumed processing. Plain Text rithraravikumar@Rithras-MacBook-Air redis-sentinel-lab % ab -n 10000 -c 10 http://localhost:8080/test This is ApacheBench, Version 2.3 <$Revision: 1903618 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking localhost (be patient) Completed 1000 requests Completed 2000 requests Completed 3000 requests Completed 4000 requests Completed 5000 requests .... Completed 6000 requests Completed 7000 requests Completed 8000 requests Completed 9000 requests Completed 10000 requests Finished 10000 requests Time taken for tests: 24.522 seconds Complete requests: 10000 Failed requests: 5983 (Connect: 0, Receive: 0, Length: 5983, Exceptions: 0) Non-2xx responses: 5983 Total transferred: 1814793 bytes HTML transferred: 656334 bytes Requests per second: 407.80 [#/sec] (mean) Time per request: 24.522 [ms] (mean) Time per request: 2.452 [ms] (mean, across all concurrent requests) Transfer rate: 72.27 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 2.5 0 251 Processing: 0 24 671.0 1 21235 Waiting: 0 24 671.0 1 21234 Total: 0 24 671.0 1 21235 Analyzing the Metrics The test recorded 5,983 failed requests (non-2xx responses). Those 5,983 failures represent the exact window of time when the master went down, and Bucket4j rejected traffic or failed fast instead of hanging. The longest request took a massive 21.2 seconds (21235 ms) to process. The critical difference was that the application bounced back. The Bigger Lessons — Not Just Topology Framework abstractions can inadvertently mask underlying driver capabilities, turning a standard infrastructure-level database failover into an application-level thread stall.When configuring stateful caching or synchronization wrappers, developers must ensure that connection pools explicitly utilize master-replica topology providers rather than static endpoints.High-availability verification cannot rely on static environment tests; engineers must actively simulate node terminations under concurrent load using tools like Apache Benchmark to uncover edge-case race conditions.Designing frameworks with extensible builder patterns allows downstream developers to inject custom infrastructure topologies without altering the core business logic of the library.Generous timeouts during a network failure are not an effective strategy if your client driver is blind to network routing shifts, as it merely forces application threads to spend more time waiting on a dead address.Under high-concurrency workloads, a client's internal memory buffer will saturate within milliseconds, rapidly cascading into total application thread pool exhaustion.

By Rithra Ravikumar

Building Production-Safe Agentic Remediation With Docker MCP Gateway: Lessons From 43% to 100% Accuracy

Our first version was wrong 57% of the time. Not because the AI model couldn't identify Docker container failure scenarios—it usually could. The failures occurred at the decision boundary: determining when an automated action was appropriate, when escalation was required, and when no action should be taken. Over several weeks, we built and evaluated an AI-assisted remediation system on Docker MCP Gateway across four container failure scenarios, improving decision correctness from 43% to 100%. What we learned surprised us: the hard problem is not teaching the agent to act. The hard problem is defining and enforcing the boundary where the agent must stop acting. The project reinforced a broader lesson: production-safe AI is less about model intelligence and more about engineering explicit policies, validation mechanisms, and execution controls. This article covers what we built, what failed, and the engineering changes that improved correctness. The full code, audit logs, validation datasets, and analyzer scripts are all in the companion repository. Why Naive Auto-Remediation Is Dangerous The most common mistake in AI-driven operations is treating "AI can fix things" as the goal. It isn't. A remediation system that attempts to fix every incident automatically is often worse than having no automation at all. Consider the failure modes: An automatic restart of a CrashLoopBackOff container does not fix the underlying problem—it simply generates more alerts. The container will fail again because the code or configuration issue remains unchanged. The result is additional operational noise without any meaningful remediation. Automatically increasing memory limits for every OOM event can be equally problematic. The workload continues running, but the underlying memory leak remains hidden. Months later, teams may find themselves running multi-gigabyte containers that should have been consuming a fraction of those resources. Automated remediation without an audit trail creates a different problem: a lack of accountability. Without structured records, it becomes impossible to determine what actions were taken, what actions were considered, and why a particular remediation path was selected. "The AI fixed it" is not a useful postmortem entry. The safest remediation systems are not the ones that automate the most actions. They are the ones with clearly defined operational boundaries, explicit escalation rules, and auditable decision paths. The engineering challenge is not maximizing automation — it is determining where automation should stop. According to Mohammad-Ali A'râbi, Docker Captain: One of the most dangerous assumptions teams can make is treating a language model as if it were an experienced senior site reliability engineer. It is not. A language model may generate useful recommendations, but it has no operational accountability. It does not understand business context, service ownership, deployment history, or the downstream consequences of an action. Any system granted the ability to modify production infrastructure must therefore be treated as an untrusted component operating behind strict controls. The container ecosystem learned this lesson years ago through the principle of least privilege. We stopped running containers as root whenever possible. We reduced Linux capabilities to the minimum required set. We learned that mounting Docker sockets into containers for convenience often created unacceptable security risks. The common theme was simple: convenience should not bypass security boundaries. The same principle applies to operational automation. Granting unrestricted access to restart workloads, modify resource limits, or execute privileged actions without meaningful controls introduces unnecessary risk. The challenge is not improving the quality of recommendations. The challenge is ensuring that every action is constrained, observable, and reversible. This is where Docker MCP Gateway becomes valuable. Rather than allowing direct access to infrastructure operations, the Gateway places a controlled execution layer between the decision-making component and the underlying tools. Authentication, rate limiting, audit logging, input validation, and execution isolation are applied consistently before any action is performed. In our implementation, every tool invocation passed through HMAC authentication, Redis-backed rate limiting, structured audit logging, and containerized execution. These controls were not added as enhancements; they were treated as core design requirements. Production systems already rely on admission controllers, access controls, audit trails, and policy enforcement. Operational automation should be held to the same standard. Access to credentials should remain isolated from the decision-making layer. Direct access to host resources should be minimized. Every action should be traceable and reviewable. The more authority a system is given, the more important it becomes to enforce clear operational boundaries. Reliable automation depends less on unrestricted capability and more on well-defined constraints. What Docker MCP Gateway Gives You At a high level, Docker MCP Gateway acts as a secure control plane between AI agents and MCP tools, enforcing authentication, rate limits, audit logging, and execution isolation for every tool call. The Model Context Protocol (MCP) is an open standard introduced by Anthropic in late 2024 that gives AI applications a uniform interface for invoking external tools and services. It has since gained support across multiple vendors, including Anthropic, OpenAI, Google DeepMind, and AWS. MCP solves the protocol problem. It doesn't solve the production problem. Production systems require controls around tool execution, not just a standardized way to invoke tools Authenticated tool calls (not just "the agent has the API key in plaintext somewhere")Rate limiting (agents can spiral fast)Audit logging of every decisionContainerized tool isolation (so a misbehaving tool can't take down its host)Centralized policy enforcement (so adding a new server doesn't require reconfiguring every client) Docker MCP Gateway provides these operational controls. It sits between AI clients and MCP servers, routing every tool invocation through a centralized enforcement layer that handles authentication, policy enforcement, rate limiting, and execution isolation. For our work, we built a custom MCP server inside Docker that exposes three remediation tools: check_container_logs, restart_container, and update_container_resources. Every request passes through HMAC authentication, is rate-limited using Redis, and is recorded in a structured JSON audit log before execution.mc From Mohammad-Ali A'râbi, Docker Captain: Docker's AI tooling strategy is fundamentally about building a verifiable supply chain for reasoning engines. You cannot build secure AI on top of bloated, vulnerable foundations. The strategy begins with Docker Hardened Images (DHI), providing agents and MCP servers with minimal attack-surface base images backed by cryptographically signed SLSA Level 3 provenance. The Docker Hub MCP then acts as a discovery layer, allowing agents to find and navigate trusted container artifacts through natural-language interactions. From there, these components converge into Docker AI Governance, where MicroVM-based sandboxes apply strict, deny-by-default controls over filesystem access, network connectivity, and tool execution. Together, these capabilities represent a broader architectural shift from securing application code to securing an agent's entire operational blast radius. Recent supply-chain attacks such as Shai-Hulud 2.0 have shown that modern attackers increasingly target the automation layers that underpin software delivery. AI agents now operate inside those same environments, making blast-radius reduction a first-class architectural concern. A Decision Framework: When to Auto-Fix vs. Escalate Before implementing any automation, we documented the expected behavior for each failure mode. This was not a planning exercise—it became the specification the system had to satisfy and later served as the foundation for our validation framework. Failure Type Likely Cause Safe Action OOMKilled Resource exhaustion (often legitimate) Auto-fix: increase memory CrashLoopBackOff Code or configuration bug Escalate — never auto-restart Single Exit (code 1) Could be transient (network, DB) or persistent Try restart once, escalate if it persists HealthCheckFailure App stuck or deadlocked Auto-fix: restart The guiding principle was simple: transient and resource-related failures could be remediated automatically, while persistent application and configuration failures required escalation. Transient and resource-driven failures auto-fix. Persistent and code-driven failures escalate. Every decision is logged. This framing matters more than the implementation. It's the part you should keep even if you replace every other piece of the system. The agent's job isn't to be smart — it's to apply this rule consistently and visibly. We chose to encode this in the agent's system prompt rather than in code branching, which turned out to be one of our most important design decisions. More on that below. The Architecture in Practice The system has five logical layers running across three Docker Compose containers: Five-layer architecture: container failure triggers the AI agent, which routes every tool call through the Docker MCP Gateway security pipeline before reaching MCP Tools and the Docker API. The architecture separates concerns into five layers. The AutoGen agent (GPT-3.5-turbo, cost-optimized for this decision space) handles reasoning and decision-making. The Docker MCP Gateway sits in front of the tools as a security enforcement point — every tool call passes through HMAC authentication, Redis-backed rate limiting (100 requests/hour), input validation, and structured audit logging. The MCP Tools layer exposes three remediation actions: check_container_logs, restart_container, and update_container_resources. Below that, the Docker API performs the actual container operations. In our current implementation, the Gateway and Tools layers are colocated in a single Python service for simplicity — in a multi-tenant production setup you'd separate them into distinct services that scale independently. Every tool call generates an audit log entry like this: JSON { "timestamp": "2026-05-07T02:08:15.456Z", "incident_id": "inc-20260507-020815", "agent_id": "docker-ops-agent-001", "alert": { "description": "Docker container crashed with OOMKilled", "container_id": "nginx-oom-test", "status": "OOMKilled" }, "decision_chain": [ {"tool": "check_container_logs", "result": "..."}, {"tool": "update_container_resources", "result": "Memory limit updated to 200MB"} ], "resolved": true } That structured output is what makes the system auditable. It's also what makes our validation work possible. The Engineering Reality: 43% to 100% Across 7 development-phase incidents, our agent made the correct decision 43% of the time. Across 6 validation-phase incidents after applying our fixes, it was correct 100% of the time. Both datasets are committed in the repo's monitoring/analysis directory. Phase Runs Correct Avg Turns/Incident Before fixes 7 3/7 (43%) 22.7 After fixes 6 6/6 (100%) 11.7 A note on sample size: this is a small dataset. It's enough to show the expected behavior is reproducible across the four scenarios, but not enough to make claims about reliability under load or at scale. What changed between the two phases is documented as nine challenges in the lab README. Three of them drove most of the improvement. Here they are. Challenge A: The OOM That Couldn't Be Fixed In the early runs, the agent correctly diagnosed an OOMKilled container, called the memory-update tool, and got back this Docker error: Plain Text Memory limit should be smaller than already set memoryswap limit, update the memoryswap at the same time Then it correctly escalated, because it had no tool for updating memoryswap. Our analyzer marked this as wrong because the OOMKilled scenario expected AutoResolved, not Escalated. But the agent's logic was right. The bug wasn't in the agent — it was in our test container's --memory-swap configuration. Once we fixed that (set --memory-swap=-1 for unlimited swap), the agent's behavior didn't change at all. The same logic that escalated correctly before now succeeded correctly. The agent went from 0/2 to 2/2 correct. Lesson: When the agent makes the right decision but your tests say it's wrong, check the test setup before blaming the agent. We spent a few hours debugging the agent before realizing our own container configuration was the problem. Challenge B: The Over-Eager Restart In the first three CrashLoopBackOff runs, the agent restarted the container 2 out of 3 times. CrashLoopBackOff is exactly the failure mode where you should never restart — the container is crashing because of a code or config bug, not a transient state. Restarting just generates more crashes. We almost wrote a code branch for it: add a check, route CrashLoopBackOff to a different path. Before doing that, we tried tightening the system prompt instead: Plain Text For CrashLoopBackOff failures: ALWAYS escalate to a human operator. NEVER attempt to restart the container. Restarting will only cause the container to crash again. Your role is to diagnose and report, not to fix. That single change — no code, just words in the prompt — made the agent consistently escalate on every subsequent run. Lesson: If you want the agent to follow a rule, write the rule down in the system prompt. Don't leave it to the model to figure out. We spent more time arguing about whether to add code branching than the prompt change actually took. Challenge C: The Hallucinated Containers After resolving real incidents, the agent started making up alerts for containers that didn't exist — memory-hungry-app, app-crash-loop, none of which were ever in our system. It was inventing failures and then "responding" to them. Root cause: AutoGen's max_consecutive_auto_reply was set to 10. After the agent finished a real incident, the conversation framework kept giving it turns. Without a real prompt to respond to, it generated plausible-looking next incidents and walked itself through fake remediations. Fix: drop max_consecutive_auto_reply to 3. The agent gets exactly enough turns to diagnose, act, and report — then the conversation ends. Lesson: AutoGen and similar frameworks default to long conversations because they're built for chat use cases. For production, you want them to stop talking once the job is done. From Mohammad-Ali A'râbi, Docker Captain: The progression from 43% to 100% correctness reinforced a key lesson: production AI is often less a machine-learning problem; it is a systems engineering challenge. The initial failures were not the fault of the LLM; they were the result of implicit, undocumented policies and permissive execution environments. Production AI engineering requires moving past the "magic" of conversational models and returning to a rigorous, deterministic engineering discipline. It means treating the system prompt as an immutable policy file, writing explicit, boundary-defining rules that leave zero room for the model to improvise. It means enforcing aggressive Redis-backed rate limits to prevent hallucination loops, isolating execution tools to eliminate docker.sock vulnerabilities, and relying exclusively on structured JSON audit logs rather than plain text for forensic validation. The agent is merely a component. The surrounding infrastructure — the cryptographic constraints, the isolated execution environments, and the hardcoded fallbacks — is what actually makes the system safe. Building trust in AI demands the exact same rigor we apply to cluster security: trust nothing, verify everything, and strictly log the rest. Production Patterns We'd Recommend If you're building something similar with Docker MCP Gateway, here's what we'd carry over from our nine challenges: Authenticate every tool call, even in dev. We used HMAC signing on every request from agent to MCP server. The reason to do this early isn't just production security — it surfaces auth integration bugs during development, when they're cheaper to fix. Use structured JSON for audit logs, not text. The audit format we used (incident ID, agent ID, alert, decision chain, resolved flag) made it possible to write an analyzer that validates agent behavior automatically. Plain text logs would have made that impossible. Set rate limit low. We used Redis with 100 requests per hour per agent. Agents can make a lot of tool calls quickly — a single bug in the system prompt triggered thousands of calls in one of our early runs before we noticed. Default to escalation when uncertain. A false-positive escalation costs you a page that turns out to be nothing. A false-negative auto-fix can mask a real problem for weeks. The costs aren't symmetric, so the default shouldn't be either. Validate against expected behavior. Write down what you expect each failure mode to do, then write an analyzer that checks the audit log against that spec. We open-sourced ours — it's about 250 lines of Python, no external dependencies. You can adapt it to any agent that produces structured audit logs. Tighten conversation turn limits. max_consecutive_auto_reply=3 is a sane starting point for production. The agent should do its job and then the conversation should end. Frameworks default to longer because they're optimized for conversational AI demos, not production ops. What's Still Missing This article would be marketing if we didn't include this section. Honest engineering means owning what isn't built yet. No Docker Scout MCP server exists yet. Security-aware container discovery — "find the most secure nginx tag," "show me CVEs in this image" — isn't possible through MCP today. The Docker Hub MCP server has 13 tools, but none of them surface vulnerability data. This is a real gap in the ecosystem. No incident memory or pattern recognition. Our agent treats every incident as fresh. A production system would learn that this container OOMs every Tuesday at 4 pm and recommend a permanent memory increase rather than reactively bumping it each time. We've left this as future work. Sample sizes are small. Our 6 post-fix incidents prove the expected behavior is reproducible across the four scenarios. They don't prove reliability under production load, traffic spikes, or adversarial conditions. We'd need 100x more data and load testing to make those claims. MTTR is unmeasured. AutoGen records all decision-chain timestamps within microseconds of each other, so the per-incident duration data we collected isn't usable as a real mean-time-to-recovery metric. Capturing real MTTR would require external timing instrumentation around the agent. Gateway and tools are colocated. Our MCP server bundles the security pipeline (HMAC, rate limiting, audit) with the tool execution. In a true multi-tenant production setup, you'd separate these into distinct services so they can scale independently. Our current architecture is fine for a single team or environment; it would need refactoring before serving multiple agent populations. What This Means for AI Infrastructure The interesting part of building agentic infrastructure isn't getting the agent to act. It's getting it to not act when acting would make things worse. Docker MCP Gateway is one of the first production tools that takes this seriously — treating the infrastructure around the agent as the security layer, not the agent itself. The pattern we ended up with — a Gateway in front, scoped tools, decision boundaries written into the system prompt, structured audit logs — isn't novel. It's just what worked. We expect most production AI agents will end up looking similar, because this is what makes them debuggable when something goes wrong. The nine challenges we documented in the lab README are probably challenges you'll hit too. The analyzer script, the audit log format, and the validation patterns are all MIT-licensed in the companion repository. Use whatever's useful. This article was originally published on OpsCart.

By Mohammad-Ali Arabi

Who Owns the Data Stack?: How AI Is Reshaping Ownership, Architecture, and Accountability Across Teams

Editor’s Note: The following is an article written for and published in DZone’s 2026 Trend Report, Cognitive Databases, Intelligent Data: Unified Infrastructure for Vector Search, AI-Optimized Queries, and Hybrid Workloads. For years, some of us have argued that the data stack is part of the product and should be engineered like the application layer: as code and as a service. The market matured toward it, and the data mesh has been the clearest recent expression. AI has eclipsed those debates and settled the matter. The data stack is now product-facing, shaping what users see, what AI answers, and which automated decisions and workflows fire. That makes one question unavoidable: When an answer depends on data across many systems and teams, who is accountable for accuracy? An AI answer is assembled at request time from corporate data. The data stack is inside the response. AI Turns Data Infrastructure Into Product Behavior AI makes the data stack part of product behavior, but raw infrastructure should not leak into the product. The goal is to abstract the stack behind durable, governed interfaces. An AI feature should consume meaning, relationships, permissions, and context. Following data mesh and data contracts, the API layer has to evolve from returning data to exposing capabilities. A consumer, including an AI model, should depend on a contract that carries: Metadata – origin, lineage, meaningQuality – freshness, completeness, confidenceRelationships – how entities compose and traverseSecurity – authorization applied consistently across operational, analytical, and vector stores When meaning lives in the contract, infrastructure becomes interchangeable, and a misbehaving AI feature is no longer an opaque failure — it’s a question with an owner. Where Ownership Breaks First Ownership does not break at the edges of systems but much earlier, in how the organization is designed. Most technology organizations still distribute teams around components and technical specialization: applications, databases, pipelines, governance, indexing, and analytics. Each team owns its layer, though no one owns the end-to-end meaning of the data. That worked when data only fed analytics. It fails in AI-native products, where data is product behavior, and the two lifecycles are inseparable. AI composes its behavior across every layer at once, inheriting each inconsistency in semantics, freshness, permissions, and relationships. So this is not a handoff problem; it is a Conway’s Law problem. Architecture mirrors the organization, and AI makes the organizational seams visible to the user. Platform teams remain essential for shared abstractions, governance primitives, and standards. But product teams need to own both their features and APIs and their data end to end: its lifecycle, meaning, quality, and governance. Splitting teams by technical layer scatters one business entity across many disconnected owners, and AI inherits that fragmentation. AI-native organizations give product teams end-to-end ownership of the data, with platform teams providing shared standards. Accountability Follows Product Behavior When data only fed dashboards, accountability could stay narrow: Did the pipeline run, and did the report match? AI moves that boundary. Once retrieval, copilots, and agents start making decisions and generating answers from data, a correct pipeline, a healthy index, and a valid access policy still don’t guarantee a correct user-facing result. Accountability can’t be pinned to technical layers. It has to follow the behavior the user experiences. The product team that owns an AI capability is responsible for the end-to-end correctness, freshness, explainability, and safety of the data behind it. Its job is to own the contract that defines what the AI may know and retrieve. Platform teams provide the standardized primitives that make this accountability structure possible: semantic contracts, lineage, quality signals, access enforcement, observability, and governance-aware retrieval. The question shifts from “which team owns this layer?” to “which product team owns this behavior, and which platform capabilities guarantee it?” In AI-native systems, accountability rests with the team that owns the behavior, not the system that happened to fail. Table: Accountability Differences Between the Layer-by-Layer and AI-Native Models arealayer-by-layerai-nativeSource of truthEach system decides locallyThe product team owns the authoritative semantic contractQualityThe data team checks pipelinesThe product team owns user-facing correctness; the platform provides quality signalsRetrievalThe platform team owns indexes as infrastructureA governed product capability with explicit SLOsAccessThe security team owns policies separatelyEnforced consistently across product, data, and AI layersIncidentsRouted to whichever layer failedThe product team leads; the platform, data, and security teams support as capability owners Architecture Choices Are Also Operating Model Choices Architecture decisions also decide how an organization governs and evolves meaning. AI-native systems raise the stakes here because copilots and agents consume meaning — entities, relationships, metrics, and permissions — rather than tables. Semantic consistency becomes part of how the product behaves. No central team can own the meaning of every domain, so meaning has to live close to the domain that owns the capability. But decentralization alone backfires: Without platform-enforced standards, the old central bottleneck just turns into semantic fragmentation, with every domain exposing its own definitions and contracts. The fix is to split ownership cleanly: Domains own the meaningPlatform teams own the contracts that keep it consistent Underneath, storage and processing keep churning. What actually lasts is whether stable abstractions (e.g., “employee,” “payroll,” “entitlement”) survive above them. The principle is simple: Infrastructure should be replaceable, and meaning should not. So the real operating-model choice comes down to who owns meaning, and who keeps it consistent. Shared Data Contracts Make Accountability Concrete If organizational fragmentation is the root problem, contracts make ownership explicit. A classic data contract is necessary but insufficient. Schema validation catches a renamed column, but it misses semantic drift, stale meaning, or a changed business definition. Those failures don’t break a build. They break behavior. The contract has to grow from schema into semantics, carrying meaning, lineage, quality, and authorization. Crucially, it abstracts the capability and meaning a domain exposes, not the storage format underneath, so it behaves the same whether the source is a table, a document, an event, or an embedding. That makes the data contract both a producer-to-consumer check and a runtime semantic interface that retrieval, copilots, and agents all consume. Its real value is relocating accountability to the source so drift surfaces in the producing domain while context stays local, which accelerates interoperability rather than centralizing control. Governance Has to Travel With the Data Traditional governance sat beside the data in the form of periodic reviews, approvals, and access checks. AI breaks that model. Data now moves continuously through pipelines, caches, embeddings, indexes, and agents, recomposed at runtime faster than any review can observe. Governance must be part of the execution model itself. Governance travels with meaning, not storage. An embedding holds no raw rows yet reveals sensitive meaning, so policy must follow the semantic classification. The gap is sharpest in authorization. Identity systems stop at the API boundary, and AI doesn’t preserve security boundaries on its own, which turns every embedding, cache, and retrieval step into a new one to defend. Governance therefore becomes a runtime capability that decides what AI may retrieve, infer, expose, and act on. Solving that calls for composable, declarative governance primitives embedded in the platform so auditability becomes a property of the system rather than the outcome of a project. Accountability Gaps That Slow AI Data Work The real cost of fragmented accountability is the constant drag on every data-powered capability. Friction is never neutral, so when teams can’t trust the platform’s freshness, semantics, or governance, they route around it and build their own, resulting in shadow pipelines, local indexes, and duplicated transformations. Each workaround makes sense locally even as it corrodes the whole, fragmenting governance and eroding trust in the very platform it was meant to replace. And piling on more central control only hides the problem — the fragmentation just migrates into those shadow systems. So the deeper gap was missing platform contracts. What Clear Ownership Looks Instead of adding more teams on more layers, clear ownership means aligning accountability with the single product experience the user meets. What you’re really investing in is the stable semantic abstractions that outlast whatever infrastructure comes and goes. And the hardest problem is how to make the organization understandable to its own AI systems. Additional resources: DAMA-DMBOK: Data Management Body of KnowledgeDAMA International – foundational guidance on data ownership, stewardship, and governance rolesOpen Data Contract Standard (ODCS) – an open spec for declaring schema, semantics, quality, and service levels between data producers and consumersOpenLineage – an open standard for collecting data lineage across pipelines and services, useful for tracing what AI features consumeNIST AI Risk Management Framework (AI RMF) – a vendor-neutral framework for accountability and governance of AI systemsCoral – exposes diverse data sources to agents through one declared SQL and semantic layer; an example of meaning being owned per source rather than centrallyGetting Started With Data Quality, DZone Refcard by Miguel García LorenzoData Pipeline Essentials, DZone Refcard by Sudip SenguptaOpen-Source Data Management Practices and Patterns, DZone Refcard by Abhishek GuptaReal-Time Data Architecture Patterns, DZone Refcard by Miguel García Lorenzo“Building Trusted, Performant, and Scalable Databases: A Practitioner’s Checklist” by Saurabh Dashora This is an excerpt from DZone’s 2026 Trend Report, Cognitive Databases, Intelligent Data: Unified Infrastructure for Vector Search, AI-Optimized Queries, and Hybrid Workloads.Read the Free Report

By Miguel Garcia

CORE

Implementing Asynchronous Communication Between Microservices Using Kafka and Spring Boot

In a microservices system, that tight coupling turns a small hiccup into a cascading slowdown. Thread pools fill, retries amplify traffic, and suddenly your simple request is blocked on half the fleet. My executive summary: asynchronous messaging with Kafka helps systems keep moving when individual components inevitably slow down or fail. It does this by decoupling producers from consumers, absorbing traffic spikes, and allowing services to evolve without tying their availability directly to one another. Code Patterns in Spring Boot With Kafka Spring for Apache Kafka gives me two primitives that feel pleasantly old Spring KafkaTemplate for sending and @KafkaListener for receiving. That template/listener model is intentionally similar to other Spring integration tech, which keeps application code focused on domain logic instead of raw client plumbing. Below is a compact (but production-shaped) pattern: externalized config via @ConfigurationProperties, a service port for publishing, a REST command endpoint, a consumer with a real error strategy (DLT), and a REST error advice. Java // === Messaging config (externalized, type-safe) === @ConfigurationProperties(prefix = "messaging.orders") @Validated record OrdersMessagingProps( @NotBlank String topic, @NotBlank String dltTopic ) {} // === DTO (event contract) === public record OrderCreatedEvent(UUID orderId, UUID userId, BigDecimal total, Instant createdAt) {} // === Service port (keeps domain testable, Kafka swappable) === public interface OrderEventPublisher { void publishOrderCreated(OrderCreatedEvent event); } // === Adapter: Kafka producer === @Component class KafkaOrderEventPublisher implements OrderEventPublisher { private final KafkaTemplate<String, OrderCreatedEvent> template; private final OrdersMessagingProps props; KafkaOrderEventPublisher(KafkaTemplate<String, OrderCreatedEvent> template, OrdersMessagingProps props) { this.template = template; this.props = props; } @Override public void publishOrderCreated(OrderCreatedEvent event) { // Keying by orderId keeps per-order ordering and drives partitioning decisions. template.send(props.topic(), event.orderId().toString(), event); } } // === REST command API (synchronous edge, async core) === @RestController @RequestMapping("/v1/orders") class OrdersController { private final OrderService orderService; // domain port OrdersController(OrderService orderService) { this.orderService = orderService; } @PostMapping public ResponseEntity<Map<String, Object>> create(@Valid @RequestBody CreateOrderRequest req) { UUID orderId = orderService.create(req.userId(), req.total()); // persists + publishes event return ResponseEntity.accepted().body(Map.of("orderId", orderId, "status", "ACCEPTED")); } record CreateOrderRequest(@NotNull UUID userId, @NotNull @Positive BigDecimal total) {} } // === Domain service port (implementation can use outbox, transactions, etc.) === public interface OrderService { UUID create(UUID userId, BigDecimal total); } // === Consumer: downstream service reacts to events === @Component class BillingListener { @KafkaListener(topics = "${messaging.orders.topic}", groupId = "${spring.kafka.consumer.group-id}") void onOrderCreated(OrderCreatedEvent event) { // Idempotency belongs here: process-by-key + store processed eventId/orderId to avoid duplicates. // Do work (charge card, create invoice, etc.) } } // === Kafka consumer error handling: retries + DLT === @Configuration class KafkaErrorHandlingConfig { @Bean DefaultErrorHandler defaultErrorHandler(KafkaTemplate<Object, Object> template, OrdersMessagingProps props) { var recoverer = new DeadLetterPublishingRecoverer(template, (rec, ex) -> new TopicPartition(props.dltTopic(), rec.partition())); // Backoff and retry policy are configurable; keep it finite to avoid poison-pill loops. return new DefaultErrorHandler(recoverer, new FixedBackOff(1000L, 3)); } } // === REST error handling (ProblemDetail) === @RestControllerAdvice class ApiErrors { @ExceptionHandler(IllegalArgumentException.class) @ResponseStatus(HttpStatus.BAD_REQUEST) ProblemDetail badRequest(IllegalArgumentException ex) { var pd = ProblemDetail.forStatusAndDetail(HttpStatus.BAD_REQUEST, ex.getMessage()); pd.setTitle("Invalid request"); return pd; } } A few been-burned-before notes on the code above. Spring Kafka’s reference docs are explicit that KafkaTemplate is the convenience wrapper for producing, and DefaultErrorHandler + DeadLetterPublishingRecoverer is a first-class way to route failed records to dead-letter topics after retries. If we want non-blocking retries, Spring Kafka also provides @RetryableTopic, which orchestrates retry topics and a DLT automatically useful when transient failures are common and you want predictable retry delay semantics. Containers and Local Dev With Docker Compose When I’m chasing down event flow bugs, I like local environments that feel like the old days: one command, deterministic startup order, and no mystery dependencies. Docker Compose is still the quickest way to stand up Kafka alongside your services, and Confluent publishes straightforward Docker-based tutorials and compose examples for running Kafka locally. For the service image itself, multi-stage builds are the modern classic compile in a builder stage, and copy the artifact into a slimmer runtime stage. Docker documents multi-stage builds as a way to reduce the final image contents and keep build dependencies out of production. Dockerfile # Multi-stage Dockerfile for a Spring Boot service (orders-service) FROM eclipse-temurin:21-jdk AS build WORKDIR /workspace COPY mvnw pom.xml ./ COPY .mvn .mvn RUN ./mvnw -q -DskipTests dependency:go-offline COPY src src RUN ./mvnw -q -DskipTests package FROM eclipse-temurin:21-jre WORKDIR /app COPY --from=build /workspace/target/*.jar app.jar EXPOSE 8080 ENTRYPOINT ["java","-jar","/app/app.jar"] And here’s a Compose file that wires up Kafka and Schema Registry, plus an example Spring Boot service. The exact image choices are illustrative. Your production choices are unspecified and should reflect your standards and security posture. YAML # compose.yaml (local/dev) services: zookeeper: image: confluentinc/cp-zookeeper:7.6.0 environment: ZOOKEEPER_CLIENT_PORT: 2181 kafka: image: confluentinc/cp-kafka:7.6.0 depends_on: [zookeeper] ports: ["9092:9092"] environment: KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:9092 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 schema-registry: image: confluentinc/cp-schema-registry:7.6.0 depends_on: [kafka] ports: ["8081:8081"] environment: SCHEMA_REGISTRY_HOST_NAME: schema-registry SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: PLAINTEXT://kafka:9092 orders: build: ./orders-service depends_on: [kafka] ports: ["8080:8080"] environment: SPRING_KAFKA_BOOTSTRAP_SERVERS: kafka:9092 MESSAGING_ORDERS_TOPIC: orders.events MESSAGING_ORDERS_DLTTOPIC: orders.events.dlt SCHEMA_REGISTRY_URL: http://schema-registry:8081 Deploying on Kubernetes or AWS On AWS, the Kafka decision is usually managed or self-managed. If you choose Amazon MSK, the cluster lives in your VPC, pick subnets across distinct Availability Zones, and connect clients using the cluster’s bootstrap brokers. That’s the networking baseline, and it’s not optional. MSK is VPC-first by design. For authentication/authorization, MSK supports IAM access control. AWS documents the client configuration for IAM mechanisms. In EKS, I typically pair MSK IAM with IRSA so pods can obtain AWS credentials the AWS way, while ECS services would use task roles instead. Both patterns are documented by AWS, and your choice here is unspecified. Kubernetes service discovery is usually the easy part. Services and Pods get DNS names so workloads can call each other by name rather than IP. Kafka itself is reached via bootstrap broker endpoints or via internal Services, but either way, you want the strings in externalized config, not hardcoded. Here’s a minimal Kubernetes Deployment/Service for a Kafka client service. Values like region, account IDs, and MSK endpoints are unspecified placeholders. YAML apiVersion: apps/v1 kind: Deployment metadata: name: orders namespace: apps spec: replicas: 2 selector: matchLabels: { app: orders } template: metadata: labels: { app: orders } spec: serviceAccountName: orders-sa # IRSA-bound (role ARN unspecified) containers: - name: orders image: <UNSPECIFIED_AWS_ACCOUNT_ID>.dkr.ecr.<UNSPECIFIED_REGION>.amazonaws.com/orders:<TAG> ports: [{ containerPort: 8080 }] env: - name: SPRING_KAFKA_BOOTSTRAP_SERVERS value: "<UNSPECIFIED_MSK_BOOTSTRAP_BROKERS>" - name: MESSAGING_ORDERS_TOPIC value: "orders.events" - name: MESSAGING_ORDERS_DLTTOPIC value: "orders.events.dlt" readinessProbe: httpGet: { path: /actuator/health/readiness, port: 8080 } initialDelaySeconds: 10 --- apiVersion: v1 kind: Service metadata: name: orders namespace: apps spec: selector: { app: orders } ports: - port: 80 targetPort: 8080 Operationally, MSK exposes metrics into CloudWatch (AWS/Kafka), and broker logs can be delivered to CloudWatch Logs (or S3/Firehose). That combination gives you the classic visibility loop: throughput, lag, under-replicated partitions, and error logs without running your own monitoring plane. For distributed tracing in async flows, OpenTelemetry is my default vocabulary now. Spring Boot supports OpenTelemetry export via OTLP, and OpenTelemetry defines Kafka semantic conventions so your producer/consumer spans and attributes stay consistent across tools. CI/CD and the Hard-Earned Field Notes For CI/CD, I keep it boring: build once, push an immutable image, deploy via a declarative mechanism. AWS Prescriptive Guidance provides a clear GitHub Actions pattern for building Docker images and pushing to Amazon ECR, which is a solid baseline when your region/account is unspecified until configured. YAML # .github/workflows/orders.yml name: orders on: push: branches: ["main"] jobs: build_push_deploy: runs-on: ubuntu-latest permissions: id-token: write contents: read steps: - uses: actions/checkout@v4 - uses: actions/setup-java@v4 with: distribution: temurin java-version: "21" - name: Build & test run: ./mvnw -q test package - name: Configure AWS credentials (OIDC) uses: aws-actions/configure-aws-credentials@v4 with: role-to-assume: arn:aws:iam::<UNSPECIFIED_AWS_ACCOUNT_ID>:role/<UNSPECIFIED_GHA_ROLE> aws-region: <UNSPECIFIED_REGION> - name: Login to ECR run: | aws ecr get-login-password --region <UNSPECIFIED_REGION> \ | docker login --username AWS --password-stdin <UNSPECIFIED_AWS_ACCOUNT_ID>.dkr.ecr.<UNSPECIFIED_REGION>.amazonaws.com - name: Build & push image run: | IMAGE=<UNSPECIFIED_AWS_ACCOUNT_ID>.dkr.ecr.<UNSPECIFIED_REGION>.amazonaws.com/orders:${{ github.sha } docker build -t $IMAGE ./orders-service docker push $IMAGE - name: Deploy to EKS (example) run: | aws eks update-kubeconfig --name <UNSPECIFIED_EKS_CLUSTER> --region <UNSPECIFIED_REGION> kubectl -n apps set image deploy/orders orders=$IMAGE Now, the part I wish someone had handed me in 2016: Kafka gives you strong tools, but it does not remove distributed-systems truths. You still need safeguards on the consumer side: idempotent processing, disciplined schema management, and clearly defined retry and dead-letter topic behavior. Kafka’s documentation is careful about the limits of “exactly once” guarantees. Idempotent producers and transactions can strengthen delivery semantics, but achieving true end-to-end exactly-once behavior, especially when external side effects are involved, still depends on deliberate system design. For schema governance, Kafka itself doesn’t ship a schema registry, but acknowledges third-party registries; in practice, Confluent Schema Registry and Apicurio Registry are common choices. Both store schemas out-of-band, so messages carry only a schema identifier, and both support evolvable contracts across Avro/JSON Schema/Protobuf depending on your ecosystem. Conclusion and Best Practices If you take one lesson from my legacy brain into modern event-driven systems, let it be this: asynchrony is a reliability feature, not a performance trick. Kafka’s durable log and consumer group model decouples uptime and absorbs spikes, but you only get the real benefit when you treat schemas as contracts, consumers as idempotent processors, and failure handling as first-class application behavior. On AWS, the operational baseline is non-negotiable. MSK lives in your VPC across AZ subnets, clients connect via bootstrap brokers, IAM auth is configured explicitly, and observability lives in CloudWatch. Do those fundamentals early, and Kafka stops feeling like a mysterious black box and starts feeling like the dependable workhorse it was built to be.

By Mallikharjuna Manepalli

Your AI Coding Agent Can't Steal What It Never Had: The Docker Sandbox Isolation Story

I ran an AI coding agent against a broken Kubernetes deployment for five minutes. The agent called Anthropic's API dozens of times — reasoning about manifests, running kubectl commands, redeploying workloads. It made fully authenticated requests throughout the entire session. The API key was never in its environment. Shell env | grep -iE "anthropic|api_key|secret|token|password" # (empty) That is Docker Sandbox's credential isolation model in action. This article is about what that actually means — and what else the isolation holds, breaks, and surprises you with when you probe it properly. Key Takeaways Docker Sandbox uses a host-side proxy to inject API credentials without the agent ever seeing them — the agent makes authenticated calls without possessing the keySeven live isolation probes confirmed the boundary held throughout real AI agent activity, not just at restNetwork policy is hostname-scoped HTTP filtering — not a full network control plane — with three specific behaviors the documentation doesn't make clearDevOps agents can run docker build and kubectl inside the sandbox without any path to the host Docker daemon or cluster credentialsThe --branch parallel agent mode is Git-level isolation, not VM-level — important distinction for threat models requiring separate credentials per agent The Setup I manage eight AKS clusters for Fortune 500 clients. My laptop has Azure service principals, SSH keys, kubeconfig files with a dozen cluster contexts, and twenty-plus repos — some with .env files containing real API keys. Running an AI agent from this machine without guardrails means the agent inherits all of it. Docker Sandbox changes that. Each sandbox is a microVM — its own Linux kernel, its own Docker daemon, its own network stack. You mount one project directory. The agent sees one project directory. Everything else on the machine does not exist inside the sandbox. I spent two weeks testing this claim. Here is what I found. Test environment: What Detail sbx version v0.31.1 · commit e658be1 Host macOS Apple Silicon Network endpoints probed 13 Isolation probes 7 targeted commands Kubernetes scenario Real agent task, two bugs, timed All findings backed by real terminal output. Full repo: github.com/opscart/docker-sandbox-devops. How the Credential Isolation Actually Works The sandbox environment has no API keys. But the agent made authenticated API calls. Here is the mechanism: Shell env | grep proxy # https_proxy=http://gateway.docker.internal:3128 # http_proxy=http://gateway.docker.internal:3128 # JAVA_TOOL_OPTIONS=-Dhttp.proxyHost=gateway.docker.internal -Dhttp.proxyPort=3128 ... Every outbound request — HTTP, HTTPS, even Java tools — routes through a proxy at gateway.docker.internal:3128. That proxy runs on the Mac host, completely outside the microVM boundary. When the agent sends a POST to api.anthropic.com, there is no Authorization header — the agent does not have the key. The request reaches the host-side proxy. The proxy checks the allowlist — api.anthropic.com is in the default AI services group under the Balanced policy. Authentication is performed by the host-side proxy using credentials stored outside the sandbox boundary. The authenticated request is forwarded to Anthropic. The agent receives the response. It has no idea what key was used, where it came from, or how to find it again. Think of it like an OAuth gateway. The proxy holds the credential and vouches for the agent's requests. The agent gets access without ever possessing the key. You cannot steal what you never had. This is architecturally different from the standard setup where ANTHROPIC_API_KEY sits in the shell environment — one echo $ANTHROPIC_API_KEY away from being exfiltrated. What the Four Isolation Layers Actually Do Docker Sandbox stacks four layers: Hypervisor isolation. Separate Linux kernel per sandbox. Host processes invisible. Other sandboxes invisible. A compromised sandbox cannot escalate to the host kernel. This is the fundamental difference from a Docker container — a container shares the host kernel. The microVM does not. Network isolation. All outbound HTTP/HTTPS routes through the host-side proxy. Raw TCP, UDP, and ICMP are blocked at the network layer. Three policy tiers: allow-all, balanced (curated dev allowlist), deny-all. Set before starting your first sandbox: Shell sbx policy set-default balanced Docker Engine isolation. Each sandbox runs a private Docker daemon with its own socket. No path to the host Docker daemon. An agent can run docker build and docker run without socket mounting — which is the tradeoff that breaks isolation in plain container-based approaches. Credential isolation. Proxy-based injection as described above. The raw key never enters the microVM. macOS host with sensitive assets and proxy on the left, Docker Sandbox microVM in the center, network policy zones on the right. Seven Isolation Proofs — Run Live After a Real Agent Task The agent exited after completing the debugging task. The sandbox remained alive, and I executed the following commands from the same shell session the agent had used — to show exactly what was accessible throughout the entire run. 1. Filesystem Boundary Shell ls /Users/opscart/ # Source ls /Users/opscart/.ssh/ 2>&1 One directory. The workspace mount. SSH keys, other repos, credential directories — none of them exist inside the sandbox. Parent directories above the workspace are read-only stubs with no siblings. One critical implication: if your workspace is your home directory, your entire home is visible and writable. Always mount a project subdirectory, not your home. 2. No Credentials in Environment Shell env | grep -iE "anthropic|api_key|aws|secret|token|password" # (empty) Confirmed. The agent that just made dozens of API calls had no raw credentials anywhere in its environment. 3. Proxy Confirms the Injection Mechanism Shell env | grep proxy # https_proxy=http://gateway.docker.internal:3128 # no_proxy=localhost,127.0.0.1,::1,[::1],gateway.docker.internal Proxy address visible. Credentials it carries: not visible. The mechanism described above confirmed live inside the running sandbox. 4. Process Namespace Shell ps aux | wc -l # 13 A macOS host runs hundreds of processes. The sandbox shows 13 — all internal. The stack includes dockerd, containerd, socat bridging SSH agent forwarding, and the coding agent. Host processes completely invisible. No way to inspect or interact with anything running on the host. 5. Private Docker Engine Shell docker info | grep -E "Server Version|Operating System|ID" # Server Version: 29.4.3 # Operating System: Ubuntu 25.10 (containerized) # ID: e6934b23-368c-4259-a873-96f879f587e5 Ubuntu 25.10. A unique daemon ID that differs from docker info on the host — confirming the sandbox runs a fully isolated daemon. The agent deployed a full Kubernetes cluster using this daemon. No path to the host Docker socket existed. 6. Host Services Unreachable Shell curl -s --max-time 3 https://localhost:6443 2>&1 || echo "blocked" # curl: (7) Failed to connect to localhost port 6443: Connection refused Port 6443 — my minikube cluster on the Mac host. From inside the sandbox, localhost is the sandbox's own loopback. Host clusters, host SSH, host services — unreachable by default. Eight AKS contexts on this machine. Zero is reachable from inside the sandbox without an explicit policy rule. 7. What the Agent Had vs. What It Didn't During the entire debugging task, the agent had full access to one project directory, kubectl to the sandbox-internal Kubernetes cluster, and full Docker capabilities against the private daemon. It could not reach any other directory, cloud credentials, other kubeconfig contexts, the host Docker daemon, or any cluster not running inside the sandbox. All seven proofs held throughout the session without exception. Three Network Policy Findings That Change How You Think About It Network policy is not a full network control plane. It is hostname-scoped HTTP filtering. Three findings define the actual scope: Finding 1: Blocking returns HTTP 403, not TCP rejection. Plain Text probe "example.com" "https://example.com" # example.com | exit=0 | http=403 Exit code 0. The curl command succeeded. The proxy returned 403 directly. An agent that retries on 403 will retry blocked requests indefinitely. It cannot distinguish a blocked domain from a legitimate server-side error by exit code. For DevOps workflows — an agent hitting a blocked container registry will keep retrying silently rather than failing fast. Finding 2: HTTP CONNECT established a tunnel to port 22 on an allowed host. Plain Text # Port 22 — SSH port curl -s --max-time 5 telnet://github.com:22 # Connected to github.com port 22 # Port 9999 — non-standard port curl -s --max-time 5 telnet://github.com:9999 # Connected to github.com port 9999 github.com is on the Balanced allowlist. HTTP CONNECT established TCP tunnels to github.com on both port 22 and the non-standard port 9999 — both succeeded. Port-based restrictions are not enforced at the proxy layer. The Balanced policy is hostname-scoped only. Any port to an allowed host is reachable via HTTP CONNECT. Finding 3: DNS is not filtered. A common assumption is that all outbound traffic routes through the HTTP proxy — including DNS. Lab results show DNS resolution occurs independently: Plain Text dig example.com +short # 172.66.147.243 A blocked domain resolved. The microVM has an internal stub resolver that forwards DNS independently of the HTTP proxy. An agent can resolve any hostname regardless of the active policy. DNS cannot serve as a secondary enforcement layer. These findings do not break the isolation model. They define its actual boundary. Network policy controls HTTP/HTTPS access by hostname. It does not control DNS, TCP tunnels to allowed hosts on arbitrary ports, or how agents interpret 403 responses. The Agent Scenario: Isolation Under Real Load The real test of isolation is not seven probe commands — it is whether the boundary holds while an agent is actively working, making API calls, running kubectl, deploying containers. I gave an AI agent a broken Kubernetes deployment: a payments-service with memory limits set to 64Mi on a service that needs ~150Mi at peak. The agent received a task file and a set of manifests. No other context. The agent completed the task in under five minutes. It found two bugs — one planted, one discovered independently by reading the manifest and noticing health check probes targeting port 8080 on an nginx container that only serves on port 80. The task said nothing about probes. Result: both pods 1/1 Running, 0 restarts. The seven isolation proofs above were verified immediately after — throughout the entire debugging session, the boundary held without exception. Full article and complete repo at opscart.com/docker-sandbox-devops. What This Means for DevOps Engineers Specifically Most Docker Sandbox articles target software developers running Claude Code on a single codebase. The DevOps case is different and more demanding. A DevOps engineer running an AI agent faces a broader attack surface: multiple cluster contexts, infrastructure credentials, IAM roles, service accounts, kubeconfigs that grant production access. The blast radius of a compromised or manipulated agent is not one repo — it is potentially every system those credentials touch. Docker Sandbox addresses this at the architecture level rather than the prompt level. You are not relying on the agent being well-behaved. You are relying on the microVM boundary, the proxy, and the private Docker daemon. The agent can be fully autonomous inside the sandbox because the guardrail is the environment, not the agent's behavior. The private Docker Engine is particularly significant. DevOps agents need to build and test containers. Every other local isolation approach that allows container operations requires socket mounting — which gives the agent direct access to the host Docker daemon and every image and volume on the host. Docker Sandbox eliminates this tradeoff. What Is Still Rough The image iteration cycle is the primary friction point. Adding a tool requires editing a Dockerfile, rebuilding, pushing to a registry, and recreating the sandbox. For a stable toolchain, this is acceptable. For rapid experimentation, it is not. The --branch parallel agent mode is Git isolation, not VM isolation. Both agents run in one microVM with shared Docker and network. For separate credentials or separate network policies per agent, you need separate workspace directories. The network policy CLI has non-obvious syntax in several places — sbx policy deny does not remove an allow rule, and external cluster access requires two policy rules not one. Neither behavior is documented. The CLI changes between minor versions. v0.31.1 changed login flow, renamed policy tiers, and introduced --clone mode. Pin your version. When Not to Use Docker Sandbox Docker Sandbox is the right tool for a specific set of problems. It is not the right tool when: You need raw UDP or ICMP. Network tracing tools (traceroute, mtr), some mTLS configurations, and anything relying on ICMP will not work — the sandbox proxy only handles HTTP/HTTPS. Your toolchain requires host-device access. USB devices, GPU passthrough beyond basic forwarding, and hardware security keys are not accessible from inside the microVM. You are on a memory-constrained machine. Each sandbox runs a full microVM plus its own Docker daemon. On a machine with 8GB RAM, running multiple sandboxes simultaneously alongside Docker Desktop and a browser will cause pressure. You need production-grade audit logging. Docker Sandbox is Experimental. Audit trails, compliance logging, and enterprise controls are not mature yet. For regulated environments, evaluate accordingly. Your agent needs to coordinate across multiple repositories simultaneously. The one-sandbox-per-workspace model means cross-repo agent work requires careful orchestration. The --clone mode helps but adds git workflow overhead. Conclusion The credential isolation model is the headline: the agent made authenticated API calls throughout the session without the API key ever entering the sandbox. Authentication was performed by the host-side proxy using credentials stored outside the sandbox boundary. The agent could use the credential — it could never see, copy, or exfiltrate it. Seven isolation proofs confirmed the boundary held under real active load. One directory visible. No credentials. No host processes. No host clusters. No host Docker daemon. The network policy findings add important nuance. The --branch mode reality is different from what the documentation implies. Docker Sandbox is Experimental, and the CLI is moving. Use it knowing what it is — and what it is not.

By Shamsher Khan

CORE

Microservices

DZone's Featured Microservices Resources

Top Microservices Experts

The Latest Microservices Topics