Deployment Resources

DZone's Featured Deployment Resources

AWS Glue ETL Design Principles for Production PySpark Pipelines

By Janani Annur Thiruvengadam

CORE

AWS Glue makes it easy to get a PySpark pipeline running quickly. It is significantly harder to build one that stays maintainable as logic grows, performs reliably at scale, and does not quietly accumulate operational debt over time. Most Glue pipelines start simple and become difficult to manage gradually — formulas get hardcoded, modules grow without boundaries, output files proliferate, and before long a single job is doing too many things in ways that are hard to test, hard to debug, and expensive to change. This article presents a set of design principles drawn from production Glue ETL pipelines processing billions of rows. Each principle is independent — you do not need to adopt all of them to benefit from any one. But together they form a coherent approach to building Glue pipelines that are modular, observable, cost-efficient, and built to last. Principle 1: Externalize Logic Into Config, Not Code The single most impactful structural decision in a Glue pipeline is where business logic lives. When formulas, dataset references, column selections, and filter conditions are hardcoded in PySpark, every change requires modifying job code, redeploying, and re-validating the full pipeline. A one-line formula change carries the same deployment risk as a structural refactor. Over time, this creates a strong disincentive to make changes, and the pipeline calcifies. The better pattern is to treat the Spark job as a generic executor and externalize all business-specific declarations into configuration. Formulas are declared as config entries with operands, rounding rules, and output names. Dataset loading behavior — which table, which columns, which filters, whether to cache — is declared per source rather than scripted per job. Schema shapes for complex types are declared explicitly rather than inlined. JSON { "source_table": "headcount_actuals", "database": "finance_db", "select_columns": ["site", "badge_type", "headcount", "fiscal_week"], "filters": [{"column": "is_active", "value": "Y"}], "rename": {"hc_count": "headcount"}, "cache": true } When a new dataset is needed, a new config entry is added — no Spark code changes. When a formula changes, the config entry is updated — no job redeployment required. The job itself becomes stable and generic; only config changes as business requirements evolve. This principle pays increasing dividends over time. Pipelines with externalized logic are faster to modify, safer to deploy, and easier to hand off because the business rules are readable independently of the execution engine. Principle 2: Design Modules With Explicit Boundaries A Glue job that does everything in one place is easy to write and hard to maintain. As pipelines grow, the instinct to add more logic to an existing job accelerates technical debt faster than almost any other decision. The more durable pattern is to decompose computation into modules with explicit input and output contracts. Each module receives one or more DataFrames, applies a focused set of transformations, and produces a named output DataFrame. Modules communicate exclusively through in-memory DataFrame references — there is no disk I/O between stages, no shared mutable state, and no implicit dependency on execution order beyond what the data flow itself requires. Utilities follow the same boundary principle, organized into two layers. Generic pipeline utilities handle cross-cutting concerns — file writing, dataset loading, filtering, deduplication, pivot operations — and are shared across all modules. Module-specific utilities implement transformation logic scoped to a single module and are never invoked outside it. This structure means adding a new module requires only writing its scoped utilities and wiring it into the pipeline. The generic layer is never touched. Existing modules are never at risk from new module development. The downstream benefit is testability. Each module with clean boundaries can be validated independently using mocked PySpark DataFrames with no Glue environment required. Engineers can run pytest locally against individual modules, iterate quickly, and deploy only after local validation passes. Principle 3: Choose Your Job Topology Deliberately A common default in complex pipelines is to split computation across multiple Glue jobs, using S3 as the handoff layer between stages. This is sometimes the right choice — but it should be a deliberate decision, not an instinct. Multi-job topologies make sense when stages have genuinely different compute profiles, when intermediate outputs need to be reused independently by other consumers, or when a stage failure should not force a full recompute from the beginning. In these cases, job separation gives you independent retry boundaries, independent DPU sizing, and the ability to schedule stages on different cadences. Single-job topologies — where the full pipeline runs within one Spark session — make sense when all computation is tightly coupled, modules share the same input datasets, and intermediate outputs have no standalone value. Running everything in one session eliminates cold start overhead for intermediate stages, avoids the cost of serializing data to S3 and deserializing it back between jobs, and keeps the execution model simple to reason about: one trigger, one job, one result. The question to ask is whether the stages truly need to be independent. If intermediate S3 persistence adds coordination complexity without adding value — no independent consumers, no differential retry requirements, no meaningful DPU difference between stages — then collapsing to a single job is usually faster, simpler, and cheaper. If stages have real independence requirements, splitting them is the right call and the operational overhead is justified. Neither topology is inherently superior. The mistake is defaulting to one without evaluating the trade-offs for the specific pipeline at hand. Principle 4: Overlap Writes With Computation When Latency Matters Overlapping writes with computation is a well-established technique in high-performance computing, deep learning training, and heavy database operations. The core idea is to hide the slow latency of I/O operations by running them in the background while the CPU or GPU continues processing data. Rather than waiting for a write to complete before starting the next computation, both proceed simultaneously — I/O latency is absorbed into computation time rather than added on top of it. In Glue ETL pipelines, the same principle applies directly. In a pipeline where multiple output DataFrames are produced, the naive write strategy — complete all computation, then write all outputs sequentially — has two compounding problems. First, it creates a peak memory spike: all computed results are held in memory simultaneously while writes proceed one by one. Second, it serializes work that does not need to be serial: every millisecond spent waiting for S3 acknowledgment is a millisecond the Spark executors are idle. This is worth addressing only when latency is a meaningful constraint. For low-frequency batch jobs running overnight with no user-facing SLA, sequential writes are perfectly adequate. But for pipelines where users or downstream systems are waiting on results — or where job duration directly affects infrastructure cost — overlapping writes with computation delivers measurable wall-clock reduction. The two-phase write strategy implements this directly. Outputs from early modules are written to S3 in background threads immediately after those modules complete, running in parallel with later computation stages. By the time all computation finishes, a significant portion of the output data has already landed in S3. Remaining outputs are then flushed concurrently in a second phase. The implementation leans on Python's concurrent.futures.ThreadPoolExecutor to manage background write threads while the main Spark session continues computation on the driver. A generic write orchestration utility can wrap this pattern so individual modules never need to manage thread lifecycle directly — they simply declare their output and the utility handles scheduling, thread management, and error propagation. Python from concurrent.futures import ThreadPoolExecutor, as_completed def write_phase_a(write_tasks): with ThreadPoolExecutor(max_workers=len(write_tasks)) as executor: futures = {executor.submit(task["fn"], task["df"], task["path"]): task["name"] for task in write_tasks} for future in as_completed(futures): name = futures[future] future.result() logger.info(f"[Phase A] Write complete: {name}") The practical effect is that peak memory pressure is distributed over the job's lifetime rather than concentrated at the end, and total wall-clock time is reduced by the overlap between I/O and CPU-bound computation. For pipelines with many output datasets and a latency SLA to meet, the savings compound significantly. Principle 5: Right-Size Output Files With a Reusable Writer Utility Right-sizing output files is the practice of tuning file sizes to balance disk I/O performance, network transfer speeds, and downstream processing efficiency. Too many small files and downstream readers spend more time on metadata operations and S3 API calls than on actual data reads. Too few large files and parallelism suffers — readers cannot split work efficiently across threads or nodes. The target is consolidated, evenly sized files that match the read patterns of downstream consumers. Spark's default output behavior writes one file per partition, and partition counts are typically tuned for computation throughput rather than output shape. A job optimized for shuffle performance might produce hundreds of partitions, each containing a few megabytes of output data — perfectly reasonable for Spark internals, but harmful for any reader that comes after. This small file problem compounds over time as output partitions accumulate in S3 and the Glue Catalog metadata grows with them. The fix is a reusable writer utility that decouples output file sizing from Spark's internal partition count. Rather than accepting the default, the utility estimates the DataFrame's actual size, calculates the appropriate number of output files for a target file size — typically 128MB to 256MB per file — and coalesces partitions before writing. Python def write_optimized(df, output_path, partition_cols, target_file_size_mb=128): estimated_size_mb = df.rdd.map(lambda row: len(str(row))).sum() / (1024 * 1024) optimal_partitions = max(1, int(estimated_size_mb / target_file_size_mb)) df.coalesce(optimal_partitions) \ .write \ .partitionBy(*partition_cols) \ .parquet(output_path, mode="overwrite") Making this a shared generic utility rather than inline logic in each module has two practical benefits. First, it enforces consistent file sizing behavior across all outputs in the pipeline — no module accidentally writes thousands of tiny files because an engineer forgot to coalesce. Second, it centralizes the tuning knob: when the target file size needs to change — because downstream query patterns shift or a new consumer has different read characteristics — it changes in one place and applies everywhere. Right-sized output files improve Athena scan performance, reduce per-query S3 API costs, keep Glue Catalog partition metadata manageable, and make the output data easier to consume for any downstream system reading from S3. This is a low-effort, high-payoff improvement that applies to virtually every Glue pipeline writing to S3. Principle 6: Use Complex Types to Defer Denormalization SQL-based pipelines are constrained to flat, fully denormalized row structures at every intermediate stage because SQL has no native complex type support. This forces denormalization to happen early, inflating data volume at every subsequent join and aggregation. PySpark has native support for structs, maps, and arrays. Using these types at intermediate stages allows related values to be grouped logically without inflating row counts. A row that would require five denormalized rows in SQL can be represented as a single row with a struct or array column in Spark. Denormalization is then deferred to the final output layer only — applied once, at write time, for consumers that require flat structures. Everything upstream of the final write benefits from reduced volume, fewer shuffles, and faster joins. This principle is particularly impactful in pipelines with multi-level aggregations or wide schemas where dozens of metrics attach to the same dimensional key. Keeping those metrics grouped in a struct until the final output stage reduces the effective row count and join complexity throughout the pipeline. Principle 7: Build Observability Into Every Stage Glue jobs that fail silently or surface errors as opaque stack traces at the end of a long execution are expensive to debug. The investment in step-level observability pays back quickly the first time something goes wrong in production. The minimum viable observability pattern is row count logging at every materialization point. After each module completes and after each write, log the output row count with a descriptive label. This gives a running picture of data volume through the pipeline and makes it immediately obvious when a transformation has dropped rows unexpectedly or produced more rows than expected. Python def log_step(df, step_name): count = df.count() logger.info(f"[{step_name}] Row count: {count:,}") return df Pair this with a try/except/finally pattern at the job level that ensures spark.catalog.clearCache() is always called on exit — whether the job succeeds or fails — to release cached DataFrames and avoid memory leaks across retries. Python try: run_pipeline() except Exception as e: logger.error(f"Pipeline failed: {e}") raise finally: spark.catalog.clearCache() CloudWatch captures all logs automatically. When a job fails, the row count trail shows exactly where in the pipeline the problem occurred, making triage faster and reducing the time between failure and fix. Principle 8: Isolate Executions for Concurrency Pipelines that share compute resources across simultaneous executions create contention that is difficult to predict and expensive to manage. The common response — queue-based serialization — adds operational complexity without solving the underlying resource constraint. AWS Glue's execution model eliminates this problem structurally. Each job execution gets its own isolated DPU allocation. There is no shared compute pool. Ten simultaneous executions consume ten independent DPU allocations and do not interfere with each other in any way. Designing for this means treating each execution as fully independent: no shared state, no cross-execution coordination, no assumption about what other executions are running. Combined with idempotent writes — using overwrite mode so a retry produces the same result as the original execution — the pipeline becomes safe to run concurrently at any scale without additional coordination logic. The cost model reinforces this. Glue bills per DPU-second of actual compute consumed. An execution that takes eight minutes on 240 DPUs costs the same whether it runs alone or alongside a hundred other executions. There is no premium for concurrency and no shared pool to provision for peak load. Putting It Together These eight principles are independent but complementary. A pipeline that applies all of them is modular enough to develop in parallel, observable enough to debug quickly, cost-efficient enough to run at scale, and stable enough to maintain over time without accumulating structural debt. The quickest wins for most existing pipelines are Principles 1, 5, and 7 — externalizing logic into config, right-sizing output files with a shared utility, and adding row count logging at every stage. Each can be applied incrementally without restructuring the full pipeline. The remaining principles become more valuable as pipeline complexity grows and concurrency requirements increase. The underlying thesis is simple: a well-designed Glue pipeline should be easy to change, easy to test, easy to debug, and cheap to run. None of those properties require exotic infrastructure. They require deliberate design decisions applied consistently from the start. More

From Bash Script to Operational Triage: What Eight Months of Kubernetes Debugging Taught Me

By Shamsher Khan

CORE

In November 2025, I published a Bash script that analyzed Kubernetes clusters in about 60 seconds. It generated HTML reports, surfaced crash loops, orphaned resources, and other operational issues that were easy to overlook. The most interesting part wasn't the script — it was what happened after people started running it. Many told me they found problems they hadn't known existed. Looking back, the bash script wasn't really solving debugging. It was solving prioritization. I just didn't have the vocabulary for it yet. That script eventually became four different experiments, then a collection of small scanners, and eventually the dashboard shown in this article. Over the next eight months, that script evolved into OpsCart Watcher — an open-source operational triage dashboard for Kubernetes. This article is about what the journey taught me, and what I think is still missing from most Kubernetes environments. OpsCart Watcher — operational triage for Kubernetes (6 minutes) The Problem the Script Revealed The script did one thing well: it looked at an entire cluster and listed what was broken. Engineers who ran it kept telling me the same thing — "I had no idea this was there." That response was the important signal. These engineers had Grafana, Prometheus, and kubectl. Visibility was not their problem. The problem was that nothing told them to look at this specific namespace, this specific pod, this specific storage volume — before it became an incident. Consider a pod in CrashLoopBackOff for 19 days with 5,000+ restarts. To a metrics dashboard, that deployment looks healthy: replica count satisfied, a pod exists in Running state between crashes, CPU and memory flat because the container barely lives long enough to consume anything. The dashboard is answering the question it was built to answer — is the cluster meeting its SLOs? — and the answer is yes. The question nobody built tooling for: what deserves attention right now? LayerWhat It AnswersToolsMetricsIs the cluster meeting its SLOs?Prometheus, Grafana, DatadogPer-resource stateWhat is this specific pod doing?kubectl, k9s, LensOperational triageWhat deserves attention right now?Prioritizing operational work across cluster state What Triage Looks Like in Practice Overview page — Incident Score 41/100, KPI bar, Top 5, War Room panel The first time I ran the rebuilt dashboard against a cluster with real failures, the top of the screen didn't show me a CrashLoopBackOff pod. It showed me four CrashLoopBackOff pods spread across three namespaces, collapsed into a single operational problem: Plain Text 1. 4 pods crash-looping CRITICAL payments/fraud-detection (1810 restarts) → kubectl logs fraud-detection-... -n payments --previous That collapsing is the entire idea. Instead of inspecting every deployment individually, I was looking at a ranked list of operational problems — each with a severity, a location, and the exact kubectl command to start investigating. The full output for this environment: Plain Text Incident Score: 41/100 (Degraded) Top 5 Things to Fix: 1. 4 pods crash-looping CRITICAL 4 pods 2. 3 image_pull_backoff issues CRITICAL 3 items 3. 1 privileged_container issue CRITICAL 1 item 4. 1 namespace missing NetworkPolicy HIGH 1 ns 5. 3 orphaned PVCs wasting money MEDIUM 80 GB None of these had triggered an alert. All were present and accumulating before the scan. The Incident Score — a composite 0–100 across reliability, security, and waste — exists for one reason. Engineers fix incidents. Managers remember numbers. "We moved the Incident Score from 41 to 67" is a sentence that sticks. The crash loops and NetworkPolicies are the work behind it. The Step After Detection Finding problems was never the hard part. Knowing where to begin was. The most common feedback on the original bash script was some version of: "I found the problem, but I still didn't know what to do next." In March, I wrote about finding a container with 24,069 restarts that had been accumulating undetected. Finding it took sixty seconds. The next hour was the actual work: what do I run first? Is this configuration or code? Is it customer-facing? The investigation page is my answer to that hour. Investigation page — OpsCart Assessment, Evidence, Recommended Investigation One click from any triage finding opens a dedicated investigation view: Plain Text OpsCart Assessment This workload has restarted 1810 times over 6 days. The restart rate appears stable, suggesting a deterministic configuration or application failure rather than an intermittent infrastructure issue. No referenced ConfigMaps or Secrets were detected in the pod spec — missing configuration is unlikely to be the root cause. Investigation should begin with previous container logs. Estimated time: 5–10 minutes. Evidence [1810 Restarts] [CrashLoopBackOff] [6d] [Deployment/fraud-detection] Recommended Investigation HIGH CONFIDENCE Check previous container logs MEDIUM Verify ConfigMaps and Secrets exist LOW Check for OOMKill in events The assessment is rules-based — no AI. It reads restart count, failure pattern (stable vs accelerating), and referenced configuration objects, then produces a deterministic, auditable summary. The confidence levels reflect how a senior engineer actually reasons: previous logs are almost always the right first move for a crash loop; OOMKill is worth checking but less likely. This is the part kubectl doesn't give you. Neither does Lens, k9s, or Headlamp. From "What Is Broken?" to "What Changed?" The biggest architectural change came when the dashboard gained memory. The first version of the tool answered: "what is broken?" The current version — backed by a small embedded database recording every scan — answers "what changed?" That sounds like a minor distinction. Operationally, it changes everything. An incident that has existed for three days deserves different attention than one that appeared five minutes ago. A cluster whose Incident Score dropped eight points overnight is telling you something that no single scan can. War Room — critical issues with visual differentiation per type Every KPI now carries a trend arrow — critical issues up three since the last scan, waste down one — and the Incident Score shows a seven-point sparkline. Each incident is tracked with first-seen and last-seen timestamps and an active/resolved status, so "CrashLoopBackOff — first detected 6 days ago, still active" replaces "CrashLoopBackOff." Operational memory changed the tool from a scanner into something that remembers the history of a cluster. What This Is Not The triage pattern does not answer when an issue started at the metrics level, why an application is slow, or whether last Tuesday's deployment caused a regression. Prometheus, APM tooling, and deployment audit logs remain the right tools for those questions. The triage layer is not a replacement for observability. It is the layer that tells you which questions to ask of your observability stack. The Biggest Lesson When I started, I thought Kubernetes debugging was about collecting more information. It wasn't. Kubernetes already exposes almost everything an operator needs through its API. The difficult part is deciding what deserves attention first. Over eight months, I found myself spending less time searching for failures and more time ranking them. That is ultimately what OpsCart became — not another dashboard, but a prioritization engine for cluster operations. Why Open Source I considered keeping the dashboard private. Instead, I open-sourced it because operational patterns only become useful when they're tested across different clusters. Every environment fails differently, and I wanted the prioritization model to evolve from real-world feedback rather than a single infrastructure. The Remaining Gap The conclusion from my March article is still true: the question worth asking of your environment is not whether these conditions exist — they almost certainly do — but whether your current observability layer would surface them before they become incident preconditions. Eight months of building has only made that conclusion more specific. The gap is not data. The gap is attention: knowing which five things, out of hundreds of resources, deserve a human's time right now. Eight months ago I thought I was building a better debugging script. I wasn't. I was building something that helps operators decide where to spend the next ten minutes. About the environment: The scenarios shown in this article — CrashLoopBackOff pods, orphaned PVCs, missing NetworkPolicies, privileged containers — are representative of what OpsCart finds on real production clusters. The environment shown is a dedicated demonstration cluster configured with realistic failure scenarios. No production data was used. About the tool: OpsCart Watcher is open-source at github.com/opscart/opscart-k8s-watcher. It deploys as a single read-only container: Shell kubectl apply -f https://raw.githubusercontent.com/opscart/opscart-k8s-watcher/main/deploy/dashboard.yaml kubectl port-forward -n opscart-system svc/opscart-watcher 8080:80 More

Getting Started With RabbitMQ in Spring Boot

By Gunter Rotsaert

CORE

Azure Databricks for Scalable MLOps and Feature Engineering With Apache Spark, Delta Lake, and MLflow

By Jubin Abhishek Soni

CORE

AI Won't Keep You from Hitting the Scalability Wall

By Bru Woodring

Azure Databricks vs Microsoft Fabric: An Honest Guide to When to Use What

If you're building a data platform on Azure in 2026, you're going to be asked this question: Azure Databricks or Microsoft Fabric? Both run on Delta Lake, both integrate with ADLS Gen2, both have Spark, and both promise to be your unified data platform. The overlap is real, and the marketing doesn't help. This post is an honest breakdown of where each genuinely excels, where they overlap, and how to decide without getting lost in feature comparison tables. Architecture Comparison Decision Flow Detailed Capability Comparison CapabilityAzure DatabricksMicrosoft FabricWinnerSpark engineFull Spark, Photon, tunableSpark via Notebooks, less tunableDatabricksDelta LakeNative, full controlVia OneLake (Delta Parquet)TieMLflow / MLOpsNative, full MLflow stackBasic experiment trackingDatabricksModel servingDatabricks Model ServingAzure ML integrationDatabricksPower BI integrationDirectQuery via SQL WarehouseDirect Lake (zero-copy, faster)FabricSQL analyticsServerless SQL Warehouse + PhotonSQL Analytics EndpointTieData pipelinesDelta Live Tables, WorkflowsData Factory pipelines (mature)TieReal-time intelligenceSpark Streaming + KafkaEventstream + KQL DatabaseFabricSetup complexityMedium-highLow (SaaS)FabricFine-grained governanceUnity Catalog (mature)Purview integration (growing)DatabricksCost modelDBU + VMFabric capacity unitsComparableOpen format portabilityHigh (standard Delta/Parquet)Medium (OneLake but some lock-in)Databricks Step 1 — Reading Data from Fabric OneLake in Azure Databricks The good news: Fabric and Databricks can share data via OneLake, which speaks Delta format. You don't have to pick one and abandon the other. Python # Azure Databricks reading from Microsoft Fabric OneLake # OneLake exposes an ABFS-compatible endpoint # Authenticate using the workspace's Managed Identity or Service Principal tenant_id = dbutils.secrets.get("kv-scope", "sp-tenant-id") client_id = dbutils.secrets.get("kv-scope", "sp-client-id") client_secret = dbutils.secrets.get("kv-scope", "sp-client-secret") # OneLake uses the same ABFS protocol as ADLS Gen2 fabric_workspace_id = "your-fabric-workspace-guid" lakehouse_name = "your-lakehouse-name" onelake_host = "onelake.dfs.fabric.microsoft.com" spark.conf.set(f"fs.azure.account.auth.type.{onelake_host}", "OAuth") spark.conf.set(f"fs.azure.account.oauth.provider.type.{onelake_host}", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider") spark.conf.set(f"fs.azure.account.oauth2.client.id.{onelake_host}", client_id) spark.conf.set(f"fs.azure.account.oauth2.client.secret.{onelake_host}", client_secret) spark.conf.set(f"fs.azure.account.oauth2.client.endpoint.{onelake_host}", f"https://login.microsoftonline.com/{tenant_id}/oauth2/token") # Read a Delta table from Fabric Lakehouse fabric_path = f"abfss://{fabric_workspace_id}@{onelake_host}/{lakehouse_name}.Lakehouse/Tables/sales_gold" fabric_df = spark.read.format("delta").load(fabric_path) print(f"Rows from Fabric Lakehouse: {fabric_df.count()}") fabric_df.show(5) Step 2 — Writing Databricks Results Back to OneLake Run heavy ML feature engineering in Databricks, write results back to OneLake so Fabric Power BI can consume them via Direct Lake — zero-copy, sub-second dashboard refresh. Python from pyspark.sql.functions import current_timestamp, lit # Run your Databricks feature engineering / ML inference here result_df = spark.table("production.gold.churn_predictions") \ .withColumn("_computed_at", current_timestamp()) \ .withColumn("_source", lit("databricks-inference-job")) # Write back to Fabric OneLake as Delta output_path = f"abfss://{fabric_workspace_id}@{onelake_host}/{lakehouse_name}.Lakehouse/Tables/churn_predictions" result_df.write \ .format("delta") \ .mode("overwrite") \ .option("overwriteSchema", "true") \ .save(output_path) print(f"Written {result_df.count()} rows to Fabric OneLake.") print("Power BI Direct Lake will pick up changes automatically.") Step 3 — When to Use Fabric Notebooks vs Databricks Notebooks Not everything needs Databricks. Fabric Notebooks are good enough for lighter data prep that feeds Power BI reports. Python # This kind of transformation is fine in Fabric Notebooks # Use Fabric when: output goes directly to Power BI, team is analytics-focused, # no MLflow tracking needed, data volume < 100GB # Fabric Notebook (PySpark — same syntax as Databricks) from pyspark.sql.functions import col, sum as _sum, date_trunc df = spark.read.format("delta").load("Tables/sales_silver") summary = df \ .withColumn("month", date_trunc("month", col("sale_ts"))) \ .groupBy("month", "region", "product_category") \ .agg(_sum("revenue").alias("monthly_revenue")) \ .orderBy("month", "region") # Write to Lakehouse table — Power BI picks it up via Direct Lake summary.write.format("delta").mode("overwrite").saveAsTable("monthly_revenue_summary") # Use Databricks when: MLflow tracking needed, complex ML pipeline, # Unity Catalog governance required, data volume > 1TB, streaming workloads When to Use Which: Decision Framework Python # Use this as a mental checklist when deciding DATABRICKS_STRENGTHS = [ "Complex ML pipelines with MLflow experiment tracking", "Production model serving with A/B testing", "Fine-grained governance via Unity Catalog (row/column security)", "Spark Structured Streaming with Kafka / Event Hub", "Very large scale ETL (multi-TB, complex joins)", "Open-source tool integrations (dbt, Great Expectations, etc.)", "Multi-cloud or portability requirements", ] FABRIC_STRENGTHS = [ "Power BI as the primary consumption layer (Direct Lake = fastest)", "Analytics-focused teams without deep Spark expertise", "Microsoft 365 integration (Teams, SharePoint data sources)", "Real-time dashboards via Eventstream + KQL", "Fabric Data Factory for straightforward ELT pipelines", "Lower operational overhead — fully SaaS managed", "Already licensed via Microsoft 365 E5 / Fabric capacity", ] BOTH_TOGETHER = [ "Heavy ML/MLOps in Databricks, results published to OneLake for Power BI", "Fabric Data Factory for ingestion, Databricks for complex transformation", "Unity Catalog governing Databricks tables, Fabric consuming via shortcuts", ] Things to Watch in Production OneLake shortcuts are the integration bridge. Fabric Lakehouses support shortcuts that point to external Delta tables in ADLS Gen2 — the same storage Databricks writes to. This means Databricks writes once and Fabric reads without data movement. Set up shortcuts rather than copying data between platforms. Unity Catalog doesn't govern Fabric. Your row-level security and column masks in Unity Catalog do not apply when Fabric reads the same underlying Delta files directly. If governance is critical, either run everything through Databricks or replicate governance rules in Fabric's permission model. Fabric capacity units and Databricks DBUs are both usage-based but measure differently. Don't try to compare them directly. Run the same workload in both and compare wall-clock time and cost on your actual data sizes. Fabric ML is improving fast but isn't MLflow. As of early 2026, Fabric ML experiment tracking is functional but doesn't have the depth of MLflow's model registry, artifact storage, or model serving. If MLOps maturity matters, stay on Databricks for ML. Wrapping Up The honest answer is: most mature Azure data platforms in 2026 use both. Azure Databricks for ML, complex transformations, governance, and streaming. Microsoft Fabric for Power BI-first analytics, simpler pipelines, and teams that don't need the full Databricks stack. OneLake shortcuts and the shared Delta format make them composable rather than competitive. Pick based on your primary consumer: if it's Power BI dashboards, start with Fabric. If it's ML models and data products, start with Databricks. When you need both, they integrate cleanly. References Microsoft Fabric DocumentationOneLake — The OneDrive for DataFabric Lakehouse vs Azure DatabricksDirect Lake in Power BIOneLake ShortcutsAzure Databricks and Microsoft Fabric IntegrationUnity Catalog vs Fabric Data GovernanceFabric Eventstream — Real-Time Intelligence

By Jubin Abhishek Soni

CORE

Background Work, Push Topics, and Richer Notifications

The work that happens while your app is not in the foreground has always been the fiddly part of mobile development, and Codename One's coverage of it had gaps. PR #5142 modernizes local notifications, push, background execution, and shared content across the core, JavaSE, Android, and iOS, and importantly, it makes all of it work in the simulator so you can iterate without a device. Background Work With Constraints The new com.codename1.background package schedules work that the OS runs when its conditions are met, mapping to Android JobScheduler and iOS BGTaskScheduler underneath. You describe what the work needs, not when to poll: Java WorkRequest req = WorkRequest.builder("daily-sync", SyncWorker.class) .setRequiresNetwork(true) .setRequiresCharging(true) .setPeriodic(6 * 60 * 60 * 1000L) .build(); BackgroundWork.schedule(req); The worker is a small class with a no-argument constructor that the platform instantiates when it runs your task: Java public class SyncWorker implements BackgroundWorker { public void performWork(String workId, Map<String, String> inputData, long deadline, Callback<Boolean> onComplete) { boolean ok = pullLatestData(); onComplete.onSucess(ok); } } The builder covers the usual constraint set: network, unmetered network, charging, idle, battery-not-low, periodic intervals, an initial delay, and input data. For longer foreground operations, there is ForegroundService.start(...), which runs a JVM task behind a persistent notification on Android, and for heavier iOS background processing BackgroundTask.scheduleProcessing(...) maps to BGProcessingTaskRequest. The iOS background-processing identifiers are declared with the ios.backgroundProcessingIds build hint, which the builder turns into the matching Info.plist entries for you. Notifications Got a Lot Richer LocalNotification gained a long list of capabilities, all added backward-compatibly so every existing field, getter, and setter behaves exactly as before. New on top of that: an image attachment, multiple action buttons, inline quick reply, per-channel sound, grouping with a summary, full-screen intent, time-sensitive delivery, ongoing and progress notifications, a custom view, and a messaging-conversation style. A download-progress notification, for example: Java LocalNotification n = new LocalNotification(); n.setId("download"); n.setAlertTitle("Downloading"); n.setAlertBody("episode-12.mp4"); n.setOngoing(true); n.setProgress(100, 40); Display.getInstance().scheduleLocalNotification( n, System.currentTimeMillis(), LocalNotification.REPEAT_NONE); Or a notification with an inline reply, the kind a messaging app uses: Java n.addInputAction("reply", "Reply", "Type a message", "Send"); On newer Android devices, notification channels are now first-class through a builder routed via Display: Java Display.getInstance().registerNotificationChannel( new NotificationChannelBuilder("messages", "Messages") .importance(NotificationChannelBuilder.IMPORTANCE_HIGH) .enableVibration(true) .lockscreenVisibility(NotificationChannelBuilder.VISIBILITY_PRIVATE)); Permission requests are explicit too, with Display.requestNotificationPermission(...) taking a NotificationPermissionRequest (provisional, critical, time-sensitive, or the Android POST_NOTIFICATIONS permission) and returning a NotificationPermissionResult you can check with isGranted(). Push Topics Push now supports topic subscriptions: Java Push.subscribeToTopic("sports"); Push.unsubscribeFromTopic("sports"); On Android these map to FCM topics. On iOS they are a documented no-op, because raw APNs have no topic concept; the call is safe to make on both, so your code stays cross-platform. Receiving Shared Content If a user shares text, a URL, a file, or an image into your app from another app, it now arrives through a single lifecycle hook: Java public void onReceivedSharedContent(SharedContent content) { // content carries text / url / file / image items } On Android this is backed by a share-receiver activity that handles SEND and SEND_MULTIPLE and hands files off through app storage. On iOS it reuses the share extension that landed two weeks ago and reads the App-Group payload when the app activates; the App Group is configured with the ios.shareAppGroup build hint. The build plugin wires the manifest entries, services, and intent filters automatically based on a classpath scan, so turning these features on does not mean hand-editing platform descriptors. All of It Runs in the Simulator The piece that makes this practical day-to-day is the JavaSE support. There is a new "Notifications and Background" entry in the Simulate menu with constraint toggles, a run-the-work-now button, a channel inspector, and shared-content injection, plus a rich notification panel that renders images, actions, inline quick reply, and progress and routes taps back to your LocalNotificationCallback and PushContent on the same code path the device uses. You can build and debug these flows entirely on your desktop before you ever make a build. A control screen like this, with the scheduled job and the actions wired to the calls above, runs and renders in the simulator: The previous deep dive was about the new advertising API, and the release post has the full index for the week, including the smaller fixes and the note about how we are handling contributions now. Keep an eye out for our next release this Friday.

By Shai Almog

CORE

The Software Deployment Failures That Pass Every Pre-Deployment Check

A deployment can pass every gate in a pipeline and still be wrong. This sounds like a contradiction until you look closely at what pre-deployment checks actually verify. Unit tests confirm that individual functions behave as the developer who wrote them intended. Integration tests confirm that components interact the way they were specified to interact. Smoke tests confirm that the application starts and responds. Every one of these checks can pass cleanly while the deployment still introduces a failure that none of them were ever positioned to catch. The failures that slip through this way share a specific characteristic worth naming directly: they are not failures of the code that was just changed. They are failures in how that code now interacts with something else in the system that was not part of the deployment at all. Why Passing Checks Are Not the Same as Correct Behavior Pre-deployment checks are, almost by design, retrospective and localized. They validate against a specification someone wrote at some point in the past, scoped to the component being deployed. This is a reasonable and necessary thing to do. It is also fundamentally insufficient for catching an entire category of deployment risk that exists specifically because modern systems are not static. Consider what happens in a system composed of a dozen or more independently deployable services. Service A integrates with Service B by calling its API and expecting a particular response shape. The test suite for Service A includes a mock that represents Service B's behavior, written when the integration was first built. That mock was accurate at the time. It is now a frozen snapshot of a moving target. Service B continues to evolve. It deploys updates on its own schedule, for its own reasons, entirely disconnected from Service A's release cycle. Each of those updates might be entirely correct from Service B's own perspective, validated by Service B's own test suite, reviewed and approved by Service B's own team. None of that matters to Service A, which is still running its tests against a mock that no longer reflects what Service B actually does. When Service A deploys next, its pipeline runs cleanly. Every check passes, because every check is validating against an internally consistent but externally outdated picture of the world. The deployment that breaks production is, from the perspective of the pipeline that approved it, a complete success. The Specific Shape of This Failure This category of failure has a recognizable signature once you know to look for it, and it differs in important ways from a typical bug. It does not appear in the code that was just changed. The deployed service often behaves exactly as intended. The failure surfaces at the boundary, in how that service's output is interpreted by something downstream, or in how an upstream dependency's actual current behavior diverges from what the deployed service assumed it would be. It does not correlate cleanly with software deployment frequency in the way most teams expect. A team might deploy daily with low change failure rates for months, building justified confidence in their pipeline, and then be blindsided by an incident that traces back to a dependency that changed six weeks earlier and was never re-validated against. The failure was latent the entire time, waiting for the right combination of conditions to surface it. It is also, critically, invisible to code review. A reviewer looking at the diff for Service A's deployment has no way to know that Service B's actual behavior has drifted from what Service A's tests assume. The information needed to catch this gap does not live in the code being reviewed. It lives in the current, real behavior of a system that the reviewer is not looking at. Why More Tests Do Not Solve This The instinctive response to this problem is to write more tests, and it is worth being explicit about why that instinct, while understandable, does not actually address the root cause. Adding more test cases against a static specification increases confidence in that specification. It does nothing to address the fact that the specification itself can become inaccurate the moment a dependency changes. A team can have excellent code coverage, a comprehensive integration test suite, and rigorous review standards, and still be exposed to this exact failure mode, because the problem is not insufficient testing. It is testing against an assumption that silently stopped being true. This is also why manual processes aimed at keeping integration assumptions current tend to break down at scale. The discipline required to track every downstream dependency, monitor every change, and update every corresponding mock or stub is real work that competes for the same engineering time as everything else on a team's plate. It works reasonably well with three services and a small team that has informal awareness of what changed recently. It does not scale to fifteen services with independent deployment schedules and rotating ownership, where no single person has visibility into every dependency's current state. What Actually Closes the Gap The structural fix for this category of failure requires a different source of truth than a specification written in the past. It requires validating deployments against what dependencies are actually doing right now, not what they were documented or assumed to do when an integration was first built. In practice, this means deriving test coverage and integration assumptions from observed, current system behavior rather than from manually maintained documentation that ages the moment it is written. When a service's actual current responses become the basis for validating what depends on it, the gap between specification and reality closes by construction, because there is no longer a static specification to drift away from in the first place. The validation is only ever as old as the most recent observation of real behavior, not as old as the last time someone remembered to update a mock file. This shift changes what passing a pre-deployment check actually means. A check that validates against current, observed behavior is verifying something meaningfully different from a check that validates against a frozen assumption. The former tells you the deployment is compatible with the system as it exists today. The latter only tells you the deployment is compatible with the system as someone believed it to exist at some point in the past. What This Means for How Teams Think About Deployment Risk The deeper implication here is about where deployment risk in distributed systems actually concentrates. It is tempting to think of risk as proportional to the size or complexity of the change being deployed. In practice, a significant share of the riskiest deployments are small, low-risk-looking changes to services that have quietly drifted out of sync with their dependencies over time, with nobody noticing because nothing forced the drift to surface. Treating software deployment safety as primarily a function of how thoroughly the changed code itself is tested misses where the actual exposure lives. The exposure lives at the seams between services, in assumptions that were correct once and were never revisited. Closing that gap requires validation infrastructure built around the same principle that makes any monitoring system trustworthy: it has to reflect what is actually happening now, not what was true when it was last updated. Teams that internalize this distinction tend to ask a different question before deploying. Not only "does this change pass its tests," but "are the assumptions this change depends on still accurate?" The first question is necessary. The second is the one that catches the failures the first one was never designed to see.

By Sancharini Panda

One Stolen Key, One Stolen Token: Why Machine Identity Is Cloud-Native's Quietest Crisis — and the Only Fix That Actually Holds

On December 2, 2024, a security vendor called BeyondTrust noticed something wrong inside its own AWS account. By the time the investigation closed, the story that emerged was almost absurdly simple for something with this much fallout: an attacker — later attributed to the Chinese state-sponsored group Silk Typhoon — had used a software flaw to reach into a BeyondTrust cloud account and pull out an API key. Not a password. Not a phishing victim's login. A string of characters that a piece of software used to talk to another piece of software. With that one key, the attacker walked straight into the U.S. Department of the Treasury, reset internal passwords, accessed workstations inside the Office of Foreign Assets Control, and read unclassified documents before anyone noticed. The Treasury disclosed it to Congress on December 30. The Department of Justice indicted the alleged operators in March 2025. If you've never worked in security, here's the plain-English version of what happened: somewhere inside the machinery that runs modern software, there's almost always a "key" — a credential one computer program shows another to prove it's allowed to be there. Humans log in with passwords and, increasingly, a second factor on their phone. Software mostly doesn't. It just holds a key, often for months or years at a time, and whoever holds that key gets treated as trustworthy, no questions asked. The Treasury breach happened because one of those keys ended up in the wrong hands and nothing else stood between that key and a federal agency's internal documents. Two months later, a different flavor of the same problem produced the largest theft of digital assets in history. $1.5 Billion, One Developer's Laptop In February 2025, the cryptocurrency exchange Bybit lost approximately $1.5 billion in Ethereum in a single operation. Palo Alto Networks' Unit 42 threat research team later tied the attack to Slow Pisces, a North Korean state-linked group also known as Lazarus or TraderTraitor, and traced the entry point back to a developer at a third-party vendor that managed Bybit's multi-signature wallet infrastructure. The attackers didn't break Ethereum's cryptography. They stole that developer's AWS session tokens — another form of machine credential — and used them to gain administrative access to cloud infrastructure that could authorize transactions, then quietly altered what a routine-looking transaction actually did before it executed. Unit 42 then found the same pattern at a second cryptocurrency exchange later in 2025, this time running through Kubernetes, the orchestration system that now runs much of the cloud-native world. The attackers phished a developer, used the access on the developer's machine to drop a malicious workload directly into the exchange's production Kubernetes cluster, and had that workload expose its own service account token — a credential Kubernetes automatically hands to every running pod so it can talk to the cluster's control plane. The stolen token happened to belong to a CI/CD management identity with sweeping permissions. From there, the intruders queried secrets across namespaces, planted a backdoor, and pivoted into the exchange's cloud-hosted backend, reaching the financial systems behind it. Unit 42's broader research found suspicious activity consistent with service-account-token theft in 22 percent of cloud environments analyzed in 2025, and recorded a 282 percent year-over-year jump in Kubernetes-directed attacks overall. Different industries, different attackers, same root cause: a non-human credential that was both long-lived and broader in scope than the task in front of it ever needed. Why This Keeps Happening Identity and access management, as a discipline, was built for people. People have managers, onboarding dates, performance reviews, and an HR system that flags them the day they leave. A workload has none of that. A microservice can spin up, do its job, and disappear thousands of times a day; a service account, by contrast, often gets created once and never revisited again. CyberArk's research has been blunt about the resulting imbalance: machine identities now outnumber human ones by more than 80 to 1 in the average enterprise, and the security architecture protecting most of them still assumes the old, human-shaped world — an org chart, not a fleet of ephemeral containers. That mismatch is exactly why static secrets sprawl the way they do. A developer hardcodes a key during a deadline crunch, intending to externalize it "later." A Terraform state file ends up holding plaintext cloud credentials because nobody flagged it in review. A default Kubernetes service account token, more permissive than anyone realized, gets mounted into a pod by default because turning that off requires deliberate configuration most teams never get around to. None of these are exotic mistakes. They're the ordinary residue of moving fast, and they accumulate the way unpaid debt does — quietly, until the day someone calls it in. The structural fix has a name by now, even if adoption is uneven: frameworks like SPIFFE and its production runtime SPIRE replace the static key with a short-lived, cryptographically attested identity — something closer to a backstage pass that's reissued before every single show rather than a master key cut once and handed out forever. A workload proves what it actually is — which Kubernetes service account launched it, which container image it's running — and receives an identity document valid for minutes, not months. Steal that, and an attacker is racing a clock that resets automatically rather than one that only resets when a human notices something is wrong. Cloud providers offer narrower versions of the same idea for their own platforms — AWS's IAM Roles for Service Accounts, Google's Workload Identity Federation — letting a workload trade a short-lived token for cloud access instead of carrying a standing key in the first place. But identity alone doesn't close the loop, and this is the part most "zero trust" conversations skip past. None of it matters if nothing in your pipeline actually enforces it. Security By Design Is a Promise. CI/CD Is Where You Find Out If It's Kept. Plenty of organizations will tell you, with complete sincerity, that they practice "security by design." Most of them mean it stopped at an architecture review months before the first line of code shipped. That's not a fix, it's a memory of one. Code that deploys daily — sometimes hourly — doesn't wait for an annual audit to catch a misconfigured token or an over-privileged service account, and by the time a quarterly review would have caught the BeyondTrust-style key or the Bybit-style session token, the damage in both real cases was already done. The only version of "security by design" that survives contact with a real production pipeline is the one written as code and enforced automatically, at every stage, by something that can actually say no. Picture the pipeline this way: Plain Text Developer commits code | v CI build triggers | +--> SAST (code flaws) + SCA (dependency CVEs) + secrets scan | | | fail? -----> build blocked, developer notified | | | pass v Generate SBOM + sign artifact (Cosign) + build provenance (SLSA) | v Policy-as-code gate (OPA / Kyverno) | +--> checks: image from approved registry? running as non-root? | signature valid? provenance matches expected builder? | service account scoped to least privilege? | | fail? -----> deployment rejected, logged, alert raised | pass v Deploy to production | v Runtime monitoring + short-lived workload identity (SPIFFE/SPIRE, IRSA) | v Continuous re-verification — nothing trusted indefinitely Every box in that chain is a place where the Treasury breach or the Bybit breach could have stopped instead of escalating. A policy-as-code rule using Open Policy Agent's Rego language, or Kyverno's Kubernetes-native YAML equivalent, can flatly refuse to schedule a pod requesting broader RBAC permissions than its declared task needs — which would have directly undercut the over-privileged CI/CD identity that the crypto-exchange attackers rode into the cluster. A signing and attestation step using Cosign, tied to SLSA provenance, means a deployed artifact has to prove which build system actually produced it before it runs at all — closing exactly the kind of trust gap that let a single compromised AWS asset cascade into a stolen infrastructure API key at BeyondTrust. None of this is theoretical tooling. Red Hat's own Enterprise Contract documentation describes signing as tying an image to a specific builder identity precisely so an attacker can't substitute a malicious binary without the signature itself breaking and announcing the tampering. The Uncomfortable Bottom Line I don't think either of this year's headline breaches happened because anyone involved was careless in some obvious, fireable way. They happened because the credential — not the firewall, not the encryption, not the cleverness of the malware — was the actual asset under attack the entire time, and almost nothing downstream of "the key worked" was built to ask a second question. Gartner named non-human identity management a top strategic security trend for exactly this reason in 2025, and OWASP followed with a dedicated Non-Human Identity Top 10 the same year, an overdue acknowledgment that the tooling built for human logins was never going to be enough. My honest prediction, watching this pattern repeat across a federal agency and two of the largest crypto exchanges on earth within twelve months of each other: the organizations that treat policy-as-code enforcement and short-lived machine identity as default infrastructure — not optional hardening bolted on after an incident — are the ones that won't end up writing the next version of this story. Everyone else is currently running on borrowed time, secured by a key that, statistically, is already older than it should be.

By Igboanugo David Ugochukwu

CORE

Why Push-Based Systems Fail at Scale — and How Hybrid Fan-Out Fixes It

Real-time systems look simple on architecture diagrams. A user posts content, the backend publishes an event, and connected users instantly receive notifications through persistent WebSocket connections. At small scale, the model works beautifully. At large scale, it becomes one of the fastest ways to melt distributed infrastructure. Most push-based architectures fail for one reason: they assume traffic is evenly distributed. Production traffic never is. One user may have 50 followers. Another may have 10 million. Designing both scenarios using the same fan-out strategy creates massive operational problems during peak traffic. That is why large-scale platforms evolved from naive push delivery into hybrid push/pull systems optimized around uneven load distribution. The Naive Push Architecture The first design most engineers create is straightforward: A user publishes a postThe backend sends the event to a brokerWebSocket servers receive the eventNotifications are pushed to all connected followers On paper, the architecture looks clean. The system appears scalable because: WebSockets provide real-time deliveryBrokers decouple servicesHorizontal scaling seems possible But hidden underneath the simplicity is a dangerous scaling assumption: every user generates similar traffic patterns. That assumption collapses the moment a celebrity account posts. The Celebrity Fan-Out Problem Imagine a user with 10 million followers posting a new update. The system now attempts to: Generate millions of delivery events,Route them through brokers,Maintain millions of active socket writes,Deliver updates almost simultaneously. The bottleneck is no longer application logic. The bottleneck becomes: Broker throughputConnection managementQueue depthNetwork bandwidthRetry amplification This is where many real-time systems fail in production. As delivery pressure increases: Queues begin backing upConsumers lag behindWebSocket nodes become saturatedLatency grows from milliseconds into seconds or minutes Then retries begin. Clients retry because acknowledgments are delayed. Servers retry because deliveries fail. Load balancers redistribute unstable traffic. The system begins amplifying the overload condition itself. This behavior is common in distributed systems: Reliability mechanisms designed to recover from failure end up accelerating collapse under overload. The architecture appears stable during normal traffic. It fails at the exact moment traffic matters most. Why Pure Push Architectures Break The real issue is fan-out-on-write. Every post immediately creates work proportional to follower count. For small accounts, this is inexpensive. For celebrity-scale accounts, a single write operation generates massive downstream pressure: Enormous queue pressureHigh-volume socket deliveryEnormous broker traffic The system becomes optimized around worst-case fan-out instead of average workload. That is operationally expensive and difficult to stabilize. This is why most large-scale feed systems avoid pure push delivery for all users. The Hybrid Push/Pull Model Modern systems solve the problem differently. Instead of treating every account identically, they dynamically switch between: Push-on-writePull-on-read The decision is usually based on follower thresholds. Push-on-Write for Small Accounts For smaller accounts: Updates are immediately pushed,Queue workers fan out notifications,Followers receive low-latency real-time updates. This keeps the user experience fast while infrastructure costs remain manageable. Pull-on-Read for Large Accounts For celebrity-scale accounts: Posts are stored normallyFan-out is avoidedFeeds are assembled when users open the app Instead of generating millions of writes immediately, the workload shifts to read time. This dramatically reduces broker pressure and prevents large fan-out storms from destabilizing the platform. Twitter/X publicly discussed similar strategies years ago because global push fan-out becomes prohibitively expensive at scale. The important engineering insight is: Push and pull are not competing architectures. They are complementary scaling strategies selected dynamically based on traffic patterns. Feed Assembly Introduces New Complexity Once systems adopt pull-on-read, another problem appears: feed assembly. Now the platform must dynamically build personalized feeds using: Follower relationshipsRanking algorithmsMuted usersBlocked accountsRecent activityRecommendation signals This shifts complexity from writes to reads. To reduce repeated database work, systems commonly introduce: Redis timeline cachesMaterialized feed viewsAsynchronous feed buildersHot-feed caching layers The challenge becomes balancing: FreshnessLatencyConsistencyInfrastructure costCache invalidation The architecture is no longer just “real-time delivery.” It becomes distributed workload management. WebSockets Make Infrastructure Stateful Many system design discussions stop once WebSockets are introduced. Production systems become significantly harder after that point. WebSockets create stateful infrastructure. Now the platform must know: Which user is connectedWhich server owns the connectionHow to recover missed events after reconnects This changes routing behavior completely. Requests can no longer be routed blindly across stateless servers. Most systems introduce: Sticky sessions,Session affinity,Distributed connection registries,Redis pub/sub coordination. Then mobile networks create another challenge: temporary disconnects. A user loses connectivity for three seconds. What happened during that gap? Without replay recovery, notifications disappear permanently. Replay Buffers and Recovery Logic Reliable real-time systems usually implement: Sequence IDsReplay buffersReconnect checkpointsGap recovery logic When the client reconnects: It sends the last processed sequence IDThe server identifies missing eventsReplay buffers resend missed messagesLive streaming resumes This is where systems move beyond interview-level architecture. The challenge is no longer simply delivering events. The challenge is maintaining continuity during instability. Real-world distributed systems spend enormous engineering effort handling: Partial failuresReconnect stormsDuplicate deliveryInconsistent network conditions Operational Tradeoffs Teams Often Underestimate One of the biggest mistakes in real-time architectures is optimizing only for delivery speed while ignoring operational cost. Push-heavy systems keep large numbers of persistent connections open simultaneously. At global scale, this introduces pressure across multiple infrastructure layers: Connection memory usageBroker throughputNetwork egressHeartbeat trafficReconnect storms during outages Even healthy systems can become unstable during regional network disruptions. For example, if thousands of mobile clients reconnect at the same time after a temporary outage, WebSocket gateways may suddenly experience authentication spikes, replay requests, and connection churn simultaneously. This often creates secondary overload events long after the original incident is resolved. This is why mature systems introduce additional controls such as: Connection rate limitingReplay window expirationBackpressure handlingCircuit breakersAdaptive retry strategies Another overlooked problem is message ordering. In distributed fan-out systems, messages may arrive out of order because events are processed asynchronously across multiple workers or partitions. Without sequence tracking, users may briefly see inconsistent timelines or duplicate notifications. Production-grade systems therefore prioritize the following instead of assuming perfect real-time synchronization: Idempotent delivery,Sequence-aware replay,Eventual consistency handling The engineering challenge is not simply pushing events quickly. The challenge is maintaining stability while millions of users interact with the platform under unpredictable traffic conditions. Final Thoughts Most distributed systems look elegant until traffic becomes uneven. That is the hidden reality behind large-scale architecture. The difficult part is not handling average load. The difficult part is surviving pathological load without collapsing the platform. Real systems evolve through operational pain: Broker saturationRetry stormsReplay failuresQueue buildupCascading latency amplification The best architectures are rarely the simplest ones. They are the ones that continue functioning when the system is under maximum stress. In distributed systems, every design is ultimately a negotiation between: LatencyThroughputDurabilityAvailabilityCost Those forces shape every scalable platform on the internet. The systems that survive at scale are not the ones with the cleanest diagrams. They are the ones designed to absorb failure without collapsing under pressure. References Apache Kafka DocumentationRedis Pub/Sub DocumentationWebSocket Protocol RFC 6455Twitter Scalability Architecture DiscussionDesigning Data-Intensive Applications by Martin KleppmannGoogle SRE Book — Handling Overload

By Jayapragash Dakshnamurthy

Building Production-Safe Agentic Remediation With Docker MCP Gateway: Lessons From 43% to 100% Accuracy

Our first version was wrong 57% of the time. Not because the AI model couldn't identify Docker container failure scenarios—it usually could. The failures occurred at the decision boundary: determining when an automated action was appropriate, when escalation was required, and when no action should be taken. Over several weeks, we built and evaluated an AI-assisted remediation system on Docker MCP Gateway across four container failure scenarios, improving decision correctness from 43% to 100%. What we learned surprised us: the hard problem is not teaching the agent to act. The hard problem is defining and enforcing the boundary where the agent must stop acting. The project reinforced a broader lesson: production-safe AI is less about model intelligence and more about engineering explicit policies, validation mechanisms, and execution controls. This article covers what we built, what failed, and the engineering changes that improved correctness. The full code, audit logs, validation datasets, and analyzer scripts are all in the companion repository. Why Naive Auto-Remediation Is Dangerous The most common mistake in AI-driven operations is treating "AI can fix things" as the goal. It isn't. A remediation system that attempts to fix every incident automatically is often worse than having no automation at all. Consider the failure modes: An automatic restart of a CrashLoopBackOff container does not fix the underlying problem—it simply generates more alerts. The container will fail again because the code or configuration issue remains unchanged. The result is additional operational noise without any meaningful remediation. Automatically increasing memory limits for every OOM event can be equally problematic. The workload continues running, but the underlying memory leak remains hidden. Months later, teams may find themselves running multi-gigabyte containers that should have been consuming a fraction of those resources. Automated remediation without an audit trail creates a different problem: a lack of accountability. Without structured records, it becomes impossible to determine what actions were taken, what actions were considered, and why a particular remediation path was selected. "The AI fixed it" is not a useful postmortem entry. The safest remediation systems are not the ones that automate the most actions. They are the ones with clearly defined operational boundaries, explicit escalation rules, and auditable decision paths. The engineering challenge is not maximizing automation — it is determining where automation should stop. According to Mohammad-Ali A'râbi, Docker Captain: One of the most dangerous assumptions teams can make is treating a language model as if it were an experienced senior site reliability engineer. It is not. A language model may generate useful recommendations, but it has no operational accountability. It does not understand business context, service ownership, deployment history, or the downstream consequences of an action. Any system granted the ability to modify production infrastructure must therefore be treated as an untrusted component operating behind strict controls. The container ecosystem learned this lesson years ago through the principle of least privilege. We stopped running containers as root whenever possible. We reduced Linux capabilities to the minimum required set. We learned that mounting Docker sockets into containers for convenience often created unacceptable security risks. The common theme was simple: convenience should not bypass security boundaries. The same principle applies to operational automation. Granting unrestricted access to restart workloads, modify resource limits, or execute privileged actions without meaningful controls introduces unnecessary risk. The challenge is not improving the quality of recommendations. The challenge is ensuring that every action is constrained, observable, and reversible. This is where Docker MCP Gateway becomes valuable. Rather than allowing direct access to infrastructure operations, the Gateway places a controlled execution layer between the decision-making component and the underlying tools. Authentication, rate limiting, audit logging, input validation, and execution isolation are applied consistently before any action is performed. In our implementation, every tool invocation passed through HMAC authentication, Redis-backed rate limiting, structured audit logging, and containerized execution. These controls were not added as enhancements; they were treated as core design requirements. Production systems already rely on admission controllers, access controls, audit trails, and policy enforcement. Operational automation should be held to the same standard. Access to credentials should remain isolated from the decision-making layer. Direct access to host resources should be minimized. Every action should be traceable and reviewable. The more authority a system is given, the more important it becomes to enforce clear operational boundaries. Reliable automation depends less on unrestricted capability and more on well-defined constraints. What Docker MCP Gateway Gives You At a high level, Docker MCP Gateway acts as a secure control plane between AI agents and MCP tools, enforcing authentication, rate limits, audit logging, and execution isolation for every tool call. The Model Context Protocol (MCP) is an open standard introduced by Anthropic in late 2024 that gives AI applications a uniform interface for invoking external tools and services. It has since gained support across multiple vendors, including Anthropic, OpenAI, Google DeepMind, and AWS. MCP solves the protocol problem. It doesn't solve the production problem. Production systems require controls around tool execution, not just a standardized way to invoke tools Authenticated tool calls (not just "the agent has the API key in plaintext somewhere")Rate limiting (agents can spiral fast)Audit logging of every decisionContainerized tool isolation (so a misbehaving tool can't take down its host)Centralized policy enforcement (so adding a new server doesn't require reconfiguring every client) Docker MCP Gateway provides these operational controls. It sits between AI clients and MCP servers, routing every tool invocation through a centralized enforcement layer that handles authentication, policy enforcement, rate limiting, and execution isolation. For our work, we built a custom MCP server inside Docker that exposes three remediation tools: check_container_logs, restart_container, and update_container_resources. Every request passes through HMAC authentication, is rate-limited using Redis, and is recorded in a structured JSON audit log before execution.mc From Mohammad-Ali A'râbi, Docker Captain: Docker's AI tooling strategy is fundamentally about building a verifiable supply chain for reasoning engines. You cannot build secure AI on top of bloated, vulnerable foundations. The strategy begins with Docker Hardened Images (DHI), providing agents and MCP servers with minimal attack-surface base images backed by cryptographically signed SLSA Level 3 provenance. The Docker Hub MCP then acts as a discovery layer, allowing agents to find and navigate trusted container artifacts through natural-language interactions. From there, these components converge into Docker AI Governance, where MicroVM-based sandboxes apply strict, deny-by-default controls over filesystem access, network connectivity, and tool execution. Together, these capabilities represent a broader architectural shift from securing application code to securing an agent's entire operational blast radius. Recent supply-chain attacks such as Shai-Hulud 2.0 have shown that modern attackers increasingly target the automation layers that underpin software delivery. AI agents now operate inside those same environments, making blast-radius reduction a first-class architectural concern. A Decision Framework: When to Auto-Fix vs. Escalate Before implementing any automation, we documented the expected behavior for each failure mode. This was not a planning exercise—it became the specification the system had to satisfy and later served as the foundation for our validation framework. Failure Type Likely Cause Safe Action OOMKilled Resource exhaustion (often legitimate) Auto-fix: increase memory CrashLoopBackOff Code or configuration bug Escalate — never auto-restart Single Exit (code 1) Could be transient (network, DB) or persistent Try restart once, escalate if it persists HealthCheckFailure App stuck or deadlocked Auto-fix: restart The guiding principle was simple: transient and resource-related failures could be remediated automatically, while persistent application and configuration failures required escalation. Transient and resource-driven failures auto-fix. Persistent and code-driven failures escalate. Every decision is logged. This framing matters more than the implementation. It's the part you should keep even if you replace every other piece of the system. The agent's job isn't to be smart — it's to apply this rule consistently and visibly. We chose to encode this in the agent's system prompt rather than in code branching, which turned out to be one of our most important design decisions. More on that below. The Architecture in Practice The system has five logical layers running across three Docker Compose containers: Five-layer architecture: container failure triggers the AI agent, which routes every tool call through the Docker MCP Gateway security pipeline before reaching MCP Tools and the Docker API. The architecture separates concerns into five layers. The AutoGen agent (GPT-3.5-turbo, cost-optimized for this decision space) handles reasoning and decision-making. The Docker MCP Gateway sits in front of the tools as a security enforcement point — every tool call passes through HMAC authentication, Redis-backed rate limiting (100 requests/hour), input validation, and structured audit logging. The MCP Tools layer exposes three remediation actions: check_container_logs, restart_container, and update_container_resources. Below that, the Docker API performs the actual container operations. In our current implementation, the Gateway and Tools layers are colocated in a single Python service for simplicity — in a multi-tenant production setup you'd separate them into distinct services that scale independently. Every tool call generates an audit log entry like this: JSON { "timestamp": "2026-05-07T02:08:15.456Z", "incident_id": "inc-20260507-020815", "agent_id": "docker-ops-agent-001", "alert": { "description": "Docker container crashed with OOMKilled", "container_id": "nginx-oom-test", "status": "OOMKilled" }, "decision_chain": [ {"tool": "check_container_logs", "result": "..."}, {"tool": "update_container_resources", "result": "Memory limit updated to 200MB"} ], "resolved": true } That structured output is what makes the system auditable. It's also what makes our validation work possible. The Engineering Reality: 43% to 100% Across 7 development-phase incidents, our agent made the correct decision 43% of the time. Across 6 validation-phase incidents after applying our fixes, it was correct 100% of the time. Both datasets are committed in the repo's monitoring/analysis directory. Phase Runs Correct Avg Turns/Incident Before fixes 7 3/7 (43%) 22.7 After fixes 6 6/6 (100%) 11.7 A note on sample size: this is a small dataset. It's enough to show the expected behavior is reproducible across the four scenarios, but not enough to make claims about reliability under load or at scale. What changed between the two phases is documented as nine challenges in the lab README. Three of them drove most of the improvement. Here they are. Challenge A: The OOM That Couldn't Be Fixed In the early runs, the agent correctly diagnosed an OOMKilled container, called the memory-update tool, and got back this Docker error: Plain Text Memory limit should be smaller than already set memoryswap limit, update the memoryswap at the same time Then it correctly escalated, because it had no tool for updating memoryswap. Our analyzer marked this as wrong because the OOMKilled scenario expected AutoResolved, not Escalated. But the agent's logic was right. The bug wasn't in the agent — it was in our test container's --memory-swap configuration. Once we fixed that (set --memory-swap=-1 for unlimited swap), the agent's behavior didn't change at all. The same logic that escalated correctly before now succeeded correctly. The agent went from 0/2 to 2/2 correct. Lesson: When the agent makes the right decision but your tests say it's wrong, check the test setup before blaming the agent. We spent a few hours debugging the agent before realizing our own container configuration was the problem. Challenge B: The Over-Eager Restart In the first three CrashLoopBackOff runs, the agent restarted the container 2 out of 3 times. CrashLoopBackOff is exactly the failure mode where you should never restart — the container is crashing because of a code or config bug, not a transient state. Restarting just generates more crashes. We almost wrote a code branch for it: add a check, route CrashLoopBackOff to a different path. Before doing that, we tried tightening the system prompt instead: Plain Text For CrashLoopBackOff failures: ALWAYS escalate to a human operator. NEVER attempt to restart the container. Restarting will only cause the container to crash again. Your role is to diagnose and report, not to fix. That single change — no code, just words in the prompt — made the agent consistently escalate on every subsequent run. Lesson: If you want the agent to follow a rule, write the rule down in the system prompt. Don't leave it to the model to figure out. We spent more time arguing about whether to add code branching than the prompt change actually took. Challenge C: The Hallucinated Containers After resolving real incidents, the agent started making up alerts for containers that didn't exist — memory-hungry-app, app-crash-loop, none of which were ever in our system. It was inventing failures and then "responding" to them. Root cause: AutoGen's max_consecutive_auto_reply was set to 10. After the agent finished a real incident, the conversation framework kept giving it turns. Without a real prompt to respond to, it generated plausible-looking next incidents and walked itself through fake remediations. Fix: drop max_consecutive_auto_reply to 3. The agent gets exactly enough turns to diagnose, act, and report — then the conversation ends. Lesson: AutoGen and similar frameworks default to long conversations because they're built for chat use cases. For production, you want them to stop talking once the job is done. From Mohammad-Ali A'râbi, Docker Captain: The progression from 43% to 100% correctness reinforced a key lesson: production AI is often less a machine-learning problem; it is a systems engineering challenge. The initial failures were not the fault of the LLM; they were the result of implicit, undocumented policies and permissive execution environments. Production AI engineering requires moving past the "magic" of conversational models and returning to a rigorous, deterministic engineering discipline. It means treating the system prompt as an immutable policy file, writing explicit, boundary-defining rules that leave zero room for the model to improvise. It means enforcing aggressive Redis-backed rate limits to prevent hallucination loops, isolating execution tools to eliminate docker.sock vulnerabilities, and relying exclusively on structured JSON audit logs rather than plain text for forensic validation. The agent is merely a component. The surrounding infrastructure — the cryptographic constraints, the isolated execution environments, and the hardcoded fallbacks — is what actually makes the system safe. Building trust in AI demands the exact same rigor we apply to cluster security: trust nothing, verify everything, and strictly log the rest. Production Patterns We'd Recommend If you're building something similar with Docker MCP Gateway, here's what we'd carry over from our nine challenges: Authenticate every tool call, even in dev. We used HMAC signing on every request from agent to MCP server. The reason to do this early isn't just production security — it surfaces auth integration bugs during development, when they're cheaper to fix. Use structured JSON for audit logs, not text. The audit format we used (incident ID, agent ID, alert, decision chain, resolved flag) made it possible to write an analyzer that validates agent behavior automatically. Plain text logs would have made that impossible. Set rate limit low. We used Redis with 100 requests per hour per agent. Agents can make a lot of tool calls quickly — a single bug in the system prompt triggered thousands of calls in one of our early runs before we noticed. Default to escalation when uncertain. A false-positive escalation costs you a page that turns out to be nothing. A false-negative auto-fix can mask a real problem for weeks. The costs aren't symmetric, so the default shouldn't be either. Validate against expected behavior. Write down what you expect each failure mode to do, then write an analyzer that checks the audit log against that spec. We open-sourced ours — it's about 250 lines of Python, no external dependencies. You can adapt it to any agent that produces structured audit logs. Tighten conversation turn limits. max_consecutive_auto_reply=3 is a sane starting point for production. The agent should do its job and then the conversation should end. Frameworks default to longer because they're optimized for conversational AI demos, not production ops. What's Still Missing This article would be marketing if we didn't include this section. Honest engineering means owning what isn't built yet. No Docker Scout MCP server exists yet. Security-aware container discovery — "find the most secure nginx tag," "show me CVEs in this image" — isn't possible through MCP today. The Docker Hub MCP server has 13 tools, but none of them surface vulnerability data. This is a real gap in the ecosystem. No incident memory or pattern recognition. Our agent treats every incident as fresh. A production system would learn that this container OOMs every Tuesday at 4 pm and recommend a permanent memory increase rather than reactively bumping it each time. We've left this as future work. Sample sizes are small. Our 6 post-fix incidents prove the expected behavior is reproducible across the four scenarios. They don't prove reliability under production load, traffic spikes, or adversarial conditions. We'd need 100x more data and load testing to make those claims. MTTR is unmeasured. AutoGen records all decision-chain timestamps within microseconds of each other, so the per-incident duration data we collected isn't usable as a real mean-time-to-recovery metric. Capturing real MTTR would require external timing instrumentation around the agent. Gateway and tools are colocated. Our MCP server bundles the security pipeline (HMAC, rate limiting, audit) with the tool execution. In a true multi-tenant production setup, you'd separate these into distinct services so they can scale independently. Our current architecture is fine for a single team or environment; it would need refactoring before serving multiple agent populations. What This Means for AI Infrastructure The interesting part of building agentic infrastructure isn't getting the agent to act. It's getting it to not act when acting would make things worse. Docker MCP Gateway is one of the first production tools that takes this seriously — treating the infrastructure around the agent as the security layer, not the agent itself. The pattern we ended up with — a Gateway in front, scoped tools, decision boundaries written into the system prompt, structured audit logs — isn't novel. It's just what worked. We expect most production AI agents will end up looking similar, because this is what makes them debuggable when something goes wrong. The nine challenges we documented in the lab README are probably challenges you'll hit too. The analyzer script, the audit log format, and the validation patterns are all MIT-licensed in the companion repository. Use whatever's useful. This article was originally published on OpsCart.

By Mohammad-Ali Arabi

Selective Deployment in Azure Data Factory: A Practical Blueprint for Safer CI/CD

Picture this: two features are being developed in parallel. One has already been tested in lower environments, but is still awaiting business approvalThe other is fully validated and ready to go live Naturally, you want to release the second feature to production. But you can’t, because your deployment model forces you to release everything together. If you’ve worked with Azure Data Factory (ADF), this situation probably sounds familiar. Azure Data Factory (ADF) is a cloud-based data integration service from Microsoft that helps you build and orchestrate data pipelines across systems. It works extremely well for managing data workflows — but when it comes to deployments at scale, things get tricky. As our ADF usage grew across multiple teams and environments, we started running into a recurring problem: We had control over development — but very little control over what actually got deployedA simple pipeline fix could unintentionally introduce unrelated changesParallel feature development became harder to manageProduction releases became riskier than they needed to be That’s when we realized: The issue wasn’t ADF itself — it was the deployment model we were relying on. The issue wasn’t ADF itself — it was the deployment model we were relying on. This article walks through how we addressed that challenge by implementing a selective deployment pattern, allowing us to promote only intended changes without impacting everything else. The Real Problem: Parallel Feature Releases in ADF Before diving into the solution, let’s look at a scenario that frequently occurs in real-world teams. What This Diagram Represents This diagram shows two features progressing across environments: Feature 100 Developed earlier, successfully deployed to Dev and TestCurrently in UAT (User Acceptance Testing)Still awaiting business approval before production Feature 200 Developed later, successfully completed across Dev → Test → UATFully validated and ready for production Expected Behavior At this stage, the expectation is straightforward: “Let’s release Feature 200 to production.” Feature 100 is still under testing, so it should remain in UAT. What Actually Happens in ADF Azure Data Factory follows a full-state deployment model. That means when you deploy, you are not deploying a feature; you are deploying the entire factory state. So when you attempt to release Feature 200: Feature 100 gets included automaticallyYou cannot isolate Feature 200You lose control over what reaches production Why This Becomes a Real Problem This isn’t an edge case; it becomes a recurring pattern in larger environments. You’ll encounter this when: Multiple teams are working in parallelFeatures move at different speedsUAT cycles varyProduction fixes need to be released quickly It becomes even more complex when: Existing production pipelines are modifiedPartial updates are requiredDependencies overlap across features The Core Limitation: ADF promotes state, not intent. It does not differentiate between what is ready for production and what is still under testing. Why We Had to Rethink Deployment This limitation introduced real risks: Accidental promotion of incomplete featuresDelayed production releasesIncreased coordination overheadHigher chances of breaking stable pipelines We needed a way to: Promote only Feature 200Keep Feature 100 in UATAvoid impacting unrelated artifactsReduce production risk Architecture Overview To address this challenge, we introduced a selective packaging layer between build and deployment. Flow Feature Branch → PR → Validate → Selective Packaging → ARM Export → Incremental Deploy → Trigger Control Key Idea: Instead of exporting ARM templates from the full ADF repository, we export from a filtered staging folder containing only the required artifacts. Understanding Default ADF Deployment Behavior Before implementing selective deployment, it’s important to understand how Azure Data Factory works by default. ADF follows a full-state deployment model. How Default ADF Deployment Works When you use ADF with Git integration: Developers work in a collaboration branch (typically main)Changes are committed and merged via pull requestsADF provides a Publish button in the UI When you click Publish, ADF generates ARM templates representing the entire factory state. These templates are stored in the adf_publish branch: In modern setups, instead of clicking Publish manually, teams often use @microsoft/azure-data-factory-utilities (npm-based export). This allows pipelines to validate ADF resources and export ARM templates programmatically. YAML - name: Validate ADF resources run: | set -euo pipefail FACTORY_ID="/subscriptions/${{ env.SUBSCRIPTION_ID }/resourceGroups/${{ env.RESOURCE_GROUP }/providers/Microsoft.DataFactory/factories/${{ env.SOURCE_FACTORY_NAME }" npm run build validate "${{ github.workspace }" "$FACTORY_ID" YAML - name: Export ARM templates (CI publish) run: | set -euo pipefail FACTORY_ID="/subscriptions/${{ env.SUBSCRIPTION_ID }/resourceGroups/${{ env.RESOURCE_GROUP }/providers/Microsoft.DataFactory/factories/${{ env.DEV_FACTORY_NAME }" npm run build export "${{ github.workspace }" "$FACTORY_ID" "${{ env.ARM_OUTPUT_DIR }" Whether you click Publish manually or use npm export in CI/CD, the outcome is the same: Full factory deploymentNo control over individual featuresAll changes get bundled together Selective Deployment Layer (Core Design) We can address this requirement and the associated challenges by introducing a workflow driven by a manifest to define the deployment scope, and a program to identify all necessary ADF dependencies for each manifest file. As a developer, I can now control which release is promoted to production, without worrying about releasing any other features that are not ready. The manifest controls which pipelines to deploy and which optional categories to include. Below is an example of a manifest file JSON { "pipelines": ["pl_ingest_population_selective"], "includeTriggers": false, "includeIntegrationRuntimes": false, "includeAllGlobalParameters": true, "includeLinkedServices": true, "validateLinkedServicesExist": true, "includeManagedVirtualNetwork": false, "includeManagedPrivateEndpoints": false } Workflow Explanation Let's understand the crux of the selective deployment workflow now. I am working in the release branch on my feature branch directly in ADF Studio. Since ADF Studio is integrated with Git, my development changes will be saved to my branch. Here are the steps I can take to promote my change to a higher environment. 1) Validation of ADF on PR validation This is an early validation step and a guardrail: if the PR fails, it's because objects are invalid and misaligned. This is equivalent to the "validation all" button in the ADF ui, here is this workflow Trigger: Pull requests targeting the branch selective_deployment. Purpose: Validate that the ADF JSON in the PR is valid in the context of the target factory. Main steps: CheckoutSet up Node.js 20npm installAzure login using OIDC (azure/login@v2)Validate with ADF Utilities: YAML FACTORY_ID="/subscriptions/${AZURE_SUBSCRIPTION_ID}/resourceGroups/${AZURE_RESOURCE_GROUP}/providers/Microsoft.DataFactory/factories/${DEV_FACTORY_NAME}" npm run build validate "$GITHUB_WORKSPACE" "$FACTORY_ID" 2) Release build + selective deploy to DEV adf-release-build-selective-deploy.yml Triggers: Push to selective_deploymentManual run (workflow_dispatch) with optional manifest inputDefault: deploy/manifests/release.json This workflow has two jobs: Job A: adf-build (staging + export + sanitize + artifacts) Checkout (full history)Azure login using OIDCSet up Node.js 20Install build dependencies inside build/ (npm install in build)Stage selective subset python scripts/select_adf_subset.py <manifest>, a code snippet below for the complete script, refer to the GitHub repository link given Python import json import re import shutil import sys from pathlib import Path from typing import Dict, Set, Tuple, List from collections import defaultdict # Your repo layout has pipeline/, dataset/, linkedService/ at ROOT. REPO_ROOT = Path(".") STAGE_ROOT = Path("build/adf_subset") RESOURCE_DIRS = { "pipeline": REPO_ROOT / "pipeline", "dataset": REPO_ROOT / "dataset", "linkedService": REPO_ROOT / "linkedService", "dataflow": REPO_ROOT / "dataflow", "trigger": REPO_ROOT / "trigger", "integrationRuntime": REPO_ROOT / "integrationRuntime", "credential": REPO_ROOT / "credential", "managedVirtualNetwork": REPO_ROOT / "managedVirtualNetwork", } # Copy these if present so ADF utilities behave the same on staged subset. ROOT_FILES_TO_COPY = [ "publish_config.json", "arm-template-parameters-definition.json", "arm_template_parameters-definition.json", "package.json", "package-lock.json", ] Produces: build/adf_subset/ (staged tree)build/adf_subset_report.json (dependency report)Refer to logs below (showing output of stage selective subset and debug to view output generated after select_adf_subset.py )Export ARM templates from the staged subset via ADF Utilities: npm --prefix build run build -- export "adf_subset" "$FACTORY_ID" "ArmTemplate"Produces: build/ArmTemplate/ARMTemplateForFactory.jsonbuild/ArmTemplate/ARMTemplateParametersForFactory.jsonStrip infra-owned resources scripts/strip_arm_resources.py to produce a safe template: build/ArmTemplate/ARMTemplateForFactory.safe.json⚠️ Note on Infrastructure Components (Refer to the “Future Work & Next Steps” section for follow-up topics in this series) The step above intentionally strips infrastructure-dependent components from the generated subset to avoid overwriting existing shared resources such as linked services. This implementation focuses on developer-owned artifacts (pipelines, datasets, and triggers) and assumes that infrastructure components — such as Integration Runtimes, managed private endpoints, and linked services — are pre-provisioned and managed outside of this deployment workflow.Upload artifacts: ARM templates (adf-arm)metadata (adf-release-meta)subset report (adf-subset-report) Job B: deploy_dev (deploy safe template) Download ARM artifactAzure login using OIDCEnsure az Data Factory extension is installedValidate JSON files exist/parseDeploy via azure/arm-deploy@v2(Incremental) to DEV RG/factory: Template: ARMTemplateForFactory.safe.jsonParameters: ARMTemplateParametersForFactory.json + factoryName=<DEV_FACTORY_NAME> Lesson Learned Setting up selective deployment in ADF was more than a technical task. It made us rethink our approach to deployments, ownership, and CI/CD design. Here are the main things we learned: 1. The Problem Is Not Tooling; It’s Deployment Granularity At first, we thought the limitation came from the tools we used, like UI publish or npm export. However, both methods yielded the same result: full factory templates. The real problem was that we couldn’t control the scope of deployments, not how the templates were made. 2. Dependency Awareness Is Critical Selective deployment only works when every dependency is found and included. We learned that: Pipelines often reference multiple datasets and linked services. Missing even one dependency results in deployment failure You must automate dependency discovery. 3. “Incremental” Is Often Misunderstood Incremental deployment is important, but it doesn’t work like a patch. It reapplies the full configuration for all included resources. This means: Your generated templates need to be complete for all the artifacts you include. If you use partial definitions, deployments can fail. 4. Separation of Concerns Is Key Not all ADF artifacts are the same. We began to separate them into different groups: Application-owned artifacts: pipelines, datasets, triggers Infrastructure-owned artifacts: linked service, managed virtual networks, managed private endpoints, and integration-runtime, among others. This separation proved crucial for safe, scalable deployments. 5. Selective Deployment Adds Complexity, But It’s Worth It It’s true that implementing this approach brings in additional scripts, manifest management, and CI/CD complexity. But in exchange, we gained precise control over releases, reduced production risk, and faster hotfix deployments. Future Work and Next Steps While selective deployment solved a major gap in ADF CI/CD, it also opened up new areas for improvement and standardization. 1. Defining Infrastructure vs Application Ownership One of the biggest follow-up areas is clearly defining ownership boundaries. In our experience: Application teams should own pipelines, datasets, and triggers Platform or infrastructure teams should own linked services, managed virtual networks, and managed private endpoints, among other things. Future work can focus on: Enforcing this separation in CI/CD. Preventing accidental deployment of infrastructure components Integrating Terraform or platform pipelines for infrastructure provisioning 2. Governance Around Linked Services Linked services are often shared across multiple pipelines and teams. Future improvements include: Centralizing linked service management Using Key Vault and Managed Identity consistently Preventing direct modifications through application pipelines

By Sauhard Bhatt

A Tool Is Not a Platform (And Your Team Knows the Difference)

Most infrastructure teams have a moment where someone says “we should build a platform.” The motivation is real: teams are duplicating work, the current setup is hard to use consistently, and a more structured approach would help. A few months later, the platform is a Terraform module collection, a GitLab CI template, a shared repository of scripts, and a README that several people have tried to keep current. That is a useful thing. It is not a platform. The distinction is worth being clear about, not to dismiss the work, but because the word “platform” creates expectations. When internal teams hear “we have a platform,” they assume stability, a usable interface, a versioning model, and some mechanism for raising problems when things break. A toolchain with documentation does not deliver those things by default. What Makes Something a Platform A platform is defined by its contract, not its technology. The contract describes what the consumer can expect: what they call, what parameters they provide, what outputs they receive, and what stability guarantees apply to that interface. A Terraform module with a published interface is closer to a platform primitive than a pipeline that provisions the same resources through environment variables, undocumented flags, and positional arguments. The module has a contract. The pipeline has a process. The contract does not have to be formal. It needs three things. A stable surface. Consumers should be able to call the same interface next month and receive the same type of result. Internal changes to how it works do not break consumers.A versioning model. When the interface changes, that change is communicated, and consumers are not silently broken. A git tag is enough to start with. Semantic versioning is better.A feedback path. Consumers can report when the contract is violated or the interface does not behave as documented. Someone is responsible for responding. A Terraform module with these three properties is a platform primitive. A set of modules with a shared versioning model, a stable registry entry, and a team responsible for maintaining the contract is starting to look like a platform. What Teams Actually Experience The gap between a toolchain and a platform shows up in how teams actually use it. With a toolchain, onboarding a new team means pointing them at the repository and telling them to read the README. Anything not in the README requires asking someone who has been around for a while. Changes to the toolchain break existing consumers silently because there is no versioning model. The team that maintains the toolchain treats every consumer as having kept up with the latest state of the repository. With a platform, onboarding means pointing teams at interface documentation with a working example. Changes go through a version increment. Consuming teams that pin to a version are not broken by changes they did not ask for. Plain Text # Consuming a module with a pinned version module "vm" { source = "registry.example.com/hybridops/vm/proxmox" version = "~> 2.1" name = "web-01" cores = 2 memory = 4096 } This looks like a small detail. For teams consuming infrastructure modules across a growing estate, it is the difference between a managed dependency and a shared folder everyone is afraid to touch. When a Toolchain Is the Right Call Not every infrastructure system needs to be a platform. A toolchain is appropriate when the team is small and holds the full mental model, the surface area is limited, and the rate of change is low enough that everyone stays current without a formal versioning model. When those conditions hold, the overhead of maintaining a platform contract is not justified. The problem is not having a toolchain. The problem is calling it a platform when it is not, and then finding that the expectations it created are not being met. Teams told they have a stable platform, then hit with a broken workflow from an unannounced change, lose confidence quickly. That confidence is hard to rebuild. HybridOps has been working in this space: publishing Terraform modules to a registry, versioning releases, and treating module interfaces as contracts. It is not a finished platform. It is a direction, and being explicit about that direction changes how the work gets done. A Simple Test If a consuming team pins to the current version of your toolchain today, will it still work in three months without any changes on their side? If you cannot answer yes with confidence, you have a toolchain, not a platform. Both are useful. Only one creates the kind of trust that makes a growing engineering organisation move faster rather than slower. Knowing which one you have is the first step toward building the right one.

By Jeleel Muibi

Deploying Infrastructure With OpenTofu

OpenTofu is an open-source infrastructure as code (IaC) tool maintained by the Linux Foundation. It lets you define cloud infrastructure in configuration files and deploy it with a single command-line tool called tofu. This tutorial explains how to deploy infrastructure with OpenTofu, from installing the CLI to provisioning and destroying a real cloud resource. What You Need Before You Start You need three things to follow along: An AWS account with credentials configured locally (the AWS command-line interface reads them from ~/.aws/credentials or standard environment variables).Basic comfort in a terminal.About 15 minutes. The resources in this tutorial cost almost nothing, and the final step deletes everything you create. Pick a region you are happy to work in, such as us-east-1. A Quick Word on OpenTofu OpenTofu is a fork of Terraform, created in 2023 after Terraform moved to the Business Source License (BSL), a source-available license that is not OSI-approved open source. OpenTofu is a Linux Foundation project and was accepted into the Cloud Native Computing Foundation (CNCF) as a sandbox project in 2025. The configuration language is the same HashiCorp Configuration Language (HCL) you may already know, every provider works the same way, and the command-line interface is tofu instead of terraform. If you have written Terraform before, you already know most of this. Step 1: Install OpenTofu Install OpenTofu using whichever method works best for your machine. On macOS or Linux with Homebrew: Shell brew install opentofu On Linux or macOS without Homebrew, use the official installer script: Shell curl --proto '=https' --tlsv1.2 -fsSL https://get.opentofu.org/install-opentofu.sh -o install-opentofu.sh chmod +x install-opentofu.sh ./install-opentofu.sh --install-method standalone rm install-opentofu.sh The standalone installer verifies the integrity of what it downloads, so it expects cosign or GnuPG to be available. If you don't have either and just want to try it quickly, add --skip-verify to the install command. On Windows, use winget: Shell winget install --exact --id=OpenTofu.Tofu Confirm the install worked: Shell tofu --version You should see OpenTofu v1.12.0 or later. Step 2: Write Your First Configuration Create a new directory and a single file inside it called main.tf: Shell mkdir tofu-demo && cd tofu-demo Open main.tf and add the following. Each block is explained right after. Shell terraform { required_providers { aws = { source = "hashicorp/aws" version = "~> 6.0" } random = { source = "hashicorp/random" version = "~> 3.0" } } } provider "aws" { region = "us-east-1" } resource "random_pet" "suffix" { length = 2 } resource "aws_s3_bucket" "demo" { bucket = "tofu-demo-${random_pet.suffix.id}" } output "bucket_name" { value = aws_s3_bucket.demo.bucket } A few things worth understanding here. The terraform block declares which providers your configuration depends on and where to download them. OpenTofu keeps this block name for backward compatibility, so the same configuration runs on either tool. The provider "aws" block sets the region you deploy into. The random_pet resource generates a short, readable suffix such as clever-mongoose, which keeps your bucket name globally unique without you having to invent one. The aws_s3_bucket resource is the infrastructure you are actually creating, and it references the random suffix, so OpenTofu knows to create the suffix first. The output block prints the final bucket name once everything is deployed. Step 3: Initialize the Project Run tofu init from inside your project directory: Shell tofu init This reads your terraform block, downloads the AWS and random providers from the OpenTofu registry, and sets up the working directory. You only rerun it when you add a new provider or module. You should see a message confirming OpenTofu has been initialized. Step 4: Preview the Changes Before OpenTofu touches your account, ask it what it intends to do: Shell tofu plan The plan is the most important habit in IaC. It shows you exactly what will be created, changed, or destroyed before anything happens. For this configuration, the plan shows two resources to add: the random pet and the S3 bucket. Read it. Make sure it matches what you expect. A clean plan-and-review step is what stops a one-line config change from accidentally deleting a database. Step 5: Deploy When the plan looks right, apply it: Shell tofu apply OpenTofu shows you the plan one more time and waits for you to type yes. Confirm, and it provisions the bucket. When it finishes, you see your bucket_name output. Open the S3 console in AWS and your new bucket is there, created entirely from code. You now have real infrastructure under version control. Change the configuration, run tofu plan to see the diff, and tofu apply to roll it out. That loop, edit, then plan, then apply, is the whole job. Step 6: Understand State After your first apply, OpenTofu creates a file called terraform.tfstate in your directory. This is the state file, which OpenTofu uses to map the resources in your configuration to the actual resources in your account. When you run a plan, OpenTofu compares your configuration, the state file, and the actual infrastructure to work out what changed. On your laptop, with one person and one project, a local state file is fine. It stops being fine the moment a second engineer needs to run a deployment. Two people with two copies of the state file will overwrite each other's work. The state file also holds resource metadata you do not want sitting in a Git repository or on a shared drive. This is the problem every team hits once IaC moves beyond a single person. Step 7: Clean Up Tear down everything you created so it costs you nothing: Shell tofu destroy OpenTofu shows you what it will delete and waits for a yes. Confirm, and your bucket and the random suffix are gone. The destroy command is the counterpart to apply, and it reads the same state file to know what to remove. Where This Goes Next Running tofu from your laptop is the right way to learn. It is the wrong way to run infrastructure for a team. Once more than one engineer is involved, you need shared remote state with locking so two applies cannot collide, a record of who changed what and when, policy checks that run before an apply rather than after an incident, and a way to catch drift when someone makes a manual change in the console. You can assemble these pieces yourself with a remote state backend, a continuous integration pipeline, and a set of scripts. Many teams start there. As the number of stacks and engineers grows, that homegrown setup becomes its own maintenance burden, which is the point where teams adopt an infrastructure orchestration platform. A platform like Spacelift manages OpenTofu runs against shared remote state, gates changes with policy as code, and detects drift between your configuration and what is actually deployed, with an audit trail across every change. It is one option in a category of tooling built to solve the team-scale problems this tutorial only hints at. The decision of whether and when to adopt one depends on how many people and environments you are managing. For now, you have the foundation: install, write, init, plan, apply, and destroy. Every OpenTofu project, from a single bucket to a fleet of production environments, runs on the same loop you just learned. Next, read the OpenTofu documentation on modules and remote backends to see how that loop scales from one file to a real codebase.

By Mariusz Michalowski

Implementing Asynchronous Communication Between Microservices Using Kafka and Spring Boot

In a microservices system, that tight coupling turns a small hiccup into a cascading slowdown. Thread pools fill, retries amplify traffic, and suddenly your simple request is blocked on half the fleet. My executive summary: asynchronous messaging with Kafka helps systems keep moving when individual components inevitably slow down or fail. It does this by decoupling producers from consumers, absorbing traffic spikes, and allowing services to evolve without tying their availability directly to one another. Code Patterns in Spring Boot With Kafka Spring for Apache Kafka gives me two primitives that feel pleasantly old Spring KafkaTemplate for sending and @KafkaListener for receiving. That template/listener model is intentionally similar to other Spring integration tech, which keeps application code focused on domain logic instead of raw client plumbing. Below is a compact (but production-shaped) pattern: externalized config via @ConfigurationProperties, a service port for publishing, a REST command endpoint, a consumer with a real error strategy (DLT), and a REST error advice. Java // === Messaging config (externalized, type-safe) === @ConfigurationProperties(prefix = "messaging.orders") @Validated record OrdersMessagingProps( @NotBlank String topic, @NotBlank String dltTopic ) {} // === DTO (event contract) === public record OrderCreatedEvent(UUID orderId, UUID userId, BigDecimal total, Instant createdAt) {} // === Service port (keeps domain testable, Kafka swappable) === public interface OrderEventPublisher { void publishOrderCreated(OrderCreatedEvent event); } // === Adapter: Kafka producer === @Component class KafkaOrderEventPublisher implements OrderEventPublisher { private final KafkaTemplate<String, OrderCreatedEvent> template; private final OrdersMessagingProps props; KafkaOrderEventPublisher(KafkaTemplate<String, OrderCreatedEvent> template, OrdersMessagingProps props) { this.template = template; this.props = props; } @Override public void publishOrderCreated(OrderCreatedEvent event) { // Keying by orderId keeps per-order ordering and drives partitioning decisions. template.send(props.topic(), event.orderId().toString(), event); } } // === REST command API (synchronous edge, async core) === @RestController @RequestMapping("/v1/orders") class OrdersController { private final OrderService orderService; // domain port OrdersController(OrderService orderService) { this.orderService = orderService; } @PostMapping public ResponseEntity<Map<String, Object>> create(@Valid @RequestBody CreateOrderRequest req) { UUID orderId = orderService.create(req.userId(), req.total()); // persists + publishes event return ResponseEntity.accepted().body(Map.of("orderId", orderId, "status", "ACCEPTED")); } record CreateOrderRequest(@NotNull UUID userId, @NotNull @Positive BigDecimal total) {} } // === Domain service port (implementation can use outbox, transactions, etc.) === public interface OrderService { UUID create(UUID userId, BigDecimal total); } // === Consumer: downstream service reacts to events === @Component class BillingListener { @KafkaListener(topics = "${messaging.orders.topic}", groupId = "${spring.kafka.consumer.group-id}") void onOrderCreated(OrderCreatedEvent event) { // Idempotency belongs here: process-by-key + store processed eventId/orderId to avoid duplicates. // Do work (charge card, create invoice, etc.) } } // === Kafka consumer error handling: retries + DLT === @Configuration class KafkaErrorHandlingConfig { @Bean DefaultErrorHandler defaultErrorHandler(KafkaTemplate<Object, Object> template, OrdersMessagingProps props) { var recoverer = new DeadLetterPublishingRecoverer(template, (rec, ex) -> new TopicPartition(props.dltTopic(), rec.partition())); // Backoff and retry policy are configurable; keep it finite to avoid poison-pill loops. return new DefaultErrorHandler(recoverer, new FixedBackOff(1000L, 3)); } } // === REST error handling (ProblemDetail) === @RestControllerAdvice class ApiErrors { @ExceptionHandler(IllegalArgumentException.class) @ResponseStatus(HttpStatus.BAD_REQUEST) ProblemDetail badRequest(IllegalArgumentException ex) { var pd = ProblemDetail.forStatusAndDetail(HttpStatus.BAD_REQUEST, ex.getMessage()); pd.setTitle("Invalid request"); return pd; } } A few been-burned-before notes on the code above. Spring Kafka’s reference docs are explicit that KafkaTemplate is the convenience wrapper for producing, and DefaultErrorHandler + DeadLetterPublishingRecoverer is a first-class way to route failed records to dead-letter topics after retries. If we want non-blocking retries, Spring Kafka also provides @RetryableTopic, which orchestrates retry topics and a DLT automatically useful when transient failures are common and you want predictable retry delay semantics. Containers and Local Dev With Docker Compose When I’m chasing down event flow bugs, I like local environments that feel like the old days: one command, deterministic startup order, and no mystery dependencies. Docker Compose is still the quickest way to stand up Kafka alongside your services, and Confluent publishes straightforward Docker-based tutorials and compose examples for running Kafka locally. For the service image itself, multi-stage builds are the modern classic compile in a builder stage, and copy the artifact into a slimmer runtime stage. Docker documents multi-stage builds as a way to reduce the final image contents and keep build dependencies out of production. Dockerfile # Multi-stage Dockerfile for a Spring Boot service (orders-service) FROM eclipse-temurin:21-jdk AS build WORKDIR /workspace COPY mvnw pom.xml ./ COPY .mvn .mvn RUN ./mvnw -q -DskipTests dependency:go-offline COPY src src RUN ./mvnw -q -DskipTests package FROM eclipse-temurin:21-jre WORKDIR /app COPY --from=build /workspace/target/*.jar app.jar EXPOSE 8080 ENTRYPOINT ["java","-jar","/app/app.jar"] And here’s a Compose file that wires up Kafka and Schema Registry, plus an example Spring Boot service. The exact image choices are illustrative. Your production choices are unspecified and should reflect your standards and security posture. YAML # compose.yaml (local/dev) services: zookeeper: image: confluentinc/cp-zookeeper:7.6.0 environment: ZOOKEEPER_CLIENT_PORT: 2181 kafka: image: confluentinc/cp-kafka:7.6.0 depends_on: [zookeeper] ports: ["9092:9092"] environment: KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181 KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:9092 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1 schema-registry: image: confluentinc/cp-schema-registry:7.6.0 depends_on: [kafka] ports: ["8081:8081"] environment: SCHEMA_REGISTRY_HOST_NAME: schema-registry SCHEMA_REGISTRY_KAFKASTORE_BOOTSTRAP_SERVERS: PLAINTEXT://kafka:9092 orders: build: ./orders-service depends_on: [kafka] ports: ["8080:8080"] environment: SPRING_KAFKA_BOOTSTRAP_SERVERS: kafka:9092 MESSAGING_ORDERS_TOPIC: orders.events MESSAGING_ORDERS_DLTTOPIC: orders.events.dlt SCHEMA_REGISTRY_URL: http://schema-registry:8081 Deploying on Kubernetes or AWS On AWS, the Kafka decision is usually managed or self-managed. If you choose Amazon MSK, the cluster lives in your VPC, pick subnets across distinct Availability Zones, and connect clients using the cluster’s bootstrap brokers. That’s the networking baseline, and it’s not optional. MSK is VPC-first by design. For authentication/authorization, MSK supports IAM access control. AWS documents the client configuration for IAM mechanisms. In EKS, I typically pair MSK IAM with IRSA so pods can obtain AWS credentials the AWS way, while ECS services would use task roles instead. Both patterns are documented by AWS, and your choice here is unspecified. Kubernetes service discovery is usually the easy part. Services and Pods get DNS names so workloads can call each other by name rather than IP. Kafka itself is reached via bootstrap broker endpoints or via internal Services, but either way, you want the strings in externalized config, not hardcoded. Here’s a minimal Kubernetes Deployment/Service for a Kafka client service. Values like region, account IDs, and MSK endpoints are unspecified placeholders. YAML apiVersion: apps/v1 kind: Deployment metadata: name: orders namespace: apps spec: replicas: 2 selector: matchLabels: { app: orders } template: metadata: labels: { app: orders } spec: serviceAccountName: orders-sa # IRSA-bound (role ARN unspecified) containers: - name: orders image: <UNSPECIFIED_AWS_ACCOUNT_ID>.dkr.ecr.<UNSPECIFIED_REGION>.amazonaws.com/orders:<TAG> ports: [{ containerPort: 8080 }] env: - name: SPRING_KAFKA_BOOTSTRAP_SERVERS value: "<UNSPECIFIED_MSK_BOOTSTRAP_BROKERS>" - name: MESSAGING_ORDERS_TOPIC value: "orders.events" - name: MESSAGING_ORDERS_DLTTOPIC value: "orders.events.dlt" readinessProbe: httpGet: { path: /actuator/health/readiness, port: 8080 } initialDelaySeconds: 10 --- apiVersion: v1 kind: Service metadata: name: orders namespace: apps spec: selector: { app: orders } ports: - port: 80 targetPort: 8080 Operationally, MSK exposes metrics into CloudWatch (AWS/Kafka), and broker logs can be delivered to CloudWatch Logs (or S3/Firehose). That combination gives you the classic visibility loop: throughput, lag, under-replicated partitions, and error logs without running your own monitoring plane. For distributed tracing in async flows, OpenTelemetry is my default vocabulary now. Spring Boot supports OpenTelemetry export via OTLP, and OpenTelemetry defines Kafka semantic conventions so your producer/consumer spans and attributes stay consistent across tools. CI/CD and the Hard-Earned Field Notes For CI/CD, I keep it boring: build once, push an immutable image, deploy via a declarative mechanism. AWS Prescriptive Guidance provides a clear GitHub Actions pattern for building Docker images and pushing to Amazon ECR, which is a solid baseline when your region/account is unspecified until configured. YAML # .github/workflows/orders.yml name: orders on: push: branches: ["main"] jobs: build_push_deploy: runs-on: ubuntu-latest permissions: id-token: write contents: read steps: - uses: actions/checkout@v4 - uses: actions/setup-java@v4 with: distribution: temurin java-version: "21" - name: Build & test run: ./mvnw -q test package - name: Configure AWS credentials (OIDC) uses: aws-actions/configure-aws-credentials@v4 with: role-to-assume: arn:aws:iam::<UNSPECIFIED_AWS_ACCOUNT_ID>:role/<UNSPECIFIED_GHA_ROLE> aws-region: <UNSPECIFIED_REGION> - name: Login to ECR run: | aws ecr get-login-password --region <UNSPECIFIED_REGION> \ | docker login --username AWS --password-stdin <UNSPECIFIED_AWS_ACCOUNT_ID>.dkr.ecr.<UNSPECIFIED_REGION>.amazonaws.com - name: Build & push image run: | IMAGE=<UNSPECIFIED_AWS_ACCOUNT_ID>.dkr.ecr.<UNSPECIFIED_REGION>.amazonaws.com/orders:${{ github.sha } docker build -t $IMAGE ./orders-service docker push $IMAGE - name: Deploy to EKS (example) run: | aws eks update-kubeconfig --name <UNSPECIFIED_EKS_CLUSTER> --region <UNSPECIFIED_REGION> kubectl -n apps set image deploy/orders orders=$IMAGE Now, the part I wish someone had handed me in 2016: Kafka gives you strong tools, but it does not remove distributed-systems truths. You still need safeguards on the consumer side: idempotent processing, disciplined schema management, and clearly defined retry and dead-letter topic behavior. Kafka’s documentation is careful about the limits of “exactly once” guarantees. Idempotent producers and transactions can strengthen delivery semantics, but achieving true end-to-end exactly-once behavior, especially when external side effects are involved, still depends on deliberate system design. For schema governance, Kafka itself doesn’t ship a schema registry, but acknowledges third-party registries; in practice, Confluent Schema Registry and Apicurio Registry are common choices. Both store schemas out-of-band, so messages carry only a schema identifier, and both support evolvable contracts across Avro/JSON Schema/Protobuf depending on your ecosystem. Conclusion and Best Practices If you take one lesson from my legacy brain into modern event-driven systems, let it be this: asynchrony is a reliability feature, not a performance trick. Kafka’s durable log and consumer group model decouples uptime and absorbs spikes, but you only get the real benefit when you treat schemas as contracts, consumers as idempotent processors, and failure handling as first-class application behavior. On AWS, the operational baseline is non-negotiable. MSK lives in your VPC across AZ subnets, clients connect via bootstrap brokers, IAM auth is configured explicitly, and observability lives in CloudWatch. Do those fundamentals early, and Kafka stops feeling like a mysterious black box and starts feeling like the dependable workhorse it was built to be.

By Mallikharjuna Manepalli

Deployment

DZone's Featured Deployment Resources

Top Deployment Experts

The Latest Deployment Topics