Sri

A Google Cloud zone in London was offline for about 6 hours last week. The facility had backup generators and two redundant controllers to start them. Both failed the same way, at the same moment. Here is what Google's incident report says, and the three lessons I take from it. What happened, from the report: - The site was mid-repair: one utility feed was locked out for fixes, so a single feed carried everything - That live feed then failed, from an underground cable fault on the utility network - The generator controllers sensed voltage on the wrong side of the locked-out breaker, concluded power was fine, and never started the generators - The UPS batteries bridged about 5 minutes, then the zone lost power - Failover traffic moved to the surviving zones and saturated them, causing errors across the region Lesson 1: redundancy is not independence. Duplicated components with one shared design flaw fail together. The useful question is not 'do we have two?' but 'can one mistake take out both?' Lesson 2: planned maintenance is your most fragile state. The site was intentionally degraded when the accident hit. Before maintenance, ask what must still work while the component is out, and test that state, not the normal one. Lesson 3: failover is a second event. The surge from the dead zone hurt the healthy ones. Capacity for failover traffic is part of disaster recovery design, not an afterthought. One thing Google got right: they recovered slowly on purpose, letting storage systems that lost power abruptly finish recovery before re-admitting traffic, because restoring services sooner would have made things worse. And a caveat worth keeping: testing lowers the chance of surprises. It does not prove recovery will work. When did you last test failover while part of your redundancy was already offline? Report in here: status.cloud.google.com/inci…

If your AI agent calls enterprise systems, your logs should tell you three things: - who acted, on whose behalf, and with what access. If you can't, the solution isn't production-ready. Imagine an agent that reads a support ticket, looks up customer data, and updates a record. If it runs on one shared account, the audit trail shows the same name for every action. If it forwards one token to every system, the agent is invisible, and a stolen token can reach further. What I'd want instead: - A token meant for one system only - Narrow permissions, asked for when needed - A log that records both the user and the agent An agent can act for a user, but it needs the right token for each system it connects to. It should get that token through the system's supported authorization process, with only the permissions it needs. A shared service account can work too, if its access is limited and you can trace each action back to who requested it. The point is to choose deliberately, not just use whatever gets the demo working. This is usually one of the first questions in an enterprise security review. Getting it right early shapes the architecture. Fixing it later is expensive. Could you tell from your logs which agent action was done for which user?

Before you draw the architecture, understand the customer's constraints. I have been working with multiple orgs for the past 5+ years. In enterprises, the design is often decided by things that are not in the spec: - Can data leave their environment? - Can workloads reach the internet or image registries? - Which identity system must we use? - When are changes allowed? - Who runs it after go-live? The answers change the design. No outbound internet changes how you handle updates, images and API calls. A strict identity system changes how services authenticate. A good solution is not just technically right. It fits the place it has to run. What's the one customer constraint that most often changes your design?

Your pod says "Running" but the app is down. Running doesn't mean ready to serve requests. One place to look: readiness vs liveness probes. Readiness asks: should this pod get traffic? A failed check marks it not ready. Normal Service traffic stops going to it, but the container isn't restarted. Liveness asks: does this container need a restart? After the configured number of consecutive failures, the kubelet restarts it. Two mistakes worth avoiding: - No readiness check: traffic can reach an app before it's ready. - Overly strict liveness checks: a slow-starting app can get stuck restarting. Slow to boot? Use a startup probe with enough time for startup. Readiness and liveness checks wait until it succeeds. Readiness controls traffic. Liveness controls restarts. Which has bitten you more: missing readiness checks or an over-eager liveness probe? #Kubernetes #DevOps #SRE

Getting agents to work in a demo is one thing. But making it reliable enough for people to use every day in an enterprise environment is another. I worked with orgs to build cloud infrastructure and CI/CD for enterprise environments earlier. Now, as a forward-deployed engineer, I work with enterprises to take solutions from understanding the problem to running them in production. For me, that means a few practical questions: - Can the solution connect to the tools and data the team already uses? - How can we determine whether its answers and actions are correct? - Can it access only the data it's allowed to use? - Can we be able to figure out what happened when something goes wrong? - Who fixes it, and can we undo a bad change? A demo/PoC shows what's possible. Production means people can rely on it for their daily work. If you've taken agentic systems from a demo to production, what was the hardest part?

Disaster Recovery & Business Continuity: Most teams pick a DR strategy before they know their numbers. It works better the other way around. Sketch 1: the four classic DR strategies - Backup & Restore, Pilot Light, Warm Standby, and Multi-Site (Active-Active) - and the core trade-off: faster recovery and lower data loss cost more. Sketch 2: the metrics that drive the decision: - RPO: how much data you can afford to lose - RTO: how long you can be down - MTD: the maximum tolerable downtime - the real deadline - WRT: catch-up time after systems are back - MTTR & MTBF: and how they add up to the availability "nines" Rule of thumb: agree RPO and RTO with the business first. The strategy picks itself after that. #DevOps #SRE #SystemDesign #DisasterRecovery #BusinessContinuity

This is how your production EKS cluster should look like 👇👇 VPC • Multiple AZs • Private subnets for nodes • Public subnets only for load balancers EKS Control Plane • Managed by Amazon Elastic Kubernetes Service • Endpoint access restricted Worker Nodes • Managed Node Groups • Spread across multiple AZs Networking • Amazon VPC CNI • AWS Load Balancer Controller for ALB/NLB • Security Groups for Pods when needed Ingress • Application Load Balancer for HTTP services • Network Load Balancer for TCP workloads Autoscaling • Cluster Autoscaler or Karpenter • Horizontal Pod Autoscaler Security • IAM Roles for Service Accounts (IRSA) • Pod Identity • Secrets stored in AWS Secrets Manager with KMS • Network policies enabled Observability • Metrics → Amazon Managed Service for Prometheus • Logs → Amazon CloudWatch • Tracing → AWS X-Ray Reliability • Multi-AZ node groups • Pod disruption budgets • Readiness & liveness probes Storage • Amazon EBS via CSI driver • Amazon S3 for object storage Deployment • GitOps via Argo CD