Howdy Friends,

I’m back after a few months away. Hope you haven’t forgotten me!

After Diwali, I returned to Kathmandu. My team had a compliance Disaster Recovery event scheduled for November 29, and I was fully focused on preparing for it.

Today, I want to share what I learned from this experience and why it matters beyond just passing a compliance check.

What Is a Disaster Recovery Event

Every year, my company runs a Disaster Recovery drill.

The goal is to spin up the entire infrastructure and applications in a separate AWS account, in a different region (West instead of East).

We then validate data, calculate RPO and RTO, and prepare a report for the security team and our clients.

As part of the infrastructure team, we own everything from infrastructure to application deployment.

I work at a health tech company handling critical health data, so this is not optional. It’s required for compliance and client trust.

The Real Challenge

On paper, this sounds simple.

If your infrastructure is written as Infrastructure as Code (IaC), you just replicate it and move on, right?

Well, yes and no.

Infrastructure spins up perfectly. VPCs, subnets, load balancers, instances. Everything looks green.

But a green infrastructure does not mean a working system.

The real challenge is data and application behavior.

Even after successful deployment in the West region, applications were still trying to connect to services in the East region.

This is not the hardest part. The challenge is redirecting application requests to West region resources by changing configurations and related mechanisms.

These configurations must stay updated as the system evolves. And this is the kind of failure dashboards do not show.

Our Approach to DR

Before the DR event

• Ensure backups are complete and restorable

• Validate configurations are region-aware

During the DR event

• Document every success and failure

• Capture what broke, how, and why

After the DR event

• Fix issues with the development team

• Improve automation to make the next DR smoother

How We Execute Disaster Recovery

We use a third-party tool called Arpio combined with in-house automation. Our stack includes Ansible, CloudFormation, Terraform, Bash, and Python.

We create a private DNS and configure VPN access so anyone connected is routed only to the DR environment. This prevents accidental access to production.

Simple applications
Applications with minimal configuration needs are replicated directly using Arpio. Resources are copied from East to West with identical networking and naming.

Complex applications
Applications that require config changes are deployed in two steps. First, infrastructure is deployed using IaC. Then we use Ansible to deploy the application with updated configurations.

Configs that support the West region are stored in S3 and updated through automation. Applications pull these configs at runtime. They include region-specific values like bucket names and service endpoints.

For APIs, databases, and load balancers, we migrated all endpoints to domain names. During DR, we only update Private Route 53 records. Applications resolve endpoints through DNS and connect to the correct region without code changes.

Testing Is Where Reality Hits

Once applications are up, the QA team gets VPN access to the DR environment and begins testing. We allocate several hours for this.

If issues are found, we try to fix them during the DR window. Otherwise, we create tickets and address them later.

This phase consistently reveals real gaps in application logic and data replication.

After testing, we finalize issue tickets and calculate RPO and RTO.

One Failure That Surprised Us

A large portion of client data in the West region was stale.

In 2025, we focused heavily on security and moved many resources to private access, including S3, SQS, SNS, and KMS. These changes were tested thoroughly in production.

However, they caused severe failures in the DR environment. Due to these security updates, a significant amount of client data was not replicated to the West region.

The application did not crash. It behaved incorrectly.

We then created a ticket to add automation that monitors S3 bucket replication and alerts us when replication breaks.

Major Issues We Faced This Year

• IAM and network permissions did not fully replicate

• Security hardening caused S3 replication failures

• OpenSearch and QuickSight data replication remains difficult

• Large infrastructure changes like OS migrations and AWS SCPs increased complexity

• Many hardcoded values in applications were discovered unexpectedly

What We Will Do Differently Next Time

Test security changes against DR earlier

Push harder on removing hardcoded values

Automate more validation checks before QA starts

The Takeaway

Disaster Recovery is not about copying infrastructure. It is about whether your system can operate independently in a new environment.

Configuration management, DNS strategy, and data replication matter as much as spinning up resources.

Security improvements can break DR workflows, so every change should be validated against Disaster Recovery.

The success of the 2025 DR exercise was built on lessons from previous years. Repeated practice creates confidence and maturity. The work we did this year will make the 2026 DR smoother and more predictable.

If you had to run a DR drill tomorrow, which part of your system would you trust the least? I would love to hear your experience.

Until next time,
Alon

Postscript

I recently earned the AWS Certified Developer Associate certification, my last one of the year. I will share more about this in an upcoming newsletter.

My company was acquired by a much larger organization. Big changes are happening, and I will share that experience soon.