I recently led a production migration that sounds simple from the outside but took a lot of planning to execute.

The task was simple: replace the default internet route from an EC2 NAT instance to a NAT Gateway.

That is it. Just update the route table.

But this was not a small test VPC. This was a production VPC in us-east-1, and a large part of the environment depended on those NAT instances for outbound traffic.

Once I understood that, the project stopped looking simple.

And because I was leading it, I had to think beyond the technical change and plan around risk, cost, downtime, and communication.

How We Got Here

Back in 2021, two NAT instances were created to save cost.

At that time, the product was still growing slowly. Traffic was small, downtime during maintenance was acceptable, and using EC2 instances as NAT made sense from a cost point of view.

The setup was simple:

• two EC2 NAT instances running Amazon Linux 2
• a few route tables using those NAT instances
• one route table set as the default

For that stage of the product, it worked.

But systems do not stay the same. And honestly, that is one reason I find this job interesting.

Over time, the product grew. Monthly traffic moved from GiBs to TiBs. More resources were added, more dependencies were created, and some were documented while many were not.

By 2023, those NAT instances had become too critical to reboot casually.

That meant no easy maintenance, no easy patching, and no simple replacement.

And because they were still running Amazon Linux 2 with no patching, the security risk kept increasing over time.

The Risk Became Visible

In 2025, we started noticing bottlenecks during heavy business days.

One NAT instance was a1.large, and the other was t3.medium. These were not designed to become long-term regional traffic gateways for a growing production environment.

During architecture discussions, we started listing them as a single point of failure and a critical resource.

Everyone agreed it was risky. But agreement is not the same as migration.

The organization had grown, and changing a critical shared component was no longer easy. Many systems depended on it, and some dependencies were unknown. So the risk stayed there for a while.

Then in 2026, another project created the right timing: migrating Amazon Linux 2 because it was approaching end of life.

I was also leading that project, and while planning the work, my attention kept coming back to these two NAT instances.

I knew we had two options:

1. replace them with new EC2 NAT instances on a different OS
2. move to AWS NAT Gateway

The second option was better operationally, but it came with a hard conversation.

Cost.

The Cost Conversation

When I first shared the risk, everyone understood why we needed to move.

The NAT instances were old, critical, hard to patch, and difficult to maintain. We were lucky they had been stable for so long, but luck is not an architecture strategy.

Then I shared the estimated cost increase of moving to NAT Gateway. It was almost double what we were paying.

The room became quiet.

That was the real tradeoff.

NAT Gateway gives better availability, scaling, and lower operational overhead. But it can become expensive, especially when a lot of traffic flows through it.

So before asking the business to accept the cost, I needed to understand the traffic and bring a stronger migration plan.

I started collecting traffic flow data to answer three questions.

Where was the traffic going?

Which resources were generating it?

Could we reduce it before migration?

Another critical thing I found was route table imbalance. Because one route table had become the default pattern, many subnets were associated with it, and traffic was not distributed equally.

One NAT instance was handling around 500 GB per month, while the other had grown close to 40 TB per month.

That showed how much technical debt had built up quietly.

I did not focus on fixing that immediately. I treated it as the next phase after migration.

Understanding the Traffic

At first, I enabled VPC Flow Logs for a few days.

That helped, but not enough. I mostly got source and destination IPs, and mapping those IPs back to real services was difficult. It could also get expensive if we kept collecting logs for a long time.

Then I found a better source of information.

Because the NAT servers were Linux EC2 instances, I could inspect connection tracking from:

/proc/net/nf_conntrack

That gave me much richer detail about active connections, including the private source IP generating traffic and the destination IP.

But this is a kernel-managed connection tracking file, and it changes constantly. So I had to collect the data carefully.

I attached an additional disk to avoid filling the root volume, then captured connection tracking data there so the NAT instance itself would not crash because of storage exhaustion.

I monitored it for around two weeks and collected around 30 GB of logs.

After that, I worked on a script to analyze where the traffic was going. For AWS destinations, I used the public AWS IP range file.

The script checked whether destination IPs belonged to AWS or outside the AWS.

It was not perfect. Many IPs still appeared only as AMAZON, not a specific service name. But it was enough to reveal the important pattern.

Around 90% of the traffic was going back into AWS.

That was the turning point.

Reducing Traffic Before Migrating

If we moved everything directly to NAT Gateway, that internal AWS traffic would become expensive.

So before migration, I started reducing the NAT traffic.

We added VPC endpoints for services like:

• S3
• DynamoDB
• CloudWatch
• SSM
• and more

This allowed private resources to reach AWS services through private paths instead of sending everything through the NAT layer.

After that change, we reduced the NAT traffic significantly, almost by half compared to the earlier EC2 NAT instance traffic.

Now I had a stronger story for the migration.

We were not just saying, "NAT Gateway is better."

We were saying:

We analyzed the traffic, reduced unnecessary NAT usage, lowered the cost risk, and now the remaining traffic should move to a managed, more reliable service.

That made the decision much easier to defend.

The Elastic IP Problem

The next major question was whether to preserve the existing Elastic IPs.

This became more serious than expected.

We realized some clients had whitelisted our outbound IPs, but the list of who had whitelisted what was not fully documented.

Changing the Elastic IPs could break client integrations.

So we decided to preserve them and that created another challenge.

To reuse an Elastic IP from a NAT instance, we needed to detach it from the EC2 instance and attach it to the new NAT Gateway. That takes time, and in our environment even a few minutes of outage was a big deal.

So we looked for a safer migration path.

The Migration Plan

The idea was to use a temporary NAT Gateway.

Instead of moving the route directly from the NAT instance to the final NAT Gateway, we would first point the route table to a temporary NAT Gateway.

Then we could detach the Elastic IP from the NAT instance, create the final NAT Gateway with the preserved Elastic IP, wait for it to become available, and finally move the route again.

The goal was to keep general outbound traffic working while we prepared the final NAT Gateway.

Clients that had whitelisted the old Elastic IP might see a short interruption during the switch. But other outbound traffic should continue working through the temporary NAT Gateway.

We tested this in a sandbox account first.

I also wrote a small script that continuously ran curl every second to check whether outbound connectivity dropped during the route changes.

In sandbox, the migration looked clean. No visible network drop.

That gave us confidence.

Production Migration

Since there were two NAT instances, we decided to migrate them separately.

First, we chose the lower traffic NAT instance.

During the migration, we saw around 3 seconds of downtime when shifting to the new NAT Gateway. It was unexpected because the sandbox test did not behave that way, but we had already requested a 5-minute maintenance window, so it stayed within the planned window.

At first, we thought there might be an issue with the script.

So we tested again in sandbox and it worked fine there.

The next week, we migrated the higher traffic NAT instance.

This one was much more important because it was handling far more traffic.

During the second migration, the temporary NAT Gateway route change still worked fine. But when shifting to the final NAT Gateway, we saw around 30 seconds of downtime.

Again, sandbox did not reproduce the same behavior.

I still do not know the exact reason.

My current assumption is that a fresh NAT Gateway receiving large traffic suddenly from many resources may take a short time to fully settle or scale behind the scenes. But that is only an assumption, and in engineering assumptions are not enough.

I may dig deeper into this someday.

What Changed

The most critical NAT path is now moved from self-managed EC2 NAT instances to AWS NAT Gateway.

That means:

• less operational maintenance
• no EC2 patching burden for NAT
• better managed availability and scaling
• a cleaner architecture for future growth

We are still monitoring traffic and cost closely.

I hope the final cost stays close to what I estimated. 🤞

What I Learned

This project reminded me that cloud migration is not always about changing resources.

The hard part is understanding what depends on that migration and explaining risk to leadership.

The hard part is reducing cost before making the architecture better.

The hard part is dealing with undocumented dependencies, old decisions, and production traffic that grew slowly over years.

From the outside, this was just a migration from NAT instances to NAT Gateway.

For me, it was a real architecture lesson.

Cost matters. Reliability matters. Documentation matters.

And technical debt does not always appear as broken systems. Sometimes it appears as a server nobody wants to reboot.

More soon,

Alon