Replacing the DNS Resolver in Production Without a Postmortem: Lessons from a 180k-Line Deletion

Introduction

In 2023, a team at a mid-sized cloud provider replaced its aging, 180,000-line DNS resolver—responsible for resolving millions of queries per second—with a lean, 12,000-line Rust implementation. The migration took six months. There was no major outage. No postmortem. No panic at 3 a.m.

This isn’t a story about clever engineering tricks or heroic firefighting. It’s about discipline, incrementalism, and choosing the right abstraction boundaries when replacing critical infrastructure.

Why Replace a Resolver?

The legacy resolver was written in C++, designed in the early 2010s. Over time, it accumulated features, workarounds, and patches that turned it into a maintenance burden. Memory safety bugs were frequent. Performance bottlenecks were hard to isolate. And worst of all, no one wanted to touch it.

We needed:

• Memory safety: Eliminate buffer overflows and use-after-free bugs.
• Better observability: Native metrics, tracing, and structured logging.
• Simpler codebase: Easier to audit, test, and extend.
• Modern async I/O: Leverage tokio and async/await for concurrency.

The Plan: Replace, Don’t Refactor

We decided early on to replace rather than refactor. Refactoring risks accumulating more cruft on top of existing debt. A clean rewrite—done carefully—can eliminate entire classes of issues.

Key Principles

1. Incremental rollout: No big-bang switch.
2. Dual-running mode: Run both old and new resolvers side-by-side.
3. Traffic shadowing: Mirror traffic to the new resolver without acting on results.
4. Gradual cutover: Shift traffic percentage by percentage.
5. Rollback safety: Instant revert capability at every step.

Step-by-Step Migration

1. Build the New Resolver

We chose Rust for memory safety and performance. The new resolver, trust-dns-based, implements:

• RFC-compliant recursive resolution
• DNS-over-TLS and DNS-over-HTTPS (for upstream)
• Prometheus metrics
• Structured logging via tracing
• Graceful shutdown handling

use std::sync::Arc;
use tokio::net::UdpSocket;
use tracing::instrument;

#[derive(Clone)]
pub struct DnsResolver {
    inner: Arc<trust_dns_resolver::TokioResolver>,
}

impl DnsResolver {
    #[instrument(skip_all)]
    pub async fn new() -> Result<Self, Box<dyn std::error::Error>> {
        let resolver = trust_dns_resolver::Resolver::tokio_async(
            trust_dns_resolver::ResolverConfig::cloudflare(),
            trust_dns_resolver::ResolverOptions::default(),
        )?;
        Ok(Self { inner: Arc::new(resolver) })
    }

    #[instrument(skip_all)]
    pub async fn lookup(&self, query: &str) -> Result<Vec<String>, Box<dyn std::error::Error>> {
        let response = self.inner.lookup_ip(query).await?;
        let ips: Vec<String> = response.iter().map(|ip| ip.to_string()).collect();
        Ok(ips)
    }
}

2. Shadow Traffic

Before routing any production traffic, we mirrored 1% of queries to the new resolver and compared responses. This surfaced subtle differences in TTL handling and cache behavior.

#[instrument(skip_all)]
pub async fn resolve_with_shadow(
    &self,
    query: &str,
    shadow: bool,
) -> Result<Vec<String>, Box<dyn std::error::Error>> {
    let primary = self.legacy_resolver.lookup(query).await?;
    if shadow {
        match self.new_resolver.lookup(query).await {
            Ok(new_result) => {
                if primary != new_result {
                    tracing::warn!("Mismatch for query '{}': old={:?}, new={:?}", query, primary, new_result);
                }
            }
            Err(e) => tracing::error!("New resolver failed for '{}': {}", query, e),
        }
    }
    Ok(primary)
}

3. Canary Rollout

We shifted 1%, then 5%, then 25%, then 75% of traffic. Each phase lasted at least one hour. We monitored:

• Latency (p50, p95, p99)
• Error rates
• Cache hit ratios
• Memory/CPU usage

4. Full Cutover

After two weeks of shadow testing and a month of gradual rollout, we flipped the switch. The old resolver was decommissioned.

Gotchas We Hit

Subtle DNS Behavior Differences

DNS resolvers are expected to be deterministic, but edge cases in caching, retries, and timeouts can differ. Shadowing caught most discrepancies.

Upstream Resolver Variance

Different upstream resolvers (Cloudflare vs. Google vs. AWS) occasionally returned slightly different results. We standardized on Cloudflare for consistency.

TTL Handling

The old resolver had quirky TTL clamping logic. The new one follows RFCs strictly. This caused brief cache churn during early rollout.

What We Removed

Deleting 180k lines wasn't just about replacing code—it was about removing:

• Custom DNS parsers (replaced with trust-dns)
• Hand-rolled event loops (replaced with tokio)
• Legacy TLS stack (replaced with rustls)
• Ad-hoc metrics (replaced with metrics-exporter-prometheus)

Lessons Learned

1. Start with Observability

Build metrics and tracing into the new system from day one. Without visibility, you’re flying blind.

2. Dual-Run Everything

Never assume equivalence. Run both systems side-by-side and compare outputs.

3. Go Slow

Resist the urge to rush. Each phase should last long enough to catch regressions.

4. Make Rollback Trivial

If you can’t instantly revert, you haven’t earned the right to deploy.

5. Delete Fearlessly

Removing 180k lines of dead code felt scary—until we realized how much safer the system became.

Frequently Asked Questions

Q: Why not use a managed DNS service?

A: We needed low-latency, in-process resolution for internal services. Managed services introduced unacceptable tail latency.

Q: How long did the rewrite take?

A: About three months for the core team. Most of the time was spent on testing, shadowing, and gradual rollout.

Q: Did you consider rewriting in Go instead of Rust?

A: Yes, briefly. But Rust gave us better performance and fewer runtime surprises under load.

Conclusion

Replacing critical infrastructure is hard. But with the right approach—incremental rollout, dual-running, and a focus on observability—you can do it without incident. The biggest lesson? Don’t fear deletion. Sometimes, the best refactor is a clean rewrite followed by a bold deletion.

For more on systems design and infrastructure migrations, check out Tamiz's Insights.