how/network

Anycast: one IP from dozens of locations

1.1.1.1 answers fast from anywhere in the world because it physically lives in many places at once. The trick is BGP: each client lands on the nearest node.

When you ping 1.1.1.1 (Cloudflare DNS) from Moscow, the reply comes back in 20ms. From Tokyo, also 20ms. From San Francisco, also 20ms. Magic?

Not magic. Anycast. One IP address is announced from dozens of physical locations at once through [[bgp|BGP]]. Each client, by normal BGP logic, lands on the node nearest to it and sees low latency.

If a node goes down, BGP automatically stops announcing it, and traffic switches to the next nearest one. No application-level health checks, no config changes: failover in seconds.

Press ▶ to watch one 1.1.1.1 serve clients from 3 regions and see how failover works.

step 1/5·00 · 3 physical servers, one IP

§ steps

Cloudflare has three data centers: New York, Frankfurt, Singapore. Each one has a separate physical server, ordinary "hardware". But all three listen on 1.1.1.1 and announce this prefix through BGP to their regional ISP.

From BGP's point of view this is three independent announcements of the same prefix. Each ISP sees one of them (its own nearest).

recap

What to remember:

Anycast = many servers with one IP. At the network stack level each server is ordinary, listening on this IP. The trick is in the BGP announcement: the same prefix is announced from many AS/locations
Who lands where is decided by BGP best-path on each router along the way. Usually that is "geographically nearest", but there can be anomalies (traffic may take a long path because of peering policies between providers)
Failover is automatic: if a node goes down, it stops announcing its prefix through BGP, the announcements about it "go stale" in the tables within minutes, and traffic moves to the remaining nodes
Ideal for: DNS (1.1.1.1, 8.8.8.8 are both anycast), CDN edges, DDoS protection (the attack spreads across all nodes), global API endpoints. All 13 root DNS servers are anycast, physically there are hundreds of machines
Not suited for: TCP connections with long-lived state (if BGP switches to a different node along the way, the TCP session breaks, the new node has no conntrack record). Solutions: stateless protocols (UDP, DNS), or sticky sessions through an L7 proxy
Compared with DNS balancing: in DNS balancing you have different IPs, the client picks one. In anycast you have one IP, and the network picks for you. Anycast is faster (no extra DNS lookup) and more resilient to bad resolvers

Related to bgp (announcements and path selection) and [[dns-resolution|DNS]] (where anycast is used most often).

§ dig into the knowledge base

§ steps

Cloudflare has three data centers: New York, Frankfurt, Singapore. Each one has a separate physical server, ordinary "hardware". But all three listen on 1.1.1.1 and announce this prefix through BGP to their regional ISP.

From BGP's point of view this is three independent announcements of the same prefix. Each ISP sees one of them (its own nearest).