IPsec and IKEv2: the enterprise VPN standard

Why IPsec

The standard for enterprise and operator-grade VPN since the 90s. It works at L3 and does not depend on the application protocol. Every vendor supports it: Cisco, Juniper, Mikrotik, Linux, Windows, macOS, iOS. Interoperability without a fight. Today it lives in:

• Site-to-site between data centers and clouds. AWS VPN and Azure VPN Gateway are IPsec under the hood
• Operator-grade VPN at telecoms and banks
• Native VPN on iOS and macOS (a toggle in settings, no app needed)
• "Always-on" corporate VPN with EAP authentication

One downside: it has always been complex. A strongSwan config for a simple case runs to dozens of lines. [[wireguard|WireGuard]] solves the same task in about five. But IPsec gives you certificates, EAP, mode-config, IKEv2 mobility (MOBIKE) and a full PKI.

Architecture: ESP, AH, IKE

Three protocols:

• ESP (Encapsulating Security Payload, IP-protocol 50) encrypts and authenticates the payload. Used almost everywhere.
• AH (Authentication Header, IP-protocol 51) does authentication only, no encryption. It breaks NAT (it includes the IP header in the HMAC). You will rarely see it anywhere.
• IKE (Internet Key Exchange, UDP 500/4500) handles the negotiation: peer authentication, key generation for ESP. IKEv1 (1998) is obsolete; IKEv2 (2005, RFC 7296) is the modern, simpler version.

Transport vs tunnel mode

Transport mode (host-to-host):

  [IP] [ESP] [TCP/UDP/payload]

      ^ original IP is preserved

Tunnel mode (site-to-site):

  [new-IP] [ESP] [original-IP] [TCP/UDP/payload]

      ^ outer IP, between gateways

      ^ inner IP, the end hosts

• Transport is peer-to-peer at the same level. Used in host-to-host scenarios or with GRE on top.
• Tunnel is gateway-to-gateway or client-to-gateway. When people just say "VPN" they almost always mean tunnel-mode IPsec.

IKEv2: two exchanges

The minimal handshake fits in 4 packets:

1. IKE_SA_INIT: peers agree on a crypto suite (encryption, PRF, integrity, DH-group), run Diffie-Hellman, exchange nonces. This produces SK_d (the parent key).
2. IKE_AUTH: authentication (certificates/PSK/EAP), identity exchange, creation of the first CHILD_SA (= the ESP tunnel).

Additionally:

• CREATE_CHILD_SA rekeys or sets up a new tunnel inside the same IKE_SA.
• INFORMATIONAL carries DPD (dead peer detection) and SA deletion.

IKEv1 in main mode needed 6 packets, aggressive mode needed 3, but with known vulnerabilities.

strongSwan, the main daemon on Linux

apt install strongswan strongswan-pki libcharon-extra-plugins

The modern config is swanctl (it replaces the old ipsec.conf driven by starter):

/etc/swanctl/conf.d/site-a-to-b.conf:

connections {

    site-a-to-b {

        version = 2

        local_addrs  = 203.0.113.10

        remote_addrs = 198.51.100.20

        local {

            auth = pubkey

            certs = my-cert.pem

            id = "CN=site-a"

        }

        remote {

            auth = pubkey

            id = "CN=site-b"

        }

        children {

            net {

                local_ts  = 10.10.0.0/24

                remote_ts = 10.20.0.0/24

                esp_proposals = aes256gcm16-prfsha384-modp3072

                start_action = trap          # autostart on first packet

            }

        }

        proposals = aes256-sha384-modp3072

    }

}

Load and start:

swanctl --load-all

swanctl --initiate --child net

swanctl --list-sas                       # active tunnels

NAT traversal (NAT-T)

ESP is IP-protocol 50, not TCP or UDP. NATs and firewalls usually drop it. For that case there is NAT-T: ESP is wrapped in UDP-4500, looks like an ordinary UDP packet from the outside, and passes through NAT without trouble.

IKE detects this itself:

• The handshake first runs over UDP-500
• If a NAT is seen on the path, both peers switch to UDP-4500 for ESP

On the firewall you need to open UDP 500 + UDP 4500 (plus ESP-50 when there is no NAT).

EAP for clients

The enterprise case: thousands of clients with a username/password or Active Directory. IPsec authenticates them through EAP (RFC 3748):

• EAP-MSCHAPv2 uses a password (legacy, but still works with AD)
• EAP-TLS uses a client certificate, like [[openvpn|OpenVPN]]
• EAP-RADIUS lets strongSwan delegate the decision to a RADIUS server

The IKEv2 + EAP combination is the native VPN on iOS, macOS and Windows 10+. In this rare scenario IPsec is simpler than its rivals: nothing to install, it is already built into the system.

strongSwan vs Libreswan vs racoon

• strongSwan is in active development, with the best support for IKEv2, EAP and plugins. The default on modern Debian and Ubuntu.
• Libreswan is a fork of FreeS/WAN, the default on RHEL and CentOS. Close to strongSwan in features, with a slightly different config syntax.
• racoon (ipsec-tools) has been dead for a long time. Do not use it.

Comparison with other VPNs

When something goes wrong

• establishing CHILD_SA failed is a traffic-selector mismatch. One side's local_ts/remote_ts does not match the other side's remote_ts/local_ts. The subnets must mirror each other.
• IKE_AUTH failed: AUTHENTICATION_FAILED means a wrong PSK, a wrong CN in the certificate, or bad EAP credentials. Check id = on both sides.
• NAT-T does not activate even though a NAT is on the path means the peer did not notice it. Fix it with forceencaps = yes: traffic is forced onto UDP-4500.
• peer not responding means UDP 500/4500 is blocked somewhere. Take a capture: tcpdump -i any -nn 'udp port 500 or udp port 4500'.
• MTU issues, mid-sized packets get lost: add tfc_padding or an explicit fragmentation = yes in IKE. In the kernel it helps to clamp the MSS: iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu.
• One tunnel comes up, a second one between the same hosts does not. Most likely an SPI conflict or overlapping traffic-selectors. ip xfrm state shows the SAs, ip xfrm policy shows the selectors.