The database is up and the application cannot find it by name

You are on call for the orders-api service. This morning a message came in: the application cannot see the orders database. The database sits on the same machine, everything worked yesterday, and, as usual, "nobody changed anything". You log in and run the health check.

The symptom

The check names the database host and fails before it gets to connecting. The resolver, the part of the C library that turns a name into an address, does not know the name. The database is listening all the while.

$ orders-api-check
orders-api-check: [db] host=orders-db.internal port=5433 (/etc/orders-api.conf)
db: [Errno -2] Name or service not known

$ getent hosts orders-db.internal
$ echo $?
2

$ ping -c 1 orders-db.internal
ping: orders-db.internal: Name or service not known

$ ss -ltn 'sport = :5433'
State  Recv-Q Send-Q Local Address:Port Peer Address:PortProcess
LISTEN 0      5          127.0.0.1:5433      0.0.0.0:*

The port is open and the database process is alive, and the application never reaches it: it trips earlier, on turning the name into an address. The last command in the listing - ss -ltn - lists listening TCP sockets: -l for listening only, -t for TCP, -n for numeric ports, and the 'sport = :5433' filter keeps just the port we care about. The -2 in the error and the empty getent answer with exit code 2 say the same thing: the name