One symptom, three faults

The chapter's four-gotcha spine stops being a table and becomes an operational decision tree: read the boot symptom in fly logs, map it to the gotcha that owns it, apply the fix, and route the change to the module that owns the fix. The four faults stack — each fix surfaces the next, so the boot error barely changes — and three of the four crash with the identical :nxdomain on the database host. One command tells those three apart: getent hosts <db>.internal from another running machine in the org. A success there while the app still :nxdomains proves the name and the network are fine, so the fault is the app's address family, not the wire.

This page covers the four symptom → gotcha → fix → owning-module rows; the cross-machine getent discriminator that separates the two identical-:nxdomain faults (gotcha 2 vs gotcha 3); and the second discriminator above all four — if even the app's own echo-portal.internal fails to resolve from inside the machine, the 6PN DNS itself is the problem, not any target.

Symptom to fix The decisive discriminator Route it to its owner

The spine, read as a decision tree

Each gotcha is silent until boot — a value that "looks right" never fails a build; a wrong one fails a boot. So the runbook starts where the evidence is: the boot lines in fly logs. One symptom names one gotcha, the gotcha names a fix, and the fix lives in exactly one module of this chapter. Read the rows in order; the faults stack, so a clean boot needs every earlier fix already in place before its symptom can clear.

The first symptom is unambiguous. inet_config: file /app/inetrc not found is gotcha 1: fly.toml [env] sets ERL_INETRC = "/app/inetrc", but the image never shipped that file, so the BEAM falls back to a resolver that cannot see Fly DNS. The fix is a COPY infra/inetrc /app/inetrc in the runtime stage, after WORKDIR /app and with no --chown — and it lives in F7.1, the image. The last symptom is also unambiguous: failed to start child Portal.Engine → exit 1, then max restart count of 10 and a suspend, is gotcha 4 — Portal.Engine.init/1 does a synchronous read of the event log at boot, so an unreachable or unmigrated database hard-crashes the whole release. Its fix is upstream: make the database reachable (gotchas 2–3) and migrated (F7.3); the lazy, retried-replay hardening is named but not built, because it would touch the engine.

Between those two sit the two that share a symptom — both crash with :nxdomain on the database host — and they are the whole reason the tree needs a discriminator. Select a symptom below to read the gotcha it names, the one-line fix, and the module that owns the change.

Triage map · symptom → gotcha + fix → owning module

One symptom names one gotcha, one gotcha names one fix, and one fix lives in one module — the runbook's job is to make that chain mechanical, so a crash-loop becomes a lookup instead of a guess.

The decisive discriminator

Gotcha 2 and gotcha 3 emit the identical :nxdomain on the database host, so the symptom alone cannot tell them apart. Fly's private names — <app>.internal (direct 6PN) and <app>.flycast (Fly-proxy private) — resolve only for a running instance in the same org, and they carry only AAAA (IPv6) records. Gotcha 2 is the name not resolving at all (the database is stopped, or in another org); gotcha 3 is the name resolving fine while the app, defaulting to an IPv4 A-query, asks for a record that does not exist on an AAAA-only name. Same error, two different layers.

One command separates them. From another running machine in the org, run getent hosts <db>.internal — glibc's getaddrinfo, the same resolver path Erlang's native lookup takes. If that succeeds (it returns an AAAA) while the app still :nxdomains, the name and the network are fine, so the fault is the app's address family — gotcha 3, the decisive fix: socket_options: [:inet6] on Portal.Repo. If it fails there too, the name itself is unreachable — gotcha 2, the network: keep the database running and same-org (Managed Postgres additionally needs .flycast + fly mpg attach). A second discriminator sits above all four: if even echo-portal.internal — the app's own name — fails to resolve from inside the machine, the machine's 6PN DNS is the problem, not any target.

The 2026-06-06 bring-up settled exactly this. The inetrc COPY (gotcha 1) was corrected and the database kept warm (gotcha 2), yet the machine still :nxdomain-crashed on portal-postgres.internal. The discriminator decided it: getent hosts portal-postgres.internal succeeded from inside the running, same-org portal-postgres machine — so the name was fine, and the decisive fix was gotcha 3. Run the discriminator below to read which branch each outcome lights.

# From ANOTHER running machine in the org — the resolver path Erlang's native lookup takes
# glibc getaddrinfo, the same as the BEAM's {lookup, [native, ...]} in infra/inetrc
$ getent hosts portal-postgres.internal
fdaa:0:1234::3  portal-postgres.internal     # SUCCEEDS → an AAAA record. Name + network are fine.

# …while the app, on its IPv4 default, still logs :nxdomain on the same host.
# Verdict: not the network (gotcha 2) — the app's address family (gotcha 3). The decisive fix:
maybe_ipv6 = if System.get_env("ECTO_IPV6") in ~w(true 1), do: [:inet6], else: []
config :portal, Portal.Repo, socket_options: maybe_ipv6   # consume ECTO_IPV6, don't only set it

The getent discriminator · which branch does the outcome light?

The symptom is the same :nxdomain; the discriminator is the decision — one getent from a second machine separates the app's address family from the network, and a failed own-name lookup separates a broken machine from a broken target.

Route it to its owner

Triage is only finished when the change lands in the module that owns it. Gotcha 1 is an image fact, so its fix is a Dockerfile line in F7.1. Gotcha 3 is a runtime-config fact, so its fix is a line in runtime.exs — F7.2. Gotchas 2 and 4 are database facts: the first keeps the instance running and same-org (a deploy concern shared by F7.2 and F7.3), the second needs the schema migrated before the engine's boot read can succeed (F7.3). Routing the fix to its owner is what keeps the runbook honest: it never patches a symptom in the wrong layer, and it never re-opens a fault a prior module already closed.

The naming matters here, because three names look alike and mean different things. The Fly app is echo-portal; the database app is portal-postgres; the OTP release is portal. A symptom on portal-postgres.internal is a database-host resolution problem; a suspend of the echo-portal machine is the app crash-looping; and bin/portal is the release binary you run a migration or a rollback through. Conflating them sends a fix to the wrong owner.

the symptom you read

a boot line in fly logs — inetrc not found, a :nxdomain on the DB host, or a Portal.Engine child that exits 1 and suspends after ten restarts.

the owner you route to

the image (F7.1), the runtime config (F7.2), or the database (F7.3) — the fix lands in the one module that owns the fault, never patched in the wrong layer.

The runbook turns a crash-loop into a lookup, and the discriminator turns one ambiguous symptom into a decision. With the faults triaged, the last dive keeps the endpoint live after the first boot — F7.5.3 · day-2 operations: logs, status, the fly proxy + psql inspection, re-migration idempotency, and rollback. The decisive fix's home is the IPv6 dive, F7.2.2 · Fly's IPv6-only private network.

References

Where each gotcha's symptom appears (fly logs), the IPv6-only private network the two :nxdomain faults share, the socket_options address family that fixes the decisive one, and the resolver path getent and Erlang's native lookup both take.

Sources

Fly.io — flyctl logs — fly logs, the boot output where each gotcha's symptom appears.
Fly.io — Private networking — .internal/.flycast, the IPv6-only 6PN, and the AAAA-only resolution that is the gotcha 2/3 substrate.
Postgrex — socket_options/[:inet6] and the :nxdomain an address-family mismatch surfaces.
Erlang/OTP — Inet configuration — the resolver and the native lookup path getent shares.

Related in this course

F7.5 · Portal is live — the module hub this dive triages within.
F7.5.1 · Liveness verification — the prod standing-liveness check the runbook restores a boot to.
F7.5.3 · Day-2 operations — keeping the endpoint live after the first boot: logs, status, proxy-and-psql, rollback.
F7.2.2 · Fly's IPv6-only private network — gotcha 3's home: the set-versus-consume trap behind socket_options: [:inet6].
F7 · Deploy to Fly — the chapter landing and its four-gotcha spine.