Keep it live after the first boot

Day-2 starts with two questions an operator asks against the Fly app echo-portal: what did the machine print, and is it started on the release I expect. The first is a log read. fly logs -a echo-portal -i <machine> tails one machine’s output, and the boot lines are where the chapter’s four gotchas surface — the inetrc miss, the two identical :nxdomain faults, the engine child that exits and suspends. The trap is that a stopped machine has no live tail: reading a dead machine’s stream returns an empty stream, not the boot you wanted. To capture a fresh boot, restart the machine first — fly machine restart <id> -a echo-portal — and read the lines it prints coming up, rather than staring at a stopped machine that is printing nothing.

The second question is answered by status. fly machine status <id> -a echo-portal reports one machine’s state, fly status reports the app’s, and fly releases -a echo-portal reports which release version is current. Together they answer “is it started, and which release is running” — the two inputs the F7.5.2 triage tree branches on before it maps a boot symptom to its owning gotcha. The OTP release is portal; the Fly app is echo-portal; the database app is portal-postgres — three names that name three different things, kept distinct.

Select a day-2 operation below to read the command it runs and the fact its output supplies.

When the question is about the data, an operator inspects the production database directly — without exposing it to the public internet. fly proxy 15432:5432 -a portal-postgres opens a local tunnel: localhost:15432 forwards to portal-postgres.internal:5432 over Fly’s private network. With the proxy up, a local psql connects through it — the connection string is a shape like postgres://<user>@localhost:15432/portal, with no password written anywhere; the credential lives in fly secrets, referenced by name. The proxy is the seam: the DB never listens on a public port, and the inspection runs over a tunnel the operator opened and will close.

Two production facts from the bring-up bound the inspection. First, the connection must land on the writable primary, not a high-availability replica. SELECT pg_is_in_recovery(); answers in one boolean: f on the primary, which accepts writes, and t on a replica, which is read-only and in recovery. A write run against a t node fails for a reason that has nothing to do with the query — so the first thing to read is which node the proxy reached.

Second, a client/server psql version skew can break the \l meta-command (it issues a query the server’s catalog version does not answer the same way). The fix is to stop relying on the meta-command and query the catalog directly: SELECT datname FROM pg_database; lists the databases (the \l workaround), and SELECT version FROM schema_migrations; lists the applied migration versions. The catalog and schema_migrations are plain tables; querying them never touches a credential.

Select a check below to read its exact psql query and the result it returns over the proxy.

The migration path is the same release task the deploy ran, only now run by hand. Portal.Release.migrate/0 (in Portal.Release) loads the :portal app, starts Portal.Repo in isolation through Ecto.Migrator.with_repo/2, and runs Ecto.Migrator.run(&1, :up, all: true) — which applies only the pending migrations. That makes a re-run idempotent: against a schema that is already current it applies nothing, so re-running it after a deploy is a no-op. The 2026-06-06 run applied 20260604063825_create_courses and 20260604063826_create_events; with both already recorded in schema_migrations, a second migrate/0 adds nothing.

The reversal path is its sibling. Portal.Release.rollback/2 (in Portal.Release) runs Ecto.Migrator.run(&1, :down, to: version), run release-native as bin/portal eval "Portal.Release.rollback(Portal.Repo, <version>)". It rolls Portal.Repo back to and including the named version — the day-2 move when a migration has to be undone. Both tasks name only Portal.Repo and Portal.Release; neither reaches under the Portal facade, and neither prints a secret.

Above the migration tasks sits the recovery move. When a fault is transient — a stuck process, a machine wedged after a network blip — the first move is to redeploy (fly deploy rebuilds the image and rolls the machines) or to restart (fly machine restart brings one machine back up). When the fault is structural — a boot symptom that recurs on every restart — the move is not to restart again but to open the F7.5.2 triage tree and route the symptom to the gotcha that owns it.

# Portal.Release — the release-native migration tasks (run on the deploy machine)
# re-migration is idempotent: run/4 :up applies only the PENDING migrations
$ bin/portal eval "Portal.Release.migrate()"
  # first run (2026-06-06): applied 20260604063825_create_courses + 20260604063826_create_events
  # re-run against a current schema: NO-OP — nothing pending, nothing applied

# the reversal path: roll Portal.Repo back to (and including) a version
$ bin/portal eval "Portal.Release.rollback(Portal.Repo, 20260604063826)"
  # runs Ecto.Migrator.run(&1, :down, to: version)

# no secret is printed: DATABASE_URL / SECRET_KEY_BASE are referenced by name,
# provisioned by `fly secrets`; the psql inspection queries the catalog only.