A healthy service, a successful release, and a public 503

The publication succeeded. The new release was active. The services were running and their health endpoints responded. Visitors received HTTP 503.

This happened on a publishing platform I build and operate. A rollback restored the public site, but only temporarily: automated publishing could activate another release that the public server could not load.

The immediate task was to restore access. The lasting fix required following the release from the service that generated it to the service that served it, and checking what each component considered acceptable.

The important boundary was between a completed publication and a release the public reader could actually use.

The release was valid, but too large for its reader

Elixir generates an immutable release with rendered files, a content manifest and an inventory of render results. The Rust public server validates that metadata and its integrity hashes before serving the files.

The reader had a 16 MiB ceiling on metadata files. As publication content grew, the complete manifest crossed it.

Manifest	Size	Result under the old reader
Previous release	16,617,318 bytes	Accepted
Newly activated release	16,805,625 bytes	Rejected
Reader limit	16,777,216 bytes	16 MiB

The replacement was only 28,409 bytes over the limit. That small excess was sufficient to prevent the public reader from loading the active release.

The error described invalid metadata. In this case, the confirmed failure was the reader's size restriction; the message did not establish that the JSON was malformed. The publisher had completed its work successfully, while the reader applied an additional condition that the release did not meet.

Why the green checks did not help

The health checks showed that services were alive and could respond. They did not prove that the active release for the affected site could be loaded.

That left room for all of these observations to be true at once: the publisher completed, activation succeeded, containers were healthy, and visitors could not open the website.

I rolled back to the smaller release and confirmed HTTP 200. That established a useful difference between the two releases and restored access, but it left the underlying condition intact. Another automated publication could cross the same boundary. No Nginx configuration change was needed to resolve this failure.

Reading the manifest incrementally

I replaced the manifest's size-limited loading path with incremental parsing and hashing. As the reader parses the file, it keeps the routing information needed to serve pages and discards editorial payloads that do not belong in the serving index.

The integrity check still covers the exact bytes of the manifest. It is not a hash of a reconstructed JSON document or a selected subset of fields. The reader also continues to validate the JSON itself.

Removing the total manifest-size ceiling did not remove every limit. Bounds remain on individual tokens, nesting, routes, inventory and the memory occupied by the serving index. Those are constraints on structures the reader processes or retains, rather than a single cap on the entire document.

That distinction mattered. Simply increasing the original ceiling would have moved the same growth-dependent failure further away. Removing all constraints would have created a different resource-management problem. The change allowed a larger source document while preserving the reader's integrity and resource checks.

Two other symptoms needed separate explanations

The server also had a browser-process cleanup problem. Go and Lighthouse sidecars handled browser jobs in two deployment slots, and process inspection found 9,191 zombie children across those four containers.

Cleanup could inherit an already-cancelled job context. Chromium could also start and then fail during connection setup, or acknowledge shutdown without exiting. In another path, descendant cleanup stopped checking after the original browser leader exited.

I gave cleanup an independent bounded context, covered failed connections and incomplete shutdown, and checked owned descendants even after their leader exited. Container init was enabled so adopted orphan processes could be reaped. Docker documents this process-reaping role for Compose's init setting.

The process buildup required fixing, but it was not the demonstrated cause of the manifest rejection. Nor did its count establish the cause of every server slowdown. Keeping those explanations separate avoided turning a multi-symptom incident into one unsupported theory.

There was also a cancelled PostgreSQL advisory-lock query during publication coordination. The application uses transaction-scoped locks to serialize changes; PostgreSQL explains their scope in its advisory-lock documentation. The logs established failed lock acquisition, not a deadlock or a causal link to the browser processes. This patch did not rewrite that locking mechanism.

The old deployment slot still mattered

Updating the active slot alone would have left an old public reader and existing browser-process buildup in the retained slot.

After verified cutover, deployment now also rebuilds and recreates the former slot's reader and browser sidecars. That replaces the incompatible reader and removes the previous process environment. The lifecycle changes and init process address recurrence.

What the tests established

The most useful reader regression exercised the production Rust container with a 163.9 MB manifest containing 5,000 page payloads. It served that release successfully. An invalid integrity seal was still rejected.

Those two checks mattered together. Accepting a large file would have been weak evidence if the change had achieved it by bypassing integrity validation.

The separate process reproduction produced 160 zombies without init and zero with init. Recreating a container containing both zombies and live orphan processes cleared the existing buildup.

Broader validation included the Elixir, Rust, Go and Node suites, focused Go race tests, and production image builds. Those suites provided regression coverage around the change; they were not all newly written tests for this incident.

After deployment, I confirmed that the public site was serving again and the observed zombie buildup was gone. I did not measure a before-and-after CPU or memory improvement, and the observed recovery interval is not an exact outage-duration measurement.

The lasting lesson is a concrete check: after a release is activated, exercise the public reader against that release. A completed publication and a responsive health endpoint each establish something useful. Neither, on its own, establishes that visitors can open the site.

What's interesting is that it's happened the same day as Why a Docker Container Accumulated 75,582 Zombie Processes - Linux/Rust/Elixir/Docker/PostgreSQL and software testing work all day!