Why a Docker Container Accumulated 75,582 Zombie Processes
A Rust ingestion service kept running while Chromium helpers accumulated. The fix needed orphan reaping, bounded shutdown and cancellation-safe cleanup.
The ingestion scheduler was running. Browser requests could succeed. The container was passing its health check. It had also accumulated 75,582 zombie processes.
This happened in a production ingestion system I build and operate. The service used Rust and launched Chromium when a page needed browser rendering. The defect involved what happened after those browser processes had finished, particularly when their descendants outlived their original parents.
The production count made the problem hard to miss. A much smaller local reproduction made it possible to explain and test.
The successful request that left four zombies
I reproduced the leak with one successful browser request in a container without init. After the request completed, four exited Chromium-related processes remained as zombies under PID 1.
A timeout was not required. Neither was a failed page load. Ordinary successful work could contribute to the accumulation.
That reproduction and the production count are separate pieces of evidence. I did not inspect every process in the production total or measure its exact growth rate. Four leftovers from one local request cannot be multiplied backwards to reconstruct the entire incident.
What was still there after the browser exited
An exited process leaves a small kernel record containing information such as its PID and exit status. Its parent normally collects that information with a wait operation, allowing the record to be released. Until that happens, the process can remain in the zombie state. It is no longer executing browser code. Linux wait documentation
Chromium creates several processes. Some descendants became orphaned when their parents exited. In this container, the Rust scheduler was PID 1, so it inherited responsibility for those orphaned processes.
Managing the browser children launched directly by the application was insufficient. The scheduler did not implement a general orphan-reaping loop. The container's PID namespace explains why the host's init process did not simply collect these descendants while the container's own PID 1 remained alive. Linux PID namespace documentation
The health check only established that PID 1 existed. That answered whether the scheduler was alive, without answering whether completed browser jobs had released their resources.
Cleanup needed more than an init process
I enabled Docker init in both ingestion configurations. This adds a process responsible for forwarding signals and reaping adopted descendants. It addresses the missing orphan-reaping behavior. Docker Compose init documentation
The application's own cleanup still needed work. An init process cannot substitute for deciding when a browser should close, how long to wait, or who owns the temporary profile when a request is cancelled.
I made those responsibilities explicit:
| Change | Purpose |
|---|---|
| Request browser shutdown and wait for exit | Finish collecting the directly managed browser process |
| Separate cleanup deadlines | Allow cleanup to run after the request has timed out |
| Forced termination fallback | Bound the wait for a browser that does not shut down |
| Transfer ownership on cancellation | Keep the browser, protocol handler and profile under cleanup control |
| Retain the profile through shutdown | Avoid deleting files while Chromium can still write them |
Graceful shutdown gets a five-second deadline. If it does not finish, cleanup escalates to termination and another bounded wait. The point is to give cleanup its own opportunity to finish, rather than let the expired request deadline cancel it immediately.
The temporary-profile race
Testing exposed a second problem. Cleanup could remove a temporary profile while Chromium was still shutting down. The browser could then recreate files in the same location.
That leaves an easy trap for a test: a directory can disappear briefly without actually being cleaned up for good.
The revised ownership follows the browser through shutdown. Cancelling the request transfers the browser, protocol handler and profile into a bounded cleanup task. The profile remains owned until the browser has exited, so filesystem cleanup happens at the right point in the lifecycle.
This also shaped validation. I needed to check for both process leftovers and temporary profiles after cleanup had completed. Either check on its own would have missed part of the defect.
Containment and deployment
I added a default limit of 2,048 processes and threads to the ingestion container. That is a containment measure for future runaway behavior, not a replacement for cleanup. Linux's PID controller counts kernel tasks, including threads; reaching the limit prevents further creation rather than restarting the container. Linux PID controller documentation
The container also received a 40-second stop grace period, giving the scheduler's existing 30-second shutdown window room to finish.
I recreated ingestion to clear the existing process accumulation and apply the updated configuration. Restarting the existing container would have cleared its old processes without applying the new Compose settings. Recovery needed both the replacement environment and the lifecycle changes.
What the tests established
Validation covered 27 unit tests and 26 lifecycle scenarios. These were distinct groups, not two descriptions of the same suite.
The lifecycle checks included successful requests, failed launches, oversized responses, timeouts, cancellation and a deliberately frozen browser that required forced termination. The final Linux run left zero browser processes, zero zombies and zero temporary profiles. The regression is now included in CI.
I also did not capture production CPU, memory or latency measurements during the incident. The process count supports the diagnosis and recovery work; it is insufficient to assign every server slowdown to one cause or to publish a speed-improvement percentage.
The useful check comes at the end of the lifecycle: after successful work, a timeout or cancellation, has the browser exited and has its profile actually gone? Returning from the request handler was not enough. Those cleanup paths now have explicit ownership and regression coverage.
My Rust projects - My Docker projects - My web crawling projects - My Linux projects