Agent Horror Stories

Viewer discretion advised · Updated nightly

← Back to the feed
infra meltdown·reddit·

The Trigger Trap: How One Startup's Database Became a Runaway Queue

A recursive trigger loop transforms PostgreSQL into an accidental distributed queue, bloating a single application record to 1 million rows and crippling the database for a week while customers suffered.

Horrifying

## The Architecture of Doom

At a growing startup, the team made a fateful choice: turn PostgreSQL into a message broker. When recruiting applications received new video or audio evidence, the system relied on a global database trigger that would flag records for reprocessing. A worker would poll every minute, submit jobs for expensive async work, and those jobs would update downstream tables—triggering the flag again.

It was elegant in its horror. Clean schema. No external dependencies. What could go wrong?

## The Feedback Loop from Hell

Everything. A processing cycle formed: downstream update → trigger fires → flag set → worker resubmits → repeat. One application record metastasized to 1 million rows as each processing iteration logged another INSERT. The database, meant to serve recruiting decisions in milliseconds, became a write-locked tomb.

For a full week, customers staring at their dashboards watched load times balloon to 5-10 seconds for basic queries. The platform looked broken. It was broken.

## The Rejected Salvation

The engineer flagged the architectural rot immediately, proposing escape routes: move orchestration outside the database. Implement a transactional outbox pattern or event stream. Use a proper workflow engine like Temporal. Each suggestion was rejected. "We don't need it at our scale," leadership insisted—even as the scale was actively destroying the database.

No one had considered consistency either. Without 2-phase commit or a durable recovery mechanism, failed async jobs simply vanished. The system had no way to resume or track what actually happened.

## The Scariest Part

The engineer wasn't stupid. Their concerns were bulletproof. But in startup culture, the path of least resistance—using what's already there, avoiding "premature optimization"—often outlasts the engineer who warned about it. By then, the technical debt has compounded into technical horror.

Source: reddit.com · by u/LawBlue

More nightmares like this