DeadBro is built to give Rails developers deep visibility into what their application is doing. Depending on how users configure it, DeadBro can log all requests or only a selected sample. But when a request is stored, we try to keep the useful details: traces, SQL queries, timings, memory information, garbage collection data, queue time, database wait time, and everything else that helps explain what happened during that request.
That level of detail is useful, but it comes with a cost.
Recently, one of our request detail tables grew from around 10GB to more than 300GB in a single month.
At first, that was not a dramatic problem by itself. Large tables are not automatically bad. But the rest of the system also needs space: normal database tables, backups, logs, the operating system, temporary files, and everything else running on the server. Eventually, the OS drive reached around 80% usage, which was high enough that I needed to solve the problem properly instead of just deleting old data and hoping for the best.
The first idea: move the table to another database
The obvious first idea was to move the request details table to a separate Postgres database.
In theory, that would work. The large table would live somewhere else, and the main application database would stay smaller. But in practice, this meant adding a lot of operational complexity.
I would need to set up another Postgres server, copy existing data, manage backups for two databases, monitor two systems, and make sure the application handled both databases correctly. For a table that mostly stores large request payloads and is rarely queried directly, that felt like the wrong trade-off.
The data itself was already a strong hint that Postgres was not the ideal final storage layer. Most of the request details were already stored as JSON columns. The structure matched a document-style format very naturally.
So instead of asking, “How do I move this table to another database?”, I started asking, “Does this data need to be in Postgres at all?”
Why JSON files made sense
Request details in DeadBro are usually written once and read occasionally. The write volume is high, but the read volume is relatively low.
That made them a good fit for object storage.
The details could be serialized as JSON, compressed, and stored as files. The main database would keep the metadata needed to find and display the request, while the heavier payload would live somewhere else.
This would reduce the pressure on Postgres, keep the main database leaner, and avoid using a relational database as a storage layer for large JSON blobs.
Why not S3 or Cloudflare R2?
S3-style storage was the right model, but using S3 or Cloudflare R2 directly was not the right fit for DeadBro at this stage.
The reason was write volume.
DeadBro can generate close to 3 million writes per day, while only doing a few thousand reads. That is a very write-heavy workload. Using a hosted object storage provider would have meant paying not just for storage, but also for a large number of write operations.
For a system where the access pattern is mostly “write now, maybe read later”, the economics did not look great.
Why local storage was not enough
The next obvious option was local disk storage.
That would be simple and cheap, but it had a major downside: DeadBro runs on dedicated servers. If request details are written directly to local disk on one application server, that creates a coupling between the data and that machine.
That is not ideal. It makes moving servers harder, scaling harder, and recovery more fragile. I wanted something that behaved like object storage, but without relying on a hosted provider for millions of daily writes.
The solution: MinIO on a separate server
The final decision was to set up MinIO on a separate server.
MinIO gives DeadBro an S3-compatible object storage layer while still keeping the infrastructure under my control. From the application’s point of view, it can write request detail files using an S3-style API. From an infrastructure point of view, I can run the storage on a server that is separate from the main application and database.
This gave me the architecture I wanted:
- Postgres remains the source for structured application data and request metadata.
- Large request details are stored as compressed JSON files.
- The storage layer is separate from the OS drive and main database.
- The app can still retrieve the details when they are needed.
- I avoid maintaining a second Postgres database just for large JSON payloads.
Rolling it out safely
I did not want to switch everything over in one step.
First, I created a service that wrote the request details to both places: the existing Postgres table and the new S3-compatible storage. This made it possible to test the new storage path without risking data loss.
After a successful staging test, I enabled the dual-write system in production. For a while, DeadBro wrote the same request details to both Postgres and MinIO. That gave me confidence that the object storage path was working correctly before making it the primary storage mechanism.
Once the production test looked good, the next step was to stop writing the heavy request details to Postgres and write them only to MinIO.
The result
The main benefit is that Postgres no longer needs to absorb hundreds of gigabytes of request detail payloads every month.
That should keep the main database smaller, reduce pressure on the OS drive, and make the infrastructure easier to reason about. It also gives DeadBro a cleaner separation between structured data and large request payloads.
Postgres is still used for what it is good at: indexing, relationships, metadata, and querying. The large request details now live in a storage system that better matches how they are actually used.
I will write a follow-up after this has been running in production for a while, especially with notes on compression, storage growth, failure handling, and whether the MinIO setup remains the right trade-off at higher volume.