Observability: the read model that acts

A read model is allowed to be wrong. It is a projection of something authoritative, so it can be rebuilt from that source at any time — which is the argument Workbench, the delivery index later in this series, is built on. Observability starts from the same shape — Rails, PostgreSQL, a background queue, and an upstream that owns the truth — and then does one thing Workbench never does.

It acts.

It polls Graylog every five minutes, clusters errors by fingerprint, and files GitLab issues. That last verb is the whole article. A projection that only reads can be deleted and rebuilt as often as you like. A projection that reaches out and changes the world has produced state that no rebuild can reproduce, and no rebuild can take back.

The read half really is disposable

Worth saying plainly, because it is the half people expect to be the problem: the clusters and observations are a projection of Graylog, and they can be rebuilt. There is a backfill that re-imports a window in five-minute buckets.

bin/rails "observability:backfill[me_obseed_beta,2026-03-14T00:00:00Z,2026-03-17T00:00:00Z]"
bin/rails "observability:backfill_last_days[me_obseed_beta,3]"

So the “metrics have no upstream, history is the product” instinct — which is where I expected this post to go when I sketched the series — turns out not to apply. Graylog is the upstream. The rebuild is bounded by Graylog’s retention rather than by anything in this application, and within that window the store is exactly as disposable as Workbench’s.

The interesting boundary is somewhere else entirely.

Every backfill is a loaded gun

Read that backfill command again and imagine running it against a live GitLab sync. Three days of history get replayed, every cluster crosses the threshold again, and a few hundred issues appear in a project on a Tuesday afternoon. Nothing was corrupted. The projection is, in fact, perfectly correct. It is the world that now has three hundred rows of garbage in it.

So backfills default to producing clusters, observations and dry-run issue previews, and nothing else. Filing real issues requires an explicit fourth argument.

This is the line that separates a read model from a system that acts, and it is not a line about databases. It is about which side of the system boundary a row ends up on.

dry_run is a column, not an environment variable

The first version had a dry_run flag in configuration, which meant YAML plus an environment variable, which meant that whether a run would touch GitLab depended on how the process happened to be started.

It is now a persisted attribute on the Integration record, and the decision record states it as a rule with no escape hatch: no global environment variable may force or override dry-run behaviour.

The same reasoning moved the whole configuration into the database: Integration owns the Graylog-to-GitLab mapping, thresholds, noise filters and polling interval; GraylogConnection and GitlabConnection own endpoints, auth schemes and encrypted API tokens. Adding a monitored application stopped requiring a deploy.

The trade is real and worth stating. Configuration used to be YAML in git, with a history, a diff and a review. It is now rows in a table. We lost git log on config, and gained the property that the dashboard cannot lie about what is running.

The migration had a delete date written into it

Moving configuration to the database was done as a deliberate dual-source period: YAML and database, selected by a flag, with an import task and a parity-verification task alongside. That is the ordinary safe way to do it.

What was less ordinary is that the ADR which introduced the dual source also specified its removal — including the release ordering, which is the part that bites:

1. Deploy or keep the release that still contains the import and parity tasks.
2. Set the permanent encryption keys.
3. Run observability:import_yaml_configs, then observability:verify_parity.
4. Inspect the imported connections, integrations, token status and — specifically — the persisted dry_run values.
5. Only then deploy the removal release.

Writing the cleanup into the same decision that introduced the temporary machinery is a habit I would recommend to anyone. Dual-source periods do not end on their own. They get forgotten, and then two years later nobody can tell you which source the running system is actually reading.

There is no delete button, on purpose

The dashboard can create, edit, enable and disable an integration. It cannot delete one.

That is not an oversight and not a missing feature ticket. Disabling preserves the execution history — every run, its trigger type, stage timings and counters — and makes the operational state explicit rather than absent. An integration you disabled two months ago is a fact about your system. An integration you deleted is a gap that looks exactly like one that never existed.

The admin API keeps DELETE /api/v1/integrations/:id for controlled administrative work, which is the honest split: destructive operations exist, but they are not one mis-click away from the list view.

The guard has to sit somewhere the copy cannot reach

This is the part I would most want someone to take away, and it took a second design pass to get right.

There is a task that copies the production database into local development. It is deliberately byte-exact: credentials, ciphertext, PII, audit history and job history are all preserved, because a sanitised copy is a copy you cannot debug with. It then applies a small transactional overlay — clear pending jobs, set every integration to enabled=false and dry_run=true, flip ar_internal_metadata.environment.

Look at that overlay closely. It sets configuration. And we had just spent a release making configuration authoritative, persisted, and copied along with everything else.

If the overlay were the only protection, then a copy of production that ran one migration wrong, or a developer who enabled one integration to reproduce a bug, would be a local machine holding real Graylog and GitLab tokens with the live sync switched on.

So the actual guard is not configuration at all. Outbound Graylog and GitLab calls are blocked at both Faraday request boundaries unless Rails is running in production and the deployment sets exactly SQUIBBLE_OUTBOUND_EFFECTS_ENABLED=enabled. Copied tokens cannot get past it. Copied integration flags cannot get past it. Nothing that travelled in the database dump is on the decision path.

What I would keep

• Ask what a rebuild does to the outside world, not just to the database. “Disposable” is a property of a projection with no effects. Add one outbound write and the word means something else.
• Put the switch that governs a side effect in the same place the operator reads it. A behavioural flag in the environment means the dashboard is describing a system that may not exist.
• Write the removal of your migration scaffolding into the decision that creates it, including which release has to still be deployed when.
• Guard side effects at the request boundary, not in configuration — especially when your configuration is something you copy between environments.

The next post in this series keeps the side effects and changes the threat model: a secrets store built so that the server cannot read what it stores.