Open source · v0.1.0Project 02

Error trackingSelf-hostedOpen source

trapline

One binary, one SQLite file and four clients of the same API: REST, web panel, CLI and MCP. Designed so an agent can read the error, patch the code, run the tests and close it for the next release without leaving your machine.

GoSQLite (sin CGO)FTS5MCPHexagonalAGPL-3.0

Sentry self-hosted asks for 16 GB of RAM. trapline does the job with 19.7 MB.

Error tracking, tracing, uptime, crons and a status page in an 18 MB binary. The official Sentry SDKs stay as they are: change the DSN and it works.

Sentry self-hosted · minimum published in its repo16 GB

trapline · measured idle, nightly gate19.7 MB

At true scale the trapline bar is 0.12% of the other one. A small-install comparison: at large-company scale that stack does things that do not exist here.

19.7 MBidle RAM, minimal profile

611 ev/singest measured in bench

6official SDKs tested in CI

01 · How it works

An error travels from the SDK to the agent without leaving the process

Click through the steps or let it play. Each step lights up the piece it talks about.

Your app

official Sentry SDK

Ingest

pure parser

SQLite

one transaction

/api/v1

a single API

4 clients

panel · CLI · REST · MCP

Step 1 of 4 · plays on its own

Your app throws an exception

The official Sentry SDK catches it and packs it into an envelope, the same format as always. trapline has no SDK of its own: it reuses the six already proven in production. The only thing that changes in your code is the DSN.

The minimal version that works: you do not install a new SDK, you change one variable.

# before
SENTRY_DSN=https://<key>@o123.ingest.sentry.io/456
# after · example
SENTRY_DSN=https://<key>@errors.your-domain.com/1

02 · An end-to-end example

From throw to closed issue

A made-up case with small numbers, to see what happens at each stage before talking about architecture.

14:02 · production · release 2.4.0

Checkout breaks

A Node app throws TypeError: Cannot read properties of undefined (reading 'id') at checkout.js:42. The official SDK sends it as is.

14:02 → 14:40

Three events, one issue

The fingerprint is computed over the normalized stacktrace, without the release directory or volatile numbers. The three afternoon events land in the same issue, not in three. A deploy does not open fresh issues.

CLI or MCP

One document instead of five calls

trapline issues bundle -project 1 -issue 17 returns a markdown document: exception, symbolicated stacktrace with context, breadcrumbs, aggregated tags, frequency over 24 h and 14 d, and the suspect commits with the reason for the suspicion.

skill /fix-error

The agent writes the failing test, patches and runs the suite

It reads the bundle before opening any file. It closes the issue as resolved in the next release. It never pushes, never deploys: what reaches production is decided by a person.

release 2.4.1

If it comes back, it reopens on its own

The reopen rule lives in the same ingest transaction: an event for the issue on a release later than the one that resolved it reopens it and notifies. A regression does not get buried under a "resolved".

03 · The challenges

Five times the obvious solution was wrong

Each challenge follows the same pattern: what anyone would do first, why it does not work, what was done, and the number that proves it. The technical detail stays folded.

The headline figure, measured every night

The obvious approach

Start the process, look at how much RAM it uses, publish the number.

Why it fails

That number is only true with nobody using the product. The repository published three false figures: one login with Argon2id at 64 MiB pushed RSS from 9.6 to 142 MB and it never came back down; the zstd encoder allocated per-core state, 16 MB before the first event; 200 simultaneous logins took the process to 1.03 GB.

■ before■ afterpeak RAM

Idle, 24-thread machine (per-core zstd → concurrency 1)

33 MB

17 MB

200 logins at once (10/min per-IP limit + 2 hashes in flight)

1.03 GB

52.9 MB

What was done

The peak is measured (VmHWM, the kernel's high-water mark), not a VmRSS sample, resetting the mark before each measurement. A script measures it and a nightly gate measures it again and fails if it goes up. Argon2id dropped to the parameter OWASP recommends and the process returns the working set after the burst.

19.7 MBidle with the minimal profile. 34.2 MB with everything on, 38.7 MB peak while ingesting.

Technical detail

The gate also checks that the six background jobs of the loaded profile are running: a subsystem that failed to start would make the figure falsely good.

The rest of the footprint, measured by scripts/footprint.sh: an 18.13 MB binary against a 30 MB budget that breaks the build; 695 bytes per item on disk (86.1 MB for 130,000 items with zstd payload, indexes, FTS5, buckets and sketches); a median of 611 to 628 events/s in make bench with the gate threshold at 150.

Measured on a single machine (WSL2, Linux 6.6, Go 1.26.6). A number from one machine is an anecdote; what turns it into data is that it gets measured again every night.

Compatible with SDKs you do not control

The obvious approach

Read the protocol documentation and write test envelopes by hand.

Why it fails

That tests the project's idea of what an SDK sends, which is exactly what is in doubt. The protocol is polymorphic: timestamps in four forms, tags as an object or as pairs, exceptions wrapped or bare. And the worst failure passed every test: a CSRF middleware applied to ingest meant no official SDK could deliver an event.

✓ every PR

Python

✓ every PR

Node

✓ every PR

Browser

✓ nightly

PHP

✓ nightly

Dart

✓ nightly

sentry-cli: its /api/0/ surface was implemented by recording the real tool's traffic, not by reading its documentation.

What was done

Each row of the compatibility table is a suite that sends real errors with the official SDK, at the pinned version, against a real server, and checks what arrived and how it was grouped. The parser is pure and fuzzed.

5.8Mfuzzing runs over the parser without a crash. 6 SDKs and sentry-cli in CI.

Technical detail

CSRF only makes sense where there is an ambient credential, the panel cookie. Ingest authenticates with a key the client presents, so the middleware does not apply there. The suite with real SDKs caught it, not the API unit tests (ADR 02).

The /api/0/ surface that sentry-cli expects (releases new, set-commits, finalize, deploys new, sourcemaps upload) has its recorded fixtures committed in compat/sentry-cli/fixtures/ (ADR 13).

A thousand requests, not one

The obvious approach

Set per-request caps: 20 MiB per envelope, 4 MiB per item, 100 items, and a reader that cuts off gzip and zstd expansion.

Why it fails

All of them existed and were correct, and none of them answered the attacker's question: how many requests can hold those caps at the same time. 128 zstd bombs, 151 KB in total, took the process to 1.46 GB.

■ before■ afterMB of RAM at peak

128 gzip bombs 976:1, concurrency 32

649 MB

121 MB

128 zstd bombs 21,346:1, concurrency 32

1.46 GB

93 MB

512 zstd bombs, concurrency 128

grew without bound

121 MB

What was done

A 32 MiB memory budget shared by every in-flight ingest request, reserved up front for the decompressor. Underneath, two separate per-IP limiters: ingest at 48,000 per minute, applied before reading the body; authentication at 10 per minute, because what is being rationed is 19 MiB per attempt.

+4%more memory when the gzip flood is quadrupled (+30% for zstd), not +400%. The gate asserts it as a property.

Technical detail

The ingest 429 goes bare, with Retry-After and never with X-Sentry-Rate-Limits: that header makes the SDK stop sending the category for minutes, which would turn a defence into data loss (ADR 39).

Client identity is decided by a pure package, internal/clientip: X-Forwarded-For is only trusted if the direct peer is in TrustedProxies, IPv6 is counted per /64 and the map is capped at 16,384 entries (ADR 23).

Percentiles without storing rows

The obvious approach

Store each minute's p95 and average the sixty to get the hour's.

Why it fails

Percentiles do not compose. The average of sixty p95s is not the hour's p95, and a minute with three slow requests weighs the same as one with ten thousand fast ones.

Minute A · 3 requests

p95 = 900 ms

Minute B · 10,000 requests

p95 = 40 ms

Average the p95s

470 ms ✗

Three slow requests weigh the same as ten thousand fast ones.

Merge the histograms

≈ 40 ms ✓

10,003 requests, 3 slow: the real p95 of both minutes together.

What was done

Each window stores a DDSketch-style logarithmic histogram whose merge is exact: the hour built by folding sixty minutes is, bit for bit, the one those requests would have produced. The percentile is computed on read and never persisted. Release health follows the same rule: four counters per (release, environment, hour), never a row per session.

1%relative error in the sketch, with exact merge. Zero rows per request and zero rows per session.

Technical detail

Counting distinct sessions requires state: an in-memory window capped at 50,000 entries with a one-hour TTL, flushed every 60 s and on shutdown. A restart loses what was in flight; when the window fills up, precision is sacrificed, never stability. Both are declared to the user in every response (ADR 07, ADR 20, ADR 08).

Storing each session separately is a different product, session analytics, which is where the incumbent spends its infrastructure. It is on the list of things this product will never do.

Agent-first: the issue as a document

The obvious approach

Hand the agent the REST API and let it figure things out.

Why it fails

For a single bug it makes five calls and infers the shape from each JSON all over again. Worse: an exception message with a line break, written by anyone who holds a DSN, could open its own heading in a document the agent is about to act on.

REST · 5 calls

issueeventstagscommitshistory

and infer the shape from each JSON all over again

→

bundle · 1 document

# TypeError: … (reading 'id')
## Stacktrace
## Breadcrumbs
## Tags · Frequency
## Suspect commits

What was done

The bundle is deterministic markdown: every map is sorted before being written, it reads no clock, it has golden fixtures, and every value that comes from an event goes through a function that collapses line breaks. The MCP server is embedded in the binary and does not call the use cases: each tool declares a request to /api/v1 with the credential it arrived with. A read-only token cannot resolve an issue, by the same line that rejects the POST.

10 toolsMCP over stdio or POST /mcp. The official SDK cost 2.03 MB of binary, measured.

Technical detail

The /fix-error skill never marks something resolved whose test does not pass. Conscious debt: the bundle does not include the waterfall of a slow transaction or the release's crash-free rate; they answer a different question and have their own tools. Putting everything into one document would turn it into the dump nobody reads to the end (ADR 22, ADR 06).

"A number from a single machine is an anecdote."
Every public figure of the product has a script that measures it and a gate that measures it again: footprint, binary size, events per second, compatibility with each SDK, parity across the four clients, and even the code blocks in the documentation, which run against a real server.

04 · Decisions

One filter: does this add a process, a dependency, or idle memory?

If the answer was yes, an alternative was sought. If there was none, it was measured and published with its gate.

WHY

Why SQLite and not a columnar store?

Because the product is a dependency-free binary, and every extra process erodes the only reason to choose it. What makes SQLite viable is the precomputed aggregates written in the ingest transaction: no screen scans payloads.

Go deeper

DuckDB or ClickHouse are the right choice if you look only at the query profile. DuckDB is deferred, not discarded: the storage port allows adding a columnar adapter without touching the domain (ADR 01, ADR 04, ADR 10).

Measurement sets the priorities: with a realistic 14.6 KB event, decoding the JSON costs more than the whole SQLite transaction (421.9 µs vs 790.0 µs for four more statements than there were). The single writer turns concurrency into a queue: 8 clients give 667 to 1,090 ev/s against 705 to 747 with one.

The ADR described an ingest batching that was never implemented. The benchmark measured what batching would be worth, ~1.5x, in exchange for a window where the SDK has the event acknowledged and it is not yet durable. The text was removed instead of implementing it.

WHY

Why SQLite in pure Go, without CGO?

Because it installs with curl | sh on amd64 and arm64. With CGO, cross-compilation requires C toolchains per architecture and the binary stops being static. It is slower in microbenchmarks and the design absorbs it: reads hit aggregates, not millions of rows.

Go deeper

A C library in the process is also unmanaged memory surface in a service whose ingest endpoint is public. modernc.org/sqlite with CGO_ENABLED=0; the benchmark gate in CI is the judge. Adapters are tested against real SQLite, never against database mocks (ADR 09).

WHY

Every subsystem is opt-in and costs zero when off

A toggle that keeps consuming resources is a config checkbox. The uptime, crons, status page and digest jobs do not start: they are not a loop that checks a flag and sleeps, they do not exist until their subsystem has work. A fresh install starts with error tracking and one background job only.

Go deeper

The scheduler offers each job together with the question of whether its subsystem has work, and only the ones that answer yes exist; the question is asked again on config change, without a restart (ADR 05, ADR 14). For ingest, where the SDK initiates the work, the protocol itself is used: a disabled category returns 429 with X-Sentry-Rate-Limits and the SDK stops sending it.

The cost is marginally ~zero, not literally zero: the SDK honours the limit for the stated window and retries when it expires. Overselling it as "absolute zero" hands over the refutation of the central claim.

WHY

AGPL-3.0 for the product, Apache-2.0 for the engine

Using it commits you to nothing: the AGPL applies only if a modified version is distributed or served over the network. internal/engine imports nothing from the repository and will become a separate library under Apache-2.0, because it is code meant to be reused.

Go deeper

That boundary, together with internal/domain (stdlib only) and five more pure packages, is verified by a test in internal/arch on every go test and by depguard in lint. The duplication is intentional: the test runs even if you do not have golangci-lint installed.

trapline
Sentry self-hosted

Provision

1 core, 512 MB of RAM

4 cores, 16 GB of RAM, 20 GB of disk

Deploy

one static binary

~40 containers: Kafka, ZooKeeper, ClickHouse, PostgreSQL, Redis, Snuba, Relay…

Upgrade

replace the binary

install.sh with migrations across several services

Backup

copy one file (VACUUM INTO, while running)

pg_dump + ClickHouse + volumes

Will never do

session replay, profiling, metrics, SSO, multi-node: a contract, not a backlog

has it all, at the scale of thousands of organizations

16 GB is its published minimum, not a measurement of our own; it has changed more than once. They do not do the same thing: that stack has an event search engine that does not exist here. Figures as of September 2026.

05 · What I would do differently

Five decisions I would take the other way today

I did

Publishing the footprint by measuring an idle process

Today I would

Measure the peak under load and under attack from the first gate. The three wrong figures were true precisely at the only moment nobody cares about.

I did

Quoting "54 containers" from memory in ADR 001

Today I would

Count them before writing them down. There were ~40. The ADR is not edited: the correction goes underneath, dated.

I did

Describing in an ADR a batching that did not exist

Today I would

Write only what is implemented. The measurement gave ~1.5x in exchange for a non-durability window; the text was deleted.

I did

Global CSRF middleware, on ingest too

Today I would

A suite with real SDKs from the first endpoint. The API tests passed with a middleware that prevented delivering an event.

I did

Grouping with the release directory in the fingerprint

Today I would

A golden grouping corpus from the start. Every deploy created new issues and buried the regression signal.

06 · If you take away three things

Recap

It fits on a 512 MB VPS and the figure is measured again every night. If it goes up, the build turns red.

You do not switch SDKs: compatibility is tested with six official SDKs and with recorded sentry-cli traffic, not with hand-written envelopes.

It is designed so an agent can close the bug: one document instead of five calls, embedded MCP with the same credential, and a person decides what reaches production.

Links

View on GitHub Release v0.1.0