NAME

Punk::Plugin::Observe - mount Punk::Observe in a Punk application

SYNOPSIS

plugin 'Observe' => {
    prefix => '/observe',
    guard  => 'Web::Auth#observe_admin',       # required
    store  => '/var/lib/punk-observe',
    ingest => { prefix => '/v1', keys => '/etc/punk-observe/keys' },
    limits => { series => 1_000_000, rate_records => 50_000 },
};

DESCRIPTION

Mounts the observability UI under a guarded scope and, optionally, the OTLP ingest endpoint beside it.

The guard is not optional

Registration croaks without one.

An unguarded mount is every log line the application has ever written, served to anybody who finds the prefix. A loud failure at boot beats a silent hole that nobody notices, so the failure is at registration rather than at the first request.

Setting PUNK_OBSERVE_INSECURE is the deliberate escape, named for this distribution so that meaning it is possible and doing it by accident is not.

The UI is registered as an under scope rather than a mount. Under a scope the guard covers every path beneath it including ones added later; a mount with a guard on each route is the same thing right up until somebody adds a route and forgets, and the failure mode of forgetting is an unauthenticated page.

Ingest is a separate scope, in both directions

The ingest prefix sits outside the UI scope on purpose.

It is authenticated by key and not by the UI guard, because an exporter has no session. It is CSRF-exempt, because an exporter has no form token.

Both directions are bugs. A UI route that became CSRF-exempt by sharing this scope would be a security hole; an ingest route that required a form token would return 403 to every exporter in the world with a message about forms.

An ingest key cannot read the UI. There is no option to widen it: a key that could do both leaks a whole installation the first time it is baked into a container image, which is where ingest keys go.

Options

prefix

Where the UI mounts. Defaults to /observe.

guard

Required. A coderef, or 'Controller#action'.

store

The store root. Every path this distribution builds is rooted here.

tenant

A constant tenant id, or a coderef resolving one. Defaults to a constant, so the self-hosted shape is the whole shape and the hosted one is a callback through the same code. Whatever a resolver returns is validated before a byte of it reaches a path - a resolver is host code, and host code that returns ../other is a bug to catch rather than a value to trust.

A tenant id is never taken from anything a client sends.

ingest

{ prefix => '/v1', keys => $path }. Omit it and no ingest endpoint is registered at all.

limits

rate_records and rate_bytes per second, series for cardinality, storage in bytes, and attributes for the indexed-attribute allowlist.

The rate limit is off unless configured: a default rate limit on a self-hosted box is a surprise in the wrong direction. The cardinality limit has a default, because a store with no cardinality limit is a store waiting for one bad deploy.

rate_records and rate_bytes truncate the batch before it reaches the log and answer with an OTLP partial success naming what was dropped. That ordering is the contract: a partial success means "I did not keep these", so an exporter resending what it was told was rejected must not find those records already stored.

The counters behind all of this live in a small shared page mapped at registration, which is before the fork. One mapped afterwards would be private per worker, and the symptom is not a crash - it is a rate limit N times what was configured and a status page showing whichever shard of the traffic answered the request. Where the platform cannot share one, the status page says so rather than presenting a fraction as a total.

limits and what the status page can show

Two record counts appear on the status page and they answer different questions. accepted is what arrived, counted at ingest; records is what the store still holds. Retention makes the second smaller over time and a limit makes it smaller immediately, so a gap between them is not an error.

The store cannot report the first. Refused data never reaches it, so a receiver throwing every batch away and a receiver being sent nothing have identical stored totals.

alerts

A reader for the alert rules, as a hashref or a coderef returning one. Rules are configuration with an owner, a review and a history, so they live in the application's database rather than in a telemetry store that retention deletes from. sqitch/ ships the schema.

alerts => sub {
    # ONE argument, and it is the request - not ($id, $req). The
    # dashboards seam below takes its slug first; this one does not, and
    # reading it that way binds the request hashref to $id.
    my ($req) = @_;             # $req->{id} on /observe/alerts/:id
    return {
        rules    => [ { id, name, series, state, value, held } ],
        silences => [ { pattern, until, by, reason } ],
        events   => [ { series, to, at } ],     # optional
        can_edit => 1,
        to       => $now_ns,                    # optional
    };
};

state is one of ok, pending, firing, stale or error, per series rather than per rule - one state for a whole rule is the bug that makes an alert resolve because a different service recovered.

held and value are numbers, and the screen formats them. held is how long the series has been in its current state, in nanoseconds, rendered as a duration; value is the number the rule last compared, rendered with %.4g and no unit. A pre-formatted string does not survive either: "2m30s" is read as the number 2 and drawn as two nanoseconds. A latency is therefore worth converting to milliseconds before it is handed over, since nanoseconds under %.4g come out as 2.911e+09.

events is the state history, and the screen draws a timeline from it. Each entry is a transition: the series, the state it moved to, and the instant at which it did, as nanoseconds. That is a row of alert_events in the shipped schema. Supply it and the timeline appears; omit it and the screen is the table alone.

The timeline is drawn only from recorded transitions. One inferred from current state would be a straight line claiming the present has always been the case - which is exactly the question a reader opens it to answer, so answering it wrongly is worse than not answering it.

to is where the timeline's right edge sits, defaulting to now. The last band runs to it, because a state nobody has left is still in force.

dashboards

The same shape for dashboards: a hashref or a coderef taking a slug, returning title, cols, panels and a list of the others. A panel is an OQL string with a title, validated at save time by the parser that will run it.

What each limit does at its cap

They fail three different ways, and treating them as one limit is the mistake:

Ingest rate

Returns an OTLP partial success naming the rejected count, never a bare 429. A 429 makes the exporter re-send the whole batch, forever, at the moment the server is already under pressure - the limit becomes an amplifier.

The limiter covers the ingest prefix and nothing else. Rate-limiting a health endpoint takes the box out of a load balancer under exactly the load the limiter exists for.

Cardinality

The new series is dropped, counted, and surfaced. An existing series is never evicted to admit a new one: that converts a cardinality problem into data loss on the exact series somebody has open in a dashboard.

The counter lives in an arena mapped before the fork, so the limit is per pool. One mapped afterwards is private per worker, and the symptom is a limit silently N times what was configured.

Storage bytes

The retention job shortens retention. Writes are never refused. A store over its byte budget should lose old data; it must not lose the incident happening now.

The allowlist matters more than any of the numbers

Logs and spans carry unbounded attributes, and only the configured set becomes an index dimension. The rest stay in the record and are reachable by a residual filter, so nothing is lost - it is just slower to find.

Without it, one service putting a request id in a resource attribute takes the store down. The overflow counter names the attribute, because the person who hits this first is a self-hoster with no support contract and no dashboard telling them which one did it.

SEE ALSO

Punk::Observe, Punk::Observe::Tenant, Punk::Observe::Key, Punk::Observe::Limit