events - what the sink RECORDED

Six views over the events service: the 192-event IPC trace ring, the metric table, the log window, and capturing any of it to disk. Everything here is history the sink kept - ask twice and you get the same answer, until the window moves.

For LIVE kernel state - what is stuck right now, who can reach whom - see utilities/46_trace.md.

Status: BUILT and QEMU-verified. The ring, the metric table, the log window, and events persist with its recorder service.

  events ipc                    what happened - the ring, paged and pipeable
  events failures               the same ring, only the failures
  events log [n]                the last n log lines the sink kept (default 20)
  events log boot               the kernel's fixed copy of the boot - never wraps
  events metrics                published samples: owner, metric, value, age
  events status                 ring size, recorded, dropped
  events persist start|stop|status   capture the log to disk

events log boot - the boot, kept by the kernel

events log boot                  the first 32 KiB this machine ever logged, then a line saying how much
events log boot | match xhci     the lines that name xhci
events log boot | count          how many lines the boot was

events log is the SINK's window: the last lines a service sent, which on a busy machine has moved past the boot within minutes. The boot is what a reader most often wants back - what the machine found as it came up, and what each driver said about it - and on a machine with no serial cable it was gone.

So the kernel keeps it. Beside its log ring, which wraps and which nothing reads, it keeps a fixed copy of the first 32 KiB ever logged, kernel lines and service lines alike, in the order they were written. It fills once and then never changes. This view copies it out (InspectKernel query 27, INTROSPECT); it is never drained, so reading it twice gives the same answer, and it changes nothing about how logging works (CLAUDE.md 11.4, 2026-10-08).

A full record says so. A T630 reaches its prompt in about 18 KiB and the record holds the driver bring-up after it; a machine that logs more stops at 32 KiB and the closing line says the record is full and that later lines are on serial. It does not pretend to have them.

Text, not records. The record is 32 KiB and a table holds 4, so it pipes as lines, to match and count. The pipe holds 16 KiB; a full record piped is cut there, and the pipe says it was. The closing sentence goes to the console, never into the pipe.

No timestamps. The kernel keeps bytes, not times. A serial capture's timestamps are the terminal program's, not the machine's.

events persist - keeping a capture when the screen cannot

events persist start /log.txt                  everything, the default budget
events persist start /log.txt 7d               target a week
events persist start /log.txt 64MiB            an exact disk budget
events persist start /log.txt fs 7d sticky     only fs, a week, resumes after a reboot
events persist stop
events persist status

Every budget carries a unit - h/d/w for a duration, KiB/MiB/GiB for a size. Nothing is bare, because a bare number would have to mean megabytes or minutes by convention and 16m cannot be read as either without guessing. That also removes the last heuristic from the parser: a token without a unit is unambiguously a service name. MB is REFUSED rather than treated as MiB - they differ by 4.8%, and quietly giving less than was asked for is a small lie found late.

A duration is a target, never a promise. Converting 7d to bytes needs a fill rate, and that is a prediction about how chatty the machine will be - which the machine decides. So status reports the MEASURED rate and what the budget actually covers at it:

> events persist status
state      covers  kib_day  capacity_kib  lines  rotations  lost  path
recording  ~2d     11400    16384         842    0          0     /log.txt

Ask for a week on a box four times chattier than assumed and this says ~2d, rather than letting you discover it when the log you needed had already rotated away. covers reports the guaranteed FLOOR, not the best case, because a rotation discards a whole piece.

Rotation renames, so /log.txt is always the newest. The previous piece is /log.txt.1. An earlier version alternated between the two names, which meant the current file depended on a rotation count only status could tell you - the file you wanted was a coin toss. Renames are metadata only, so rotating costs no data movement however big a piece is.

Written by recorder, not by events, and that split is the design rather than tidiness. A file write BLOCKS on a reply from fs; events is a single-threaded recv loop, so while it waited it would stop draining its endpoint and drop the very events worth capturing. On a sick disk that is the full deadline, repeatedly - events would go blind exactly when it matters. recorder blocks instead, and nothing depends on recorder, so a stalled disk costs the capture and leaves the volatile window readable.

Spawned on demand, never at boot, and not restarted on death - a respawned recorder would not know its target and would be alive writing nothing while status said "running". The file opens with a header and closes with a footer, so one without a footer says it died.

Bounded, and not by counting. fs allocates a file's whole extent up front, so the size is fixed when the capture starts; two files rotate, so total disk use is twice that, forever. A forgotten capture cannot fill a disk. It ROTATES rather than stopping because stopping keeps the wrong half - a fixed file that stops when full preserves the start of a session and discards the crash at the end.

Reading it back. The file is written as owner: text, one line each, so read /log.txt shows a log. It is NOT records, and that is a deliberate choice rather than an omission: the shell pipe truncates a producer at 16 KiB (CAP_MAX) and every sink clips at 4 KiB, so a multi-megabyte capture cannot survive a pipeline whatever format it is in. read is the tool that will actually consume it, so the file is shaped for read. For records, use the LIVE window - events log | to json - which is small enough to pipe.

It PREPARES before it records, and status says so. The extent has to be made readable before a line can go into it, and on slow storage that takes a while - filling 4 MiB over the Pi 4's USB stick took longer than a caller was willing to wait, and the capture never began. So start allocates and answers AT ONCE, and the filling happens in the recorder's own loop:

> events persist start /log.txt 8MiB
> events persist status
state      covers  kib_day  capacity_kib  ...
preparing  -       0        8192
...a moment later...
recording  ~2d     11400    8192

Blocking a caller on an unbounded amount of device I/O was the defect, not the size - it would have bitten again on any slower medium. Nothing waits on recorder, so a slow disk delays only the capture.

Zero-filled on creation, and it has to be. OP_WRITE_NEW allocates the extent but writes no data blocks, so everything past the last chunk written carried a stored CRC of zero and fs refused the whole file - read returned storage error and the capture was unretrievable. Every file is written through once at creation, which is the pause you see when a capture starts or rotates, and the reason the default size is modest.

sticky records the capture in /system/services/events/sticky, a plain line the shell reads at the next boot and resumes unattended, announcing it. Plain text on purpose: read /system/services/events/sticky shows exactly what will happen, which is the difference between a setting and a surprise. An explicit stop deletes it, so the one command meaning "enough" is not the one that fails to take. Pinned by osdev test sticky, which is the only two-boot test here - a setting that survives a reboot cannot be proved inside one boot.

events log - what was printed, after it has scrolled away

This is not the console's scrollback, and the difference is the point. The console retains lines that scroll off its screen (docs/console-service.md §10), but a SERVICE log never reaches the console at all: ctx.log() writes the kernel ring and serial directly (CLAUDE.md §11.4), which is what keeps logging alive when everything above the kernel is dead. So on a Pi wired to a TV, with no serial attached, events log is the only way to see what a service said - scrolled away or never displayed in the first place.

gsh> events log 4
fs: disk capacity = 0 sectors (0 MiB)
fs: serving file API
nic-driver: dwc2 answered op 0x01 while we asked 0x10 - not our reply (3 mismatched...)
block-driver: capacity 0 - the USB host service replied 1 byte(s), not a capacity
events: 13 line(s) held (4 shown). This is a COPY - serial has the authoritative record, including everything printed before `events` started. Filter with `events log | where owner=<service>`.

This is a COPY, and never the authoritative record. ctx.log() performs its syscall first and unconditionally - the kernel's 16 KiB ring and serial are written whether or not events is alive, reachable, or has ever existed - and only then offers a duplicate for later querying. Re-pointing logs AT the service would make observing a failure depend on a service that can fail, which is CLAUDE.md §15's storage argument one layer up. Adding a copy takes nothing away from the floor.

Three consequences, stated rather than left to be discovered:

• Nothing printed before events started is here. Those lines are in the kernel ring, which no syscall exposes to userspace. Serial has them, and the kernel's fixed copy of the first 32 KiB is events log boot (above).
• The window is 8 KiB and wraps. The footer says how many lines are held and how many are shown; it does not say whether older lines were overwritten, though the sink's reply carries that.
• A killed events loses the scrollback and no log output. That is the design working: the tier beneath it survives anything, including a panic that halts every core.

It prints a screenful by default rather than everything it holds - about 3 KB on a booted machine is more than anyone reads. events log 100 asks for more.

It is a RECORD SOURCE, like every other view here - owner and text:

gsh> events log | where owner=fs | to json
[
  {"owner": "fs", "text": "disk capacity = 0 sectors (0 MiB)"},
  {"owner": "fs", "text": "serving file API"}
]

The owner is a FIELD, which is what makes where owner=dwc2 match a service that writes its lines as dwc2-svc: - a text-prefix match could not, and guessing where a name ends inside a string is exactly what a field removes. The name is stripped from text for the same reason: owner already holds it, and stating one fact twice is what fields exist to stop.

Printing is a rendering, not the format. events log shows owner: text because a log should read like a log and a 240-byte column would be wider than any screen; piped, it is rows. Same data, two renderings - what events ipc already does.

This is a rule rather than a convenience, and it was ratified after being broken: the log was first built as text, and filtering it by service then needed a bespoke argument hand-rolled in the shell, duplicating where and getting it wrong on the first run. docs/observability.md §9a.

events metrics - what services are counting

The ring answers "what just happened". events metrics answers "how much, so far", and it reads the other half of the same service - the metric table in events.

gsh> events metrics
owner   metric           value  age_s
fs      requests         32     1
fs      blk.outages      0      1
events  ring.recorded    139    0
events  ring.dropped     0      0
events  metrics.held     6      0
events  metrics.refused  0      0

Why age_s is not decoration. events keeps a sample after its publisher dies - which is the point, because the final value of a service that crashed is the one useful thing left to learn from it. The cost of that is a number frozen at the moment of death being indistinguishable from a number being maintained right now. The age is what tells them apart, and an instrument that cannot is worse than none.

A publisher chooses its own interval, and should not publish on every operation: fs publishes every 32 requests, because a metric costing an IPC send per request would double the traffic it exists to measure. So a value can lag by up to one interval on a quiet machine, and the age shows the lag rather than implying there is none.

msgs.received comes free, from the SDK. Every service that holds the cap publishes it without a line of its own code, because it is counted in the SDK's three receive paths - the same place trace emission lives, and for the same reason: it is the identical counter in every service, and ten hand-placed copies is ten chances to place it wrong.

It publishes on the FIRST message as well as every 64th, which matters more than it sounds. Publishing only on the interval left any service under 64 messages with no row at all, and no row is indistinguishable from dead - which is the one question this metric exists to answer. Paired with age_s it separates the two faults nothing else in the system separates: a block-driver still receiving and failing, versus one whose count last moved forty seconds ago.

There is a THIRD trigger, for the same reason: the first message after the sink itself comes back. events holds its table in memory, so a restart starts it empty with no knowledge of any service, and waiting out the interval leaves that service unrepresented for up to 63 more messages. On one that only receives when someone else does I/O - block-driver waits on fs - that is however long the machine stays quiet, and it read as dead the whole time. Republishing on reacquire is the metric half of "reacquire and re-establish" (CLAUDE.md 14.3).

Attribution depends on ctx.trace_as, and silently failed without it. A service that never declares its name publishes under a BLANK owner - and since the key is (owner, metric), every undeclared service collides into one row with its counters interleaving. That is exactly how it was found: a single msgs.received 1920 belonging to nobody, which turned out to be console plus nine others. Every internal service now declares itself, an undeclared one renders as ? rather than blank, and the view warns underneath when a ? row exists.

One name is too long and is reported rather than hidden: PEER_LEN is 12 bytes (sized for block-driver) and hw-enumerator is 13, so it reads hw-enumerato and the SDK says so once at start-up. Widening the field was the alternative and was rejected on cost - the dump is bounded by one 4 KiB message, so a 16-byte field drops this view from 110 events per screen to 95, and trading real scrollback for four characters of a name is the wrong way round.

It is a record source, so it filters and formats like events ipc:

events metrics | where owner contains fs
events metrics | select metric,value | to json

What is NOT in here, and cannot be: events reporting its own death. Its own rows are published by writing straight into the table it already owns, never by sending itself a message - a send is itself a reportable event, so a self-emit over IPC would feed the ring from the ring. That local write works right up until the process stops, and then nothing in it can say so. The supervisor's death notification and the kernel's unconditional serial write are what report that, and both sit beneath this service rather than inside it (docs/observability.md §9).

trace task <slot> and trace service <name> were folded into trace chain: they printed identical output from identical code while being named after the SUBJECT KIND, which made them look like two things and made trace service fs read oddly beside trace deps fs. The argument disambiguates itself - digits are a slot, anything else is a name.

events ipc and events failures - what happened

seq  sec  caller  peer          op        outcome
9    0    shell   fs            write     REPLY
10   2    shell   net-stack     1         TIMEOUT
6    2    fs      block-driver  capacity  REPLY

One row per completed exchange, oldest first, newest last. events failures is the same ring filtered to the failures.

Outcomes, and what each one tells you to do:

A request still IN FLIGHT is not here: one row is written per exchange, when it ends. The live view of something stuck is trace blocked.

events status - the ring itself

trace: ring 192 events; 2 recorded; 0 DROPPED (oldest overwritten before being read)
trace: the ring lives in the `events` service - the kernel records nothing.

Who is traced at all

Only exchanges made through the SDK's request/reply calls, by services whose contract grants ipc_send = ["events"]. Today that is thirteen services: block-driver, console, control, dwc2, ehci, fs, hw-enumerator, net-stack, nic-driver, recorder, shell, time, xhci. (events writes its own metric rows into its own table, as described above.) It was fs and shell alone when the ring shipped, and the difference is the point of trace_as: a service that does not declare its name publishes under a BLANK owner, and since the key is (owner, name) every undeclared service collides into one row. Tracing is authority: visible in caps <service>, revocable, and absent by default - not a global switch. A service holding no such capability emits nothing, at a cost of one relaxed atomic load.

6. Mechanism B: the event ring, if it is built

Event - fixed 32 bytes, no allocation, nothing variable-length:

Bounded (§26.6): a fixed ring in .bss, sized once. Full = drop the oldest, count the drop, and events status reports the count. A silent drop is exactly the bug just fixed in the x86 input ring; this must not repeat it (invariant 12).

Near-zero when off: one Relaxed load of an AtomicBool at the routing point, branch not taken. This must be measured, not asserted (§20 - perf claims require a benchmark). The IPC fast path is the one place this project pays for cost, so B1/B2 (IPC round-trip latency) must be run with tracing off and compared to baseline. If it is not free, it does not ship.

trace tree <message-id> is the expensive one, and it needs saying plainly: linking shell -> fs to the fs -> block call it caused requires the parent id to propagate - the SDK must carry "the request I am currently handling" and stamp it on outgoing calls. That is a change to every service's request path, and it is the same problem docs/net-tags-design.md already describes for net-stack <-> nic-driver. Proposed: out of v1, and re-read that design first if it is ever picked up, because doing it twice differently would be worse than not doing it.

Without propagation, events ipc still gives a time-ordered log per endpoint, and the requirement's failure/recovery example works, because that one is a time sequence, not a tree:

10:41:02.126 xhci:9   ENDPOINT_DEAD
10:41:02.127 supervisor -> restart xhci
10:41:02.130 xhci:15  READY
10:41:02.131 block-driver:4  req=55 PEER_LOST

xhci:9 vs xhci:15 is the generation, which the ring records for free.

7. The one line I am not certain of: recording op (RESOLVED - and the premise was WRONG)

As built: the uncertainty below was justified and its factual premise was false. Both busy protocols put something else at byte 0: shell -> fs and fs -> block-driver each PREPEND a one-byte correlation tag (added so replies are not matched by arrival order, after fs accepted one block's data as another's). So a trace recording byte 0 showed REQUEST IDS in a column labelled "op" - 183, 177, 212 against block-driver.

The answer was not to rename the column after the byte. It is that the service which speaks the protocol declares where its opcode lives - ctx.trace_op_at("fs", 1), once at startup. The SDK is generic across peers and cannot know; the kernel may not interpret a payload at all; the service can, which is where this design already puts every other piece of protocol knowledge. The shell then renders the opcode by NAME (read, write, capacity), because a number needs a lookup table the reader does not have.

The section below is kept as the reasoning that led there, per 1 (an amendment is ratified history).

op is byte 0 of the message. Every service protocol in this tree happens to put its opcode there (fs, block-driver, the block IPC protocol). Recording it is what makes events ipc readable rather than a wall of endpoint numbers.

The case for: the kernel stores a byte and attaches no meaning. It is exactly as opaque as the ResourceId badge in §7.10, where the kernel routes something whose meaning only the owning service knows.

The case against: "byte 0 is the opcode" is a convention, and the kernel recording it privileges that convention - which is the thin end of understanding a protocol. A service that puts something else in byte 0 gets a misleading trace.

Proposed: record it, name the field first_byte in the kernel and only call it op in the utility, and state in the code comment that the kernel attaches no meaning to it. If that reads as sophistry on review, drop it - events ipc is still useful without it, and the constitution is worth more than a column.

8b. As built (mechanism B) - the ring is a SERVICE, and the kernel gained nothing

Section 6 above designed the ring inside the kernel and asked whether that was worth it. It was not, and the answer arrived from the question "why can't the trace be a service?". It can, and the service version is strictly better - so what shipped is not the design in §6. That section is kept as ratified history: it is the reasoning that had to be worked through to see why it was wrong.

What changed, and why each change is an improvement rather than a compromise:

The peer is a name, and that is the whole reason this belongs out here. The kernel may not know what a message means (§4.4, §26.10), so a kernel ring could only ever have recorded endpoint 116, op 11 and left the reader to resolve it. The emitter called request_with_reply("fs", ...): it already holds the name, and putting the name in the event is what produces the symbolic output the requirement asked for. The constraint the constitution imposes and the feature the requirement wanted point the same way - which is usually the sign the design is right.

(The first attempt was a tracer service of its own. The enforcement layer refused it, correctly: the kernel holds a service_config per service, pinned as debt that may only shrink, so even a userspace ring would have cost ring 0 three lines. events is already in every one of those lists and its whole purpose is diagnostic data, so the ring there costs the kernel exactly zero.)

Layers touched:

Cost when not tracing: one Relaxed load, branch not taken - and nothing at the routing point, so B1/B2 cannot move. A service traces only if its contract granted it ipc_send = ["events"], so tracing is authority: visible in caps <service>, revocable, absent by default (§3.1). §6 asked for a benchmark before shipping a switch on the fast path; there is no switch and no fast-path write, so the thing that needed measuring does not exist.

Bounded and loud (§26.6, invariant 12): fixed ring, fixed events, no heap. Full = overwrite the oldest and count it; events status reports the count. Emission is try_send with the result discarded, so a full sink queue costs the emitter nothing and loses one event - the right trade on an observability path, and the opposite of the one made on a correctness path. An observer must never be able to slow, block or break the thing it observes.

The reader is not recorded. A call to events itself is never traced: events ipc reaches the ring by asking the service that holds it, so tracing those calls would fill the ring with the reader's own questions, two per dump, pushing out the traffic the reader came to see. Every other peer is still recorded and events status still counts every accepted event, so nothing is hidden.

Time is shown relative to the oldest event in the dump. The stored value is an epoch second; the absolute number says nothing on its own, and the gap between events is what a stall looks like.

Verified in QEMU (osdev test shell, 140/0/2 - 10 of them trace):

gsh> events status
trace: ring 192 events; 2 recorded; 0 DROPPED (oldest overwritten before being read)
trace: the ring lives in the `events` service - the kernel records nothing.
gsh> events ipc
seq  t+s  peer       op  event
0    0    net-stack  2   REQUEST
1    0    net-stack  2   REPLY

Real traffic, named, with no kernel involvement of any kind.

What hardware caught that QEMU did not: the ring was empty

The first cut instrumented request_with_reply. On the Wyse, after a selfcheck and two dir commands, events ipc still said no events recorded - while fs was demonstrably answering.

The SDK has eight request/reply variants, each an independent implementation, and the shell talks to fs through _abortable and _deadline - never the plain one. So nothing was ever emitted for the traffic that matters. QEMU had not caught it because the single event it did record came from the one net-stack call site that happens to use the plain variant, and one green row looked like success.

The fix is a wrapper per variant rather than an edit inside each body: eight functions with several early returns inside a wait loop each, and a wrapper cannot miss an exit because the value returned IS the outcome. All eight, not the busy ones - partial instrumentation is worse than none, because events ipc then shows SOME traffic and silently omits the rest with nothing on screen saying which, which is a silent gap in the instrument built to prevent silent gaps (26.4, invariant 12).

Two event kinds fell out of doing it completely, because two of the variants already distinguish outcomes the original four kinds could not express:

• QUEUE_FULL - the peer is ALIVE and its queue is full. Congestion, not absence. Recording that as a lost peer is a bug this project has already paid for once (net-stack reacquiring a capability that was never stale, backing DHCP off 34 s on a healthy link), so the ring keeps them apart.
• ABORTED - the user pressed q. Not a failure of anything, and recorded so a gap in a chain is explained rather than mysterious.

events failures shows QUEUE_FULL (a request that never arrived) and not ABORTED (a change of mind).

What the SUITE caught that neither had: the observer was stealing the shell's core

Instrumenting all eight variants took osdev test files from 222/0 to 213/9 - tab completion, move and find timing out, and Enter keystrokes being lost so commands ran together. Four experiments, in order, because the first three theories were all wrong:

Every trace event WAKES the sink, and the sink was on core 0 with the shell. Two events per fs request preempted the shell twice per round trip; the paths that do many round trips blew their window, and the shell was descheduled often enough to stop draining the console.

Three things changed as a result, and each is a rule worth keeping:

1. The sink does not live on the interactive core. ARM had already moved events off core 0 for a different reason (keeping the serial writer away from the microframe-timed USB driver); x86 now needs it for this one. An unavailable preferred core falls back to round-robin rather than failing the spawn, so a machine with fewer cores still gets its events.
2. One event per exchange, not two. A REQUEST plus an outcome doubled the traffic through the sink's single 16-deep endpoint. Every exchange still produces exactly one event carrying its fate, so nothing is lost but the duplicate. A request still IN FLIGHT is therefore not in the ring - and that is the one question the ring was never the right instrument for: trace blocked reads it from the kernel, live (mechanism A).
3. The sink stamps the time, from a cached clock. epoch_secs_monotonic is a CMOS RTC read on x86 - wait_update_clear can spin ~1 ms before seven port-I/O reads - so per-event it would cap the sink near a thousand events a second. events reads the cycle counter per event (one instruction) and refreshes the seconds only when one has actually elapsed.

The reader also retries a busy sink three times before reporting it, because a full queue is congestion, not absence, and saying "unavailable" about a service that is alive and busy is a lie the same size as any other silent fallback.

seq is the emitter's own counter, not a global one, so a mixed dump interleaves several sequences and looks unsorted. It is not - rows are in ring order, oldest first. The column earns its place by making one service's dropped events visible as a gap in its own numbering.

A row is an EDGE, not a fact: the caller

peer alone answers "who was called" and leaves "by whom" to inference - and that inference only works while exactly two services hold trace authority. So an event carries the CALLER too, and a dump reads as a call graph:

seq  sec  caller  peer          op   outcome
9    0    shell   fs            4    REPLY
10   2    shell   net-stack     1    TIMEOUT
6    2    fs      block-driver  183  REPLY

The caller is self-declared (gs::trace::as_name(&ctx, "fs"), once at startup), and that deserves the argument rather than an apology:

• A service cannot ask what it is called. There is no name in its context page and no query for one, because identity is not ambient (3.1). This is the capability model being consistent, not a gap.
• The kernel does know, unforgeably: every syscall send stamps Message.sender_ep, the sender's primary endpoint. It exists for reply-matching and its comment ends "Kernel-internal: never crosses to userspace ... so no ABI change". Surfacing it would grow the syscall surface for a diagnostic, which is not a trade this project makes.
• The obvious objection - a self-declared name is a claim, not a fact - does not survive contact with what the event already is. A service holding ipc_send = ["events"] can already write any peer and any outcome it likes, because the whole event is its testimony. caller is exactly as trustworthy as the two fields beside it: as trustworthy as the service you granted trace authority to. It opens nothing.

A service that never declares itself reads ?, which is the honest answer rather than a guess.

Reading it: pages, pipes, and a legend

events ipc is a record source, like status or dir - one producer feeding three uses, rather than a printer plus a serialiser that drift apart:

• Console: a grid, with a seven-line legend above it.
• Taller than the screen: it pages, with the shared pager's keys (arrows, space, PgUp/PgDn, Home/End, q). g/G and j/k are gone, and so is b - [b] background claims that letter. The pager was help-shaped - it called help_render_line directly - and is now given a render closure, so the one screenful-at-a-time reader in the system is shared instead of copied.
• Piped: events ipc | to json, | to yaml, | where caller=fs, | where outcome=TIMEOUT, | count, | sort. The legend is console-only; a pipe carries records and nothing else.

A dump shows the newest 64 events - what a record Table holds - while the ring keeps 192. Asking for more only produced "result exceeded the record bound - truncated" on every run, and a bound announced every time is noise that trains you to ignore it. events status remains where the true ring size and drop count live.

The shape of the whole thing

Three questions, three sources, and the kernel is only in one of them.

                    WHAT IS STUCK NOW            WHO CAN REACH WHOM            WHAT HAPPENED
                    trace blocked                trace deps                    events ipc
                    trace chain                  trace endpoint                events failures
                          |                            |                             |
                          v                            v                             v
                  +---------------+           +----------------+          +--------------------+
                  | kernel, LIVE  |           | kernel, LIVE   |          | events, a RING     |
                  | who awaits    |           | capability     |          | of past exchanges  |
                  | which endpoint|           | tables         |          |                    |
                  +---------------+           +----------------+          +--------------------+
                    read-only,                  read-only,                  written by SERVICES,
                    2 queries                   existing syscall            never by the kernel

The kernel records nothing. It answers two questions it already knew the answer to (which endpoint a blocked task awaits, and which endpoint a task owns); everything historical is written by services into a service.

How an event gets into the ring

   fs                                     kernel                    events
   |                                        |                          |
   |-- request_with_reply("block-driver") ->|                          |
   |                                        |-- deliver -> block-driver|
   |<------------- reply ------------------ |                          |
   |                                                                   |
   |   ONE event per exchange, describing how it ENDED:                |
   |   seq, caller="fs", peer="block-driver", op, outcome=REPLY        |
   |-- try_send (fire and forget, never blocks) --------------------->|
   |                                                                   |-- stamp the time
   |                                                                   |-- store in a 192 ring
   |                                                                   |   (full = drop oldest,
   |                                                                   |    and COUNT it)
   |                                                                   |
   shell -- TRACE_OP_DUMP ------------------------------------------->|
   shell <- the newest 64 events ------------------------------------ |

Three properties fall out of that picture, and each is deliberate:

• The kernel is not on the path at all. No ring in ring 0, no retention policy, no control syscall, and nothing added to the IPC fast path.
• The emitter never waits. try_send and the result is discarded: a full sink costs the emitting service nothing and loses one event, which the ring counts and events status reports.
• A service that holds no events send cap emits nothing, at the cost of one relaxed load. Tracing is authority, visible in caps <service> and revocable.

Why the sink is not on the interactive core

   BEFORE                                  AFTER
   core 0: shell  events                   core 0: shell
           ^^^^^  ^^^^^^                   core 2: events
           every event WAKES events,
           preempting the shell            the wake lands on another core
           twice per fs request

   osdev test files: 222/0 -> 213/9        osdev test files: 222/0
   tab completion timing out,              (nothing changed but the core number)
   Enter keystrokes lost

What the same hardware run says about the ring's limits

events failures came back empty after a 100-round chaos max-carnage - 559 kills. That is the design working, and worth stating rather than discovering later: events was itself killed 51 times, and a restarted events starts with an empty ring. Emission is try_send, so events aimed at a dead sink are lost too.

For the question this tool exists to answer - why is this task not progressing, right now - that is correct, and no ring in a restartable service can behave otherwise. For post-mortem of a storm that killed the recorder 51 times, it cannot help, and only persistence would - which the ring deliberately is not (see the events header on why history that survives nothing is the right call).

Still out of scope, unchanged: trace tree <message-id> needs parent-id propagation through every service's request path (§6, and docs/net-tags-design.md describes the same problem for net-stack <-> nic-driver). It is not made easier or harder by this; it is simply not built.