Networking: a Capability-Mediated Userspace Service

Status: BUILT and shipping. net-stack and nic-driver are services on all four shipping ports - x86_64, Pi 2 (LAN9514), Pi 4 (GENET) and VisionFive 2 (dwmac); DHCP, ARP, ICMP, DNS and the shell's net/ping all work on real hardware. This header said "design, being built" until 2026-08-25. v2 (networking is out of v1 scope - §23.4). Non-normative until built and pinned by an identity test, at which point the relevant decisions are amended into CLAUDE.md. This doc records the architecture and the phased plan, mirroring docs/persistence.md.

Note (2026-10-09): TCP is built too. services/net-stack/src/tcp.rs implements it (design: docs/tcp-design.md; command surface: utilities/48_tcp.md). The sentences below that call TCP "next" or "committed" were written before it landed and are left as the plan they recorded.

Decision (2026-07-04, owner): TCP/IPv4 is committed from the start, not far-future. The build order is unchanged - the layers still stack NIC -> ARP -> IPv4 -> ICMP (ping) -> transport - but TCP is now a first-class goal of this effort, not a deferred maybe. The motivation is Commandment VIII made literal: TCP is "wait on truth, including the truth of failure" - an ACK is a truth, a peer's RST is a truth, and a retransmission timeout that fires is failure surfaced, not a guess. (TCP's own protocol timers - RTO, retransmission, TIME_WAIT - are protocol-mandated timing, the same exempt category as USB/AHCI hardware timing, not the correctness-by-timing that VIII forbids.) UDP (Phase 3) stays in as TCP's stateless stepping-stone - it proves the socket-cap model with no state machine - and TCP (Phase 5, brought forward) builds the state machine, retransmission, and ordered teardown on that same socket-cap foundation. Everything below the transport layer (Phases 0-2) is shared and identical either way, so work starts there regardless.

1. Thesis

Networking is a userspace service, and the kernel gains nothing. A socket is a capability - exactly as a file is a capability (§7.10, P2). The classic "this must live in the kernel" subsystem is, here, entirely additive userspace: a NIC driver service plus a network stack service, both built from capability mechanisms that already exist. No new kernel code.

That is not a happy accident - it is the constitution paying off. The kernel anti-scope (§4.4) forbade a network stack inside the kernel; the capability model (§7) made resources delegable to services; the delegated-resource-cap work (P2 / §7.10) already lets the kernel mint, route, and revoke caps for a resource whose meaning is owned by a service (it did this for files without learning what a file is). A network endpoint is just another such resource. The kernel will route opaque socket caps exactly as it routes opaque file caps, never learning what a socket is - so §4.4 holds verbatim.

This is the headline property and the reason to be confident in the design: the hardest "kernel" subsystem in the book is, in GodspeedOS, a couple of ordinary restartable services.

2. Why a service, not the kernel

• §4.4 (kernel anti-scope): "The kernel does not contain ... network stack." Settled law.
• §3.1 (no ambient authority): a network stack in the kernel would be ambient reach to the wire for anything that could call it. As a service holding the NIC caps, the network is reached only by holding a socket capability the stack minted - authority is explicit, not ambient.
• §3.10 (30-minute whiteboard rule): a kernel TCP/IP stack is tens of thousands of lines no single engineer fully holds. Out of the kernel, the stack is just another service we can keep small.
• §6 (TCB): in the kernel, a stack bug is a kernel bug. As a restartable service (like block-driver / xhci), a stack bug kills and restarts one service - it does not own the box.

The kernel's only involvement is what it already provides to every driver: MMIO/IRQ/DMA capabilities (§12.3), IOMMU confinement (§6.4, H1), and the delegated-resource-cap routing (§7.10). All of it exists.

3. Why our own minimal stack (not a port)

Same reasoning as "our own filesystem, not ext4/btrfs" (docs/persistence.md; §3.3 identity over authority, §3.10 whiteboard rule):

• A ported POSIX/BSD stack (lwIP, or even smoltcp's socket layer) brings fd-and-ambient semantics that fight the capability model - file descriptors, blocking connect, an implicit routing table, ambient port binding. Retrofitting capability mediation onto that is more work than writing the thin layer we want, and it imports a vocabulary (§B.4) we deliberately don't have.
• It is too big to fully understand. The value here is a stack a contributor can explain on a whiteboard, where every byte on the wire traces to a capability operation.

We write a minimal, capability-native stack: ARP + IPv4 + ICMP + UDP (all built + HW-proven on the T630, §11); TCP is the committed next phase, not deferred (the owner's decision at the head of this doc). smoltcp (Rust, no_std) is a fine reference for wire formats and the TCP state machine - we may borrow its parsing shape - but the socket layer is ours (caps, not fds) and the scope is "understandable," not "feature-complete."

4. Architecture

  ┌─────────────────────────────────────────────┐
  │  Services (hold socket capabilities)         │   e.g. a future `timeserver`, `httpd`, the shell
  ├─────────────────────────────────────────────┤
  │  net-stack   (service)                       │   ARP / IPv4 / ICMP / UDP (built)  [TCP next]
  │   - owns the host IP + ports (a resource)    │   mints + revokes SOCKET caps (§7.10)
  │   - routes datagrams <-> sockets             │
  ├─────────────────────────────────────────────┤
  │  nic-driver  (service, NOT IOMMU-confined)   │   per-board: e1000, RTL8168, GENET, dwmac;
  │                                              │   + a radio bridge to wifi-driver or wifi-usb (cable wins)
  │   - raw Ethernet frames in/out via DMA rings │   MMIO + DMA + IRQ caps (§12.3, §6.4)
  ├─────────────────────────────────────────────┤
  │  Kernel  (routes opaque socket caps only)    │   NO networking - delegated-resource-cap routing
  ├─────────────────────────────────────────────┤
  │  NIC hardware                                │
  └─────────────────────────────────────────────┘

Every arrow is IPC + capability-mediated:

• A service ↔ net-stack: the service holds a socket cap; send/recv go through it.
• net-stack ↔ nic-driver: a NIC-agnostic frame interface - "transmit this raw Ethernet frame", "here is a received frame". net-stack is therefore independent of the NIC model; swapping an e1000 driver for the T630's chipset is invisible above the frame interface.
• nic-driver ↔ hardware: MMIO descriptor rings + DMA buffers + a receive IRQ (§12.2).

Two services, one clean seam (raw frames) between them, and the kernel underneath touching none of it.

5. The NIC driver

A userspace driver service, structurally identical to block-driver (AHCI) and xhci/ehci (USB):

• Capabilities (§12.3): hw_mmio (the NIC's BARs), hw_interrupt (the receive IRQ line), and a DMA arena for the TX/RX descriptor rings + packet buffers.
• NOT IOMMU-confined (§6.4). This said the opposite, and the correction matters because it is a trust claim. nic-driver is spawned hwclass::pci(0x02_00_00, BAR_AUTO, false) - the third argument is confine, and it is false (services/supervisor/src/main.rs, the nic-driver spawn row; on the boards whose NIC is not on PCI it is hwclass::NIC, with no confinement either). xhci and audio-driver are the only in-tree drivers spawned confined (an installed driver package's PCI device is asked for confinement too, CLAUDE.md 14.1); ehci and block-driver are deliberately left in passthrough because they keep a stale firmware DMA pointer that confinement would fault (kernel/src/task/mod.rs, the confine flag on HwClass::Pci). So the NIC driver's DMA is unconfined: a buggy one is bounded by the arena it was granted, but a compromised one can point the controller anywhere in RAM, which is kernel-equivalent reach by §6.4's own rule. It is restartable, and it is in the TCB on every machine today. All unsafe lives behind the SDK's audited Mmio/Dma wrappers (§18.1) - the driver itself is unsafe-free.
• Model-specific, like AHCI. The dev driver is e1000 (Intel 82540EM) - exhaustively documented and emulated by QEMU (-device e1000). (virtio-net is simpler but paravirtual-only - it cannot run on bare metal, so it is not the dev target.) Phase 0 identified the T630 as a Realtek RTL8111/8168 (10ec:8168), NOT Intel - and QEMU has no RTL8168 model (it emulates the old rtl8139, e1000, virtio-net, but not the 8168). So the two drivers are genuinely separate: e1000 in QEMU carries all the stack development, and a Realtek RTL8168 driver is Phase 4, tested only on the T630 bench. The NIC-agnostic frame interface is exactly what makes this clean - the stack never knows the difference. This is the "agnostic stack, model-specific drivers" split being load-bearing, not theoretical.
• Raw frames only. The driver knows Ethernet framing and the ring/DMA mechanics; it knows nothing about IP. TX: a service hands it a frame, it enqueues a TX descriptor, rings the doorbell. RX: the receive IRQ (routed to the driver, §12.2) wakes it; it copies the frame out of the DMA buffer and hands it up. The frame interface is the entire contract.

Phase 0 - identify the hardware

The kernel already enumerates PCI (arch/x86_64/pci.rs, used by block-driver), so the first concrete step was a probe that enumerates PCI and prints each NIC's vendor:device + BAR + IRQ at boot.

Done + hardware-confirmed (2026-06-24). pci::init now logs network controllers (class 0x02) alongside USB/AHCI. Flashed on the T630, it printed: pci: NIC (network ctrl, subclass 0x00) at 01:00.0 vendor=0x10ec device=0x8168 MMIO=0xe000 IRQ=5 - a Realtek RTL8111/8168 Gigabit NIC. (QEMU's e1000 prints vendor=0x8086 device=0x100e, confirming the detection path in the shell-test harness.) One follow-up for Phase 1: on the RTL8168 BAR0 is an I/O BAR (bit 0 set) - the printed MMIO=0xe000 is that I/O base; the real register MMIO is BAR2. Phase 0 should print all BARs and flag I/O-vs-memory so the driver grabs the right one.

Costs almost nothing and de-risked the hardware target: we now know Phase 4 writes a Realtek driver.

6. The capability model - a socket IS a capability (the centerpiece)

This is the part that makes networking cheap here, because the mechanism already shipped (P2).

net-stack owns the network resource: the host IP and the port space. There is no ambient network - a service that holds no socket cap cannot put a byte on the wire (§3.1).

• Open (bind a UDP port, or connect to a peer): the service asks net-stack, which resource_mints a fresh socket capability (a delegated resource cap, §7.10) with chosen rights (SEND/RECV), records SocketId -> (proto, local, remote), and hands a narrowed copy to the service - the exact move fs makes on Open.
• Send = invoke the cap. The holder sends on the socket cap; the kernel validates it (generation
  • required right) and routes the message to net-stack's endpoint badged with the opaque SocketId (the LastRecvBadge mechanism, §22 Test 14). net-stack builds the datagram and hands the frame to nic-driver. The kernel never learns this is a packet - it routes an opaque resource.
• Recv. Incoming datagrams are matched to a socket by net-stack and delivered to the holder's recv endpoint. No socket cap, no delivery.
• Close = revoke. net-stack resource_revokes the socket (generation bump, §7.5); every outstanding cap to it goes stale - the next use returns CapRevoked. Identical to deleting a file.

So a socket cap has every §7.3 property a file cap has - unforgeable, non-escalating (rights narrow on transfer), scoped to one SocketId, revocable, generationed. "A socket is a capability" is literally true, with zero new kernel mechanism. The kernel's resource_mint/Invoke/revoke are resource-agnostic; net-stack is simply a second RESOURCE_MINT holder alongside fs.

The security upside (Appendix D.4, for the wire)

curl evil.com cannot exist unless something explicitly grants the requester a socket cap to that destination. There is no ambient socket layer to reach. The shell (or a policy service) becomes the deliberate place where outbound-network authority is decided - the same property D.4 describes for rm -rf /, now for exfiltration. A compromised service with no socket cap is network-mute by construction.

7. Addressing and identity

• The host IP. net-stack owns it. Static config first (an IP/mask/gateway baked into the stack's contract or a tiny config); DHCP is built (itself just a UDP client of net-stack - a real lease + gateway is HW-proven on the T630, §11 Phase 3), with static config still the fallback.
• Ports are resources net-stack allocates. A UDP socket cap is authority over a (proto, local port) (and optionally a fixed remote); a future TCP socket cap is authority over one connection.
• Identity over location (invariant 11). A socket cap is stable identity; the NIC it egresses, the route it takes, even the host's L2 address are location. This is the same separation SMP made between service identity and core, and it is the on-ramp to cluster mode (Appendix C.4): EndpointId -> (NodeId, CoreId) and a remote socket are the same generalization - a capability whose target may live on another machine. We are not building that now, but the socket-as-capability model does not have to be redesigned to get there.

8. The stack - layered, phased

Each layer is small and testable; the milestone for each is a concrete wire event.

• ARP - resolve IP <-> MAC on the local segment (a tiny cache + request/reply). Needed before any IP frame can be addressed on the LAN. The shell surfaces it as net arp <ip> (resolve one host) and net scan (ARP-sweep the local /24 for live hosts).
• IPv4 - parse/emit headers, checksum, fragmentation we refuse loudly rather than implement at first (datagrams over MTU are an error, §26.7), a single static default gateway (no routing table). IPv6 is far-future.
• ICMP - echo request/reply. ping is the first end-to-end milestone - the networking analogue of v1's ping/pong: proof the whole NIC -> ARP -> IP -> ICMP -> back path works on real wire.
• UDP - stateless datagrams. The first socket-cap milestone: a service opens a UDP socket cap, sends a datagram, recvs a reply. Stateless, so it survives a net-stack restart trivially.
• TCP - the committed next phase (§11 Phase 5), not far-future. The state machine, retransmission, windows, and ordered teardown are the bulk of a real stack. Stateful, so a net-stack restart drops live connections (see §9). It builds on the now-solid UDP + socket-cap foundation.

9. TCB posture and restartability

• nic-driver - restartable, but NOT confined and therefore in the TCB (§6.4). Restartability is real and hardware-proven: its death reclaims its DMA resources, the supervisor respawns it, it re-inits the NIC and re-exposes the frame interface, and net-stack reacquires it by name and retries (§14.3). What does not follow is the TCB drop - that needs confinement, and it is spawned confine=false. Restartable bounds the ACCIDENT; only the IOMMU bounds the COMPROMISE.
• net-stack - a restartable service.
  • UDP is stateless: a restart loses nothing structural. Sockets are caps; holders see EndpointDead/CapRevoked, reacquire net-stack by name, and re-bind - the §14.3 client-recovery pattern. The host IP is config, re-read on mount.
  • TCP is stateful: a net-stack restart drops live connections - the peer sees a reset, the holder reconnects. We deliberately do not build a "TCP journal" (the way fs got one for its metadata): TCP connection state is ephemeral by the protocol's own design (a reset + reconnect is a defined, normal event), so reconnection IS the recovery. This is §14.3 ("cascading failure is the client's responsibility") applied to the wire, not a robustness gap.
• Loud failure (§26.7): a dead NIC or stack surfaces as EndpointDead/CapRevoked, never a silent black hole. A datagram that cannot be sent fails loudly to the caller.

10. Failure semantics

No silent fallbacks at any boundary. TCP's own ack/retransmit (Phase 5, next) is the protocol providing reliability on top of this best-effort substrate - not the kernel, not magic.

11. Phased plan (mirrors persistence: ahci -> gsfs -> file-cap)

Each phase gets a design beat in this doc plus QEMU + (where it applies) T630 verification, exactly like the AHCI/GSFS/file-cap ladder.

12. Constitution alignment

When a phase ships and an identity test pins it, the load-bearing decision is amended into CLAUDE.md (e.g. a "socket is a capability" line beside the file-cap one), the way persistence and naming were.

13. Non-goals (scope discipline, §26.2)

• No POSIX/BSD sockets API. Capability-native (socket caps), not file descriptors. We are not source-compatible with anything; that is a feature (§B.3, D.6).
• No full TCP/IP parity. Understandable over complete. No options soup, no congestion-control research, no TLS in the stack (TLS, if ever, is a service on top of a TCP socket cap).
• No routing/NAT/firewall. A single host on one LAN segment with a static default gateway.
• No networking in the kernel - ever (§4.4).
• Pulled into existence by need, not by "a real OS has it" (§26.2) - which is why TCP waits for a real consumer and DHCP waits for a reason.

14. Open questions (for sign-off)

1. The T630's NIC chipset - RESOLVED (2026-06-24): Realtek RTL8111/8168 (10ec:8168). QEMU has no RTL8168 model, so dev is e1000 and the Realtek driver is Phase 4 (HW-only). See §5 Phase 0.
2. Static IP first (baked config), DHCP deferred - RESOLVED: DHCP is built (a UDP client of net-stack, HW-proven on the T630); static config remains the fallback.
3. e1000 as the QEMU dev driver - agreed? Note this is purely the dev NIC: the T630 is Realtek, so e1000 does NOT run on it (vs virtio-net, which can't run on bare metal at all). The HW driver is the Phase-4 RTL8168.
4. IPv4 only to start, IPv6 far-future - agreed?
5. UDP + the cap model is the v2 milestone - RESOLVED: done + HW-proven. TCP was subsequently committed as the next phase (owner's decision, 2026-07-04, at the head of this doc), no longer far-future.
6. One net-stack service owning IP + all protocols, vs splitting (e.g. a separate udp service)? Recommendation: one net-stack for now (whiteboard-simple); split only if a real reason appears (§26.2).

15. Chaos and link resilience (hardware-proven, 2026-07-07..09)

Exercised on the T630 bench (real RTL8111/8168, live LAN). Once the stack ran end to end on hardware - a real DHCP lease, gateway, and ping - the question stopped being "does it work" and became "does it survive the two failures a real NIC actually sees": the cable moving, and the services dying. Both are now chaos-covered, held to the same discipline block-driver/fs got.

Link flap - the cable is location, not identity (invariant 11). A NIC's carrier goes up and down independently of anything the OS does (someone unplugs the cable, the switch reboots). The stack treats a link transition as a first-class, observable event, never a silent error (§26.7):

• Self-configure on link-up. When the driver observes carrier (the RTL8168 PHY reports link a few seconds after cable-in), net-stack runs its configuration dance automatically - ARP announce, then DHCP DISCOVER/REQUEST to a lease. No command, no reboot; plugging in the cable is the trigger.
• ping rides an unplug/replug. A running ping does not die when the cable is pulled - it reports the link down and resumes when the cable returns and the lease re-acquires, with an instant q abort throughout (the utility never blocks un-abortably on a dead link - Commandment VIII, and the "[q] quit" utility convention).
• net reflects the live link. net reads carrier + lease state at call time, so it never shows a stale "up" for a cable that is out.
• net renew recovers in place. A link that came up mis-configured, or a lease that expired, is repaired by re-running the configuration dance inside the running net-stack - no service restart, no reboot. Recovery is a re-configuration, not a re-spawn.

Multi-target max-carnage. The chaos utility (a userspace program) storms the networking services exactly as it storms storage. chaos max-carnage nic-driver,net-stack N takes a comma-separated target list and kills every named service each round, for N rounds (rounds uncapped - a count is not a resource, §6.2/§26.6). Killing the NIC driver and the stack together, repeatedly, exercises the full recovery order: the supervisor respawns nic-driver first, then net-stack reacquires it by name (§14.3) and re-configures on the freshly-initialised link. The random-target variant picks its victims from the live set, so no single kill-ordering is special-cased. Result on hardware: the link comes back and ping resumes after every round, with no kernel panic - "not even a blip." This is the networking half of the same restartability story the storage stack tells (docs/persistence.md §6.16, docs/naming-design.md §8 risk #11): the system reconverges from any perturbation.

16. The clock is not the network's (2026-10-01)

Status: fixed in code; passed the x86 shell suite 215/0 and a Pi 4 QEMU boot. NOT yet run on Pi 4 hardware - the card carries the old net-stack. (Run on Pi 4 hardware 2026-10-02 at 5dd1f1b8, the boot backlog/fixed/66 records: ping over the radio at one echo a second, 19 of 19.)

The coupling. net-stack fetched the wall clock (SNTP: a DNS resolve, then up to three send-and-drain tries, up to about fifteen seconds when the server or resolver was silent) INSIDE its single serve loop, and started it three ways while the clock was unset: at the end of the dance that configures the network, on the time service's nudge every twenty seconds, and on ordinary client requests - status, DNS, ping and ARP each could start one. For the whole exchange the loop served nobody. Boot 2026-10-01 11:17-11:18 on the Pi 4 shows the cost: ping echo 2 was answered after 7092 ms, while that same echo's wire round trip was 39 ms. date sync appeared to cure it only because a success latched the clock and quietened the nudges.

The decoupling.

• No client request starts clock work. The per-request retry is gone.
• Configuring the network does not fetch the clock. The dance ends when the network is configured.
• time pursues the clock, as it owns it: its nudge (op 11) starts a BACKGROUND query. net-stack sends it and returns to the loop; the poll step, which already drains frames every POLL_MS, records the answer; the loop hands it to time, or after a deadline resends or gives up - and says so.
• The nudge is recorded where it used to be lost. It is one capless byte, [11], and it used to be dropped by the dance and taken for the driver's answer by every driver wait (orphaning the real reply). It is now recorded as owed by the sifted driver waits (nic_req, nic_req_ms, nic_drain_ms), the dance's serve pass and the capless arm, and the loop serves it between requests. A nudge that finds no network stays wanted, and is served on the first pass after the network is configured. Still NOT covered: nic_status_req's try_recv clear (used by link_is_up) discards capless messages, and the unsifted waits in dns_resolve (its ARP and its rx-only drain, which would parse the nudge as a frame batch) and the ping drain's ARP acknowledgement eat it. A nudge there is lost, and time re-sends it 20 s later.
• date sync (op 10) stays synchronous, because the operator asked and is waiting - the only synchronous fetch apart from the fallback below. On an unleased, unconfigured stack it can run the whole dance first.
• Where the poll cannot run - net-stack's cycle counter never calibrated - a nudge falls back to the old synchronous fetch, and the log says so: a background query there could never be answered.
• A nudge that finds the cable in but the stack unconfigured runs the dance first, and the dance still blocks the loop (below).
• The background query goes straight to the anycast fallback (162.159.200.123, time.cloudflare.com), with no DNS: a DNS lookup would block the loop. A network that blocks that address but allows pool.ntp.org gets a clock only from date sync, which resolves the pool first. And a reply that lands while another operation is draining frames is not recorded by the job; it is recovered only by the resend.

What this does not fix, recorded rather than smoothed over. Two things still hold the serve loop. The dance itself (DHCP and ARP) blocks it while the network is being configured - backlog/28 and backlog/fixed/29 describe the incremental dance that would fix that. And on the Pi 4 every exchange with nic-driver can cost a second: backlog/fixed/66, then read as a lost wake-up in the kernel's blocked-receiver path. Corrected 2026-10-08: no wake was ever shown lost. What later failed net dns over the radio was nic-driver answering a drain that found no frame with an empty message, which the kernel refused on every port but ARM32 (fixed in e3fcf7ed); the one-second STATUS tax itself had stopped appearing by 2026-10-04, unexplained (backlog/fixed/66). The decoupling should remove the multi-second stalls the clock caused - not yet run on hardware; it does not touch that one-second tax, and a slow ping after this change is that.

16.1 Later the same day: the clock leaves net-stack entirely

The decoupling above still left clock work inside net-stack: the SNTP query and its parsing, the nudge and the four places that had to catch it, a bounded wait on time for every result it pushed (up to two seconds, twice, with every client queued behind it), and date sync running the whole exchange in the loop. The operator's call: "I don't want it in the way of a ping. We have a time service right?"

What net-stack has now: no clock code, and one generic operation. Op 12, a UDP ask: [wait_secs, ip(4), port(2), datagram..]. The datagram goes out at once from a random source port, the asker's reply capability is held in a four-slot table (UdpAsk), and the loop goes on serving. The poll step, which reads every frame anyway, hands the first datagram back from that address and port to that source port straight to the asker - [ASK_OK, payload] - and a deadline (at most 10 s) answers [ASK_TIMEOUT]. A reconfigured network answers every open ask [ASK_ABANDONED]. It knows nothing about what is inside, and it needs no more authority than opening a socket. Op 10 is retired and says so; the nudge (op 11), SET_CLOCK and the send capability to time are gone.

What time has now: the whole NTP client. It builds the 48-byte query with a hardware-random nonce in the transmit timestamp, sends it with op 12 through its existing non-blocking send, and matches the answer in its loop by tag - exactly as it already matched fs replies. It checks what net-stack used to check (server mode, a synchronised server, a real stratum, and its own nonce echoed as the originate timestamp), and Clock::set_network judges plausibility as before. A query with no answer after 8 s is given up on, and a late answer to it is refused by its nonce. OP_SET is retired: it let any client set the clock, and the only caller was net-stack. OP_SYNC (5) is date sync: time sends a query at once and answers immediately, and the shell watches the sync age in OP_NOW drop - bounded at 10 s, q quits.

Why a late answer is safe here when it was not for the stash (docs/net-tags-design.md 7.2): the client can tell. A reply that arrives after time gave up and asked again carries the old nonce, and the new question's check refuses it.

What is lost, said plainly: date sync no longer resolves pool.ntp.org first - every query goes to the anycast address (162.159.200.123), because a DNS lookup would block net-stack's loop. A network that blocks that address and allows the pool gets no clock. And an NTP answer that lands while another operation is draining frames is not delivered by the poll step; time gives up after 8 s and asks again.