szn

Bug 298 — Client freezes on a full stdout — blocking writeAll stalls input forwarding (mosh backpressure)

298. Client freezes on a full stdout — blocking writeAll stalls input forwarding (mosh backpressure)

File: src/main.zig:495 (runInteractiveClient → writeAll(stdout_fd, data)) Severity: HIGH Status: ✅ FIXED — the interactive client’s stdout is now set O_NONBLOCK; rendered frames are queued into a bounded out_buf and drained on POLL.OUT (mirroring the server’s flushDisplayClient). If the queue exceeds MAX_CLIENT_OUT_BUF the backlog is dropped and, once the pty drains, the client sends a new .redraw request so the server resets that client’s diff baseline (last_cells) and repaints a coherent full screen. Regression tests: queueStdout overflow/drop, flushStdout full-pipe remainder + later drain, and server handleClient .redraw reset.

Follow-up (flow control): the client-side fix only removed the hard block; under sustained overload (a full-screen TUI like opencode redrawing faster than the link carries) the client’s 1 MiB out_buf still overflows and frames get dropped, so the screen still looks frozen. The real fix is server-side backpressure: when a display client can’t keep up the server must throttle the child to link speed instead of piling up frames. Added DisplayClient.behind + RENDER_HIGH_WATERMARK (768 KiB) in appendClientOut — once a client’s out_buf crosses the watermark no further frames are queued (the already-buffered frame is kept, not duplicated), and behind is cleared when the buffer drains (flushDisplayClient). handlePtyEvent skips feedPty while anyDisplayClientBehind(), and pumpPaneInput arms POLL.OUT on panes with queued keystrokes so a throttled, non-blocking Pty.writeInput can drain its input_buf rather than dropping typed keys. Net effect: opencode over a slow mosh link runs at link speed (not frozen) and keystrokes are never lost. Regression tests: src/server/pty.zig writeInput queues remainder under backpressure + flushInput drains it; src/server/server.zig appendClientOut marks display client behind on watermark and skips frames.

The interactive client renders every frame with a blocking writeAll(stdout_fd, data) (src/main.zig). When the downstream terminal applies backpressure — e.g. mosh throttling the pty when the network is slow, which happens readily under a full-screen TUI like opencode that redraws constantly — write(2) blocks forever. The client is single-threaded, so while it is stuck in that write it stops polling both its stdin and the server socket: no keystrokes are forwarded, no frames are read, and the screen freezes. Killing the mosh client and reconnecting drains the pty, the write completes, and the buffered input + queued frames finally flow — which is why “everything typed while frozen appears after reconnect.”

This is the client-side mirror of bug #207: the server’s display sockets were made non-blocking, but the client’s own stdout write was never hardened, so the same class of stall silently moved to the other side of the socket. The server-side non-blocking fix (v0.6.0) is what made the server keep running; the client-side blocking write was the remaining freeze.

Root cause (the freeze that persisted): the server log from a remote mosh -a host -- szn run showed the display-client socket (fd 4) dying with ConnectionResetByPeer while opencode was still initializing, after which the pane kept running for hundreds of lines — i.e. the client process was killed, not merely stalled. The cause is SIGPIPE: a write to a closed/unreadable peer (a mosh link that dropped, or the downstream pty closing mid-redraw) raises SIGPIPE by default, which terminates the process instantly. Making stdout O_NONBLOCK fixed the blocking freeze but does not suppress SIGPIPE — a write to a closed peer still kills the client. The fix is process-wide sigaction(.PIPE, SIG_IGN) in main.zig so a closed peer returns EPIPE (handled gracefully: flushStdout drops the frame and the client keeps polling, resuming when the link returns) instead of killing the client. This protects the server too (a write to a dead display client would otherwise also SIGPIPE). Regression test: src/main.zig write to a closed fd returns EPIPE under the ignored-SIGPIPE disposition.

Final cause (client still died after the SIGPIPE fix): with the ConnectionResetByPeer still appearing at the same point after the SIGPIPE fix, the client was instead exiting on stdin EOF/HUP. mosh -a keeps the remote process (and its pty) alive across transport drops, so a lost stdin is transient, but the client treated a stdin read returning 0 or a non-AGAIN error as a detach and exited — which tears down the mosh child and freezes the session. The fix: the interactive client no longer detaches/exits when its display stdin goes away. It stops polling the dead fd (to avoid a busy-loop) and probes it again periodically; on revival it sends a .redraw so any frames dropped while the link was down are replaced with a coherent full repaint. The client now survives mosh/ssh transport drops and mosh -a resumes it. (Explicit detach — Ctrl-b d — still works, because that path is driven by a .detach message from the server, not by stdin EOF.)

Client death root cause (signal-driven): the ConnectionResetByPeer occurred at the same byte offset across logs even after the SIGPIPE and stdin-EOF fixes, ruling those out (and ruling out framing: length is total bytes, min 5). The killer was a signal on the controlling terminal — SIGHUP (hangup) or SIGTTOU/SIGTTIN (job control, since the client does raw-mode/tcsetattr I/O on the mosh pty). Hardened the client: ignore SIGTTOU/SIGTTIN (tmux practice), handle SIGHUP non-fatally (flag + log, stay attached), and log startup + every exit path. In the next reproduction (szn-3.log) the read from client fd 4 failed line is gone — the death is fixed.

The actual freeze (display not updating, client alive): with the client no longer dying, the freeze is plain frame-dropping under overload. opencode redraws constantly (thousands of CSI dispatch lines); the server forwards every frame to the client, whose stdout→mosh can’t drain on a slow link, so out_buf overflows and frames are dropped. The server-side DisplayClient.behind flow control never engaged because it keys off the client socket being full — and the client reads that socket eagerly, so it never fills. The fix is client-side backpressure: when the client’s stdout queue exceeds CLIENT_OUT_HIGH_WATERMARK it stops reading from the server socket; that fills the server socket, trips DisplayClient.behind, and the server throttles the child (via handlePtyEvent skipping feedPty) to the link’s speed. Frames are then delivered (slowly) instead of dropped, so opencode tracks at mosh’s rate rather than freezing.

Diagnostics gotcha: the interactive client never loads the server config that enables logging, so every client-side diagnostic was invisible in szn.log (the log files were 100% server-side). The server’s log path can also be overridden by the log-file config option, so the client writing to the “default” path is not guaranteed to land in the file the user inspects. The client now writes its diagnostics to a dedicated, well-known file ~/.szn/szn-client.log (log_mod.enableClientLog, forced to 0o600 so the owner can read it) and logs backpressure on/off transitions, stdin/SIGHUP survival, and every exit path. Locally verified: the client logs “interactive client connected”, survives stdin EOF and SIGHUP, and the file has correct permissions.

Current finding (client alive, opencode stalls at startup): with client logs visible, the client is confirmed alive and idle — it connects, then produces no backpressure/exit/signal diagnostics, i.e. it is waiting for frames. Meanwhile the server log (/tmp/szn.log) parses opencode’s terminal-capability queries (?6n DSR, ?p DECRQM, kitty ?u, ?c DA) and then stops (szn-6.log ends at line 51, right after the query burst) — opencode never proceeds to its full-screen render. szn does generate DSR/DA/DECRQM/kitty responses via pty.writeInput. Added server-side diagnostics to distinguish “opencode hung waiting for a response” from “server stopped reading the pane (flow control)”: each query response is logged, and the handlePtyEvent flow-control skip is logged.

Deadlock found (47M flow-control lines / 3 GiB log): a real reproduction produced a 3 GiB /tmp/szn.log containing 47,218,556 flow control: pane read throttled lines while the client log had zero backpressure lines. The client was not congested on its stdout — it was blocked in writeAll(server_fd, …) because server_fd is a blocking socket. When the server was busy (CPU-bound rendering opencode’s constant redraws), it stopped reading the client’s socket promptly, the client’s small writes (keystrokes / .redraw) filled its send buffer, and the client blocked forever in write(2). It then stopped reading the server socket, so the server’s socket filled, DisplayClient.behind set, the server throttled the pane, and the unbounded flow-control warn flooded the log and further starved the server of I/O. Fixes: (1) the client makes server_fd O_NONBLOCK and uses a best-effort writeServer() (drops the remainder on EAGAIN) so it can never block; (2) the flow-control warn is rate-limited to once per second so a stuck client can’t flood the log.

Still frozen with nothing rendered, server keeps parsing: after the deadlock fix the server keeps parsing opencode’s output (log grows, slowly) and the query responses are all delivered, but the client screen never shows anything. The client is connected, not congested, and alive. Added frame-flow counters to both sides (server queued N frames / N bytes and client received N frames / N bytes, every 200) so the next reproduction can attribute the break to server→client, client→mosh, or mosh itself.

ROOT CAUSE FOUND (request_cell_size flood), szn-specific: the user’s matrix test settled it — mosh -- opencode works, mosh -- tmux + opencode works, ssh -t opencode works, but mosh -- szn + opencode freezes. So it is NOT mosh; it is szn. Live client-log counters showed the client receiving ~9,587 .request_cell_size messages for only 12 output frames, and the server logged sendRequestCellSize firing 201,442 times. The chain: opencode sends a DCS sequence that szn’s parser treats as a sixel (\x1bP…q), which calls addSixelImage → sets cell_size_needs_refresh/pending_sixel → the server re-arms needs_cell_size_refresh on every pane read (sixel-wait, #204) and, since the client’s queryCellSize costs a 5 ms stdin poll each, the flood froze the client before any real frame could be processed. tmux has no such sixel/request path, hence it works.

Resolution (verified working): rate-limiting sendRequestCellSize to 2/sec stops the flood — each request still costs the client a 5 ms queryCellSize poll, but a handful is negligible instead of a 201,442-request freeze. mosh -a it0 -- szn + opencode now starts perfectly. (The underlying sixel false-positive on opencode’s DCS remains a cosmetic ~2 s startup pause via the #204 sixel-wait; a sixel DCS captured: len=N, preview=… log records the trigger for a future parser fix.)

Follow-up (cell-size reply leaked to the screen): img2sixel’s CSI 14 t reply (ESC [ 4 ; h ; w t) was showing up on screen. The old queryCellSize did a 5 ms blocking poll, which always times out over mosh latency; the late reply then sat in stdin and was forwarded to the pane as a keystroke, so the shell echoed it. The client now writes the query, sets an awaiting_cell_size flag, and consumes the reply in the main stdin loop (accumulating partial replies up to 32 bytes, parsing a complete ESC [ 4 ; h ; w t, and forwarding only non-reply bytes as keystrokes). Regression tests: parseCellSizeResponse, cellSizeResponseLen.

Follow-up (100% CPU during img2sixel under Alacritty): while a pane has a buffered pending_sixel awaiting a measured cell size (#204 sixel-wait), handlePtyEvent returned early without reading the pty, but the pane master kept reporting POLL.IN (level-triggered), so the loop busy-spun at 100% CPU until the cell size arrived. Fixed the same way as the flow-control spin: a new tickSixelWait() (called every loop iteration) removes POLL.IN from panes with a pending sixel, enforces the cell_size_wait_ms timeout, and re-arms the pane once the sixel is flushed. It deliberately does not re-arm while flow control is throttling (which would undo that unarm and re-introduce its spin).

Still 90% CPU after the above: a 90% CPU report persists under Alacritty despite the sixel-wait and flow-control spin fixes. Not yet reproduced locally (the pty test harness can’t deliver keystrokes to the pane). Added rate-limiting diagnostics to attribute it: the server logs server loop: N iters/2s, behind=… every 2 s (a run-away N = spin, bounded N + large frame = legitimate render), and the client logs client: out_buf=N backpressure=… every 2 s (a large N = the client relaying a big sixel frame to the terminal).

ROOT CAUSE of the 90% CPU (render re-emits the sixel every frame): the user’s logs showed renderToDisplayClient: 72690-byte content frame repeated hundreds of times, and the client’s backpressure on (out_buf=585742) / off oscillating around the 512 KiB watermark. renderSixelImages stored sixel_last_anchor but never used it — it unconditionally re-emitted every sixel image (erase-all + the full image DCS) on every render. Sixel images live out-of-band from last_cells, so the diff state can’t represent them; while a slow client (Alacritty rendering the image) kept the screen dirty, each render re-sent the whole 72 KiB sixel, which re-congested the client → the dirty flag never cleared → endless re-render at high CPU. Fix: renderSixelImages now computes each pane’s image anchors first and skips the entire erase+redraw when no pane’s image layout changed (still erasing for removed images and redrawing for moved ones).

Follow-up (display dies after many images / 12 MiB frames / freeze): with yazi showing many previews, the logs showed 12130531-byte content frame — the render re-emitted all accumulated sixel images whenever any changed, and a >1 MiB frame exceeds MAX_OUT_BUF, so it was dropped and the client diff state reset, breaking the display; the flood then froze the session. Also, anchor-only change detection missed an image replaced at the same position (same anchor, new content), leaving stale thumbnails. Fix: Screen.sixel_last_id records the image id per slot, and renderSixelImages only re-emits a slot when its anchor or id changed — a pure replacement at the same anchor draws just that image (no erase-all), and a full erase-all+redraw happens only when an image is removed or moved. Regression test: replacing an image at the same anchor emits only the new DCS.

Follow-up (single oversized image freezes szn): the trigger was a specific file — a 1024×1536 PNG whose native-resolution sixel is ~2.6 MiB. On a wide terminal its footprint (77×103 cells) fits the grid, so it gets placed, but the render emits a 2.6 MiB frame that exceeds MAX_OUT_BUF (1 MiB) and is dropped, resetting last_cells; the server then re-renders and re-drops it forever — the freeze/display-death. Fix (revised): raise the buffer caps so a large sixel is delivered, not dropped-and-looped — MAX_OUT_BUF/MAX_CLIENT_OUT_BUF/MAX_PACKET_SIZE are 16 MiB, appendClientOut queues a single large frame even when it crosses the watermark (only additional frames are skipped once the buffer is already large), the client throttle watermark stays at 512 KiB, and MAX_SIXEL_DATA is 8 MiB (only truly pathological images are dropped). yazi’s 2.6 MiB preview now relays through and renders instead of freezing.

Follow-up (ghost images persist after yazi / image replacement): with yazi displaying and exiting, the terminal kept stale sixel overlays — ghosts re-appearing (in order) after exit, and old images left behind when the preview changed. Two causes: (1) exiting the alternate screen freed the marker cells but never freed the alt screen’s sixel_images entries, so the renderer re-emitted them; (2) the renderer redrew images whose marker cells were gone. Fix: useAltScreen(false) frees the alt screen’s sixel images (dropScreenSixels); computeSixelState skips images with no visible markers (refcount 0) and images from the inactive alt_screen; and renderSixelImages draws only changed images (erase-all+redraw only for removals/moves), so stale overlays are erased but every image isn’t re-sent on each change — which, with large sixels, flooded the client and re-froze it after 2-3 img2sixel runs. Regression tests: alt-screen images freed on exit; same-anchor replacement emits only the new DCS.

Follow-up (session unresponsive after img2sixel): after a few img2sixel runs the session froze even though the server never spun (loop diag stayed at 1 iters/2s) and the client drained its output (out_buf 0). The client’s awaiting_cell_size stdin handling buffered any input starting with \x1b[ as a “possible CSI 14 t reply”; with the server re-requesting cell size every 500 ms, escape-sequence input (arrow keys etc.) could be swallowed, making the session feel frozen. Fix: only buffer input that actually starts like a cell-size reply (\x1b[4); everything else forwards immediately. Verified fixed — no CPU spike, no frozen client, no ghost images.