Serial Console

What It Is

A serial console is an RS-232-style byte-at-a-time text terminal exposed over a UART (Universal Asynchronous Receiver/Transmitter). Pre-USB, every PC had a physical DE-9 RS-232 port on the back; modern server hardware keeps it as a DE-9 header, an RJ-45 jack, or a USB-to-UART bridge you plug in.

At the protocol level, it's as simple as it sounds: one wire transmits bytes, one wire receives, both sides agree on a baud rate (bits per second) and a framing (bits per byte, parity, stop bits). There's no flow control negotiation, no link-layer handshake, no DNS. A garbled byte is just a garbled byte.

That crudeness is exactly why it's useful.

Why It Matters to FortrOS

Every other observability channel depends on something that isn't ready yet:

Channel	Requires
SSH	Network up, kernel booted, sshd running, firewall open
HTTP logs	Network up, web server running, you have the URL
Ping	Network up, routing configured
Attached display	Physical display plugged in, GPU driver loaded
IPMI / BMC	Separate management network, credentials, vendor software
Serial console	A UART that the CPU can write bytes to

A node that's stuck in firmware, crashing in the initramfs, kernel-panicking before userspace starts, or rebooting in a loop has no SSH, no network, and sometimes no display at all. The serial console works anyway because it's what the firmware, the kernel, and the initramfs were all told to write to before any of the higher-level stacks were built.

FortrOS deliberately structures itself so that every boot stage produces useful serial output. A black screen is never an acceptable failure mode during development -- the agent or admin watching the serial log must be able to see what went wrong.

FortrOS Conventions

Physical setup

FortrOS's reference hardware is a Dell OptiPlex on a lab desk attached to a WSL dev host via a USB-to-serial dongle. The dongle shows up on the dev host as /dev/ttyUSB0 and is forwarded into WSL through usbipd-win. A persistent tail -F on the resulting capture file is the primary observability channel for everything that happens before gossip is alive.

On nodes that also have a display, the serial console and the display both receive the same stream -- see the kernel cmdline below.

Kernel cmdline

All three boot stages (bootstrapper, preboot, main-OS generation) boot with:

console=tty0 console=ttyS0,115200n8

console=tty0 -- tty0 = the kernel's primary virtual console, which is whatever attached display is active (HDMI / DP / etc.). Multi-head boards get the first active framebuffer.
console=ttyS0,115200n8 -- ttyS0 = the first UART. 115200 bits/sec, n = no parity, 8 = 8 data bits. Stop bits default to 1.

Both are listed, so the kernel mirrors to both. The last console listed also becomes the controlling tty (for /dev/console-based writes). Putting ttyS0 last means init scripts, panic messages, and printf > /dev/console all land on the serial capture in addition to the local screen.

115200 is chosen because it's universally supported by USB-UART bridges, fits into an 8-bit UART buffer comfortably at modern CPU speeds, and is fast enough that kernel log floods don't become a bottleneck. 9600 was the old default -- do not accept it silently, it garbles half the boot.

Log line convention

FortrOS binaries write structured log lines with a component: message prefix:

fortros-preboot: /persist unlocked and mounted
fortros-bootstrapper: invite pubkey=95bed88ba7f3...
generation-authority: returning node authenticated via H(preboot_secret)
provisioner: /preboot-auth ok, nonce=abc..., peer=10.0.0.5
maintainer: MEMBER UP Node([fd92:...]:7100:1776548779)
persist-mount: LUKS volume at /dev/nvme0n1p2

The prefix is the source component (a Rust binary or an s6-rc service run script). It's there specifically so grep -E 'fortros-preboot:| maintainer:' reliably extracts one service's output from an interleaved stream. Kernel messages and external tool output (dnsmasq, efibootmgr) keep their native prefixes; FortrOS code does not pretend to be those.

Convention: what goes to the console

Boot stages (bootstrapper, preboot, main-OS init): everything of narrative interest. A human or agent reading the log should be able to reconstruct what happened step by step.
Long-running services (maintainer, provisioner, gen-auth): lifecycle events (start, stop, MEMBER UP/DOWN, major error paths). Avoid per-request spam on the serial channel -- it's shared with everything else.
One-off commands: output via the normal stdout/stderr of the command, captured separately. Don't push it to /dev/console.
Panics / fatal errors: must reach the console. Rescue shells drop to ttyS0 so remote debug is possible.

"FortrOS for Agents" Tie-In

FortrOS is being built in a paired-agent workflow: a human admin + Claude Code operating on the same codebase and the same running cluster. The serial console is the primary observability channel that an LLM agent can consume reliably, and that shaped several of the conventions above.

Why serial is LLM-friendly

Plain text, one event per line. The agent's monitor tool streams stdout lines to the conversation; each log line becomes a notification. No binary formats, no HTML, no browser state, no JSON schema drift.
Structured prefixes. component: message lets a monitor grep filter pull exactly one narrative thread out of an interleaved boot. The pattern is stable across all FortrOS binaries, so the agent's grep expressions don't need to know the list of binaries ahead of time; matching on the prefix shape is enough.
Everything lands there eventually. During firmware / initramfs / pre- network stages, the serial log is the only thing the agent can read. Once gossip + SSH are up, the agent has more channels, but serial remains the baseline "did the box actually boot?" signal.
Write-only from the node side. The agent never has to send input to the serial console during normal operation; it's a one-way event stream. That matches the agent's monitor model well (stream events, react by calling other tools like SSH or the web UI, don't try to drive a TUI).

The observability loop in practice

The loop looked like this for most of the work that built FortrOS:

  (kernel / FortrOS binary)
        |
        v  writes to /dev/console
  [ttyS0 UART]
        |
        v  USB-to-serial dongle, usbipd-win into WSL
  /dev/ttyUSB0 on the dev host
        |
        v  minicom / picocom captures to a file
  /tmp/dell-serial.log
        |
        v  tail -F | grep --line-buffered -E '<filter>'
  Monitor tool in the agent session
        |
        v  each matching line = one notification
  Agent reasons about the event, decides next action:
    - ssh in and inspect?
    - push a fix?
    - flag it to the human?

The filter is intentionally wide (match on component prefixes + a small list of urgent signals like Kernel panic / rescue shell). Silence during a boot is itself a signal -- if fortros-preboot: doesn't emit anything for 30s, something wedged.

Conventions that make this work

Never use ANSI escapes on the serial console. They don't render in a line-based log capture and they confuse grep. Terminal colors belong on interactive TTYs, not on /dev/console.
Never emit progress bars. A \r-based spinner in a byte-stream log is unreadable. Print one line per meaningful state change; if something takes a long time, print start + end lines and let the reader infer the duration from the timestamps.
Include enough prefix in identifiers for partial-match grep. nonce=abc..., pubkey=ffee1234... -- first 12+ hex digits is the convention. Lets the agent cross-correlate events across services without leaking full credentials into the log.
Panic / error messages name the component. fortros-preboot: FATAL: no CA pubkey at /etc/... is greppable; FATAL: no CA pubkey alone isn't, because an agent scanning a multi-service log can't tell who died.

Together these conventions make serial console + monitor grep a first-class development interface, not a last-resort debugging tool.

Out-of-Band Management -- IPMI / BMC alternatives when serial isn't available (rare on our target hardware).
UKI -- UKIs carry kernel cmdlines, including the console= setup.
Intel ME -- AMT-over-serial is the out-of-band equivalent Intel ME provides on business-class hardware; usually disabled in FortrOS.