The garbage burst when we open /dev/ttyACM0

Investigated 2026-09-21 on solomon (Manjaro, Linux 7.1.9-1-MANJARO, in-tree cdc_acm, pyserial 3.5, Python 3.14.7) against the LilyGO T-Display-K230’s CH342K console bridge (1a86:55d2, two CDC-ACM ports: ttyACM0 = UART0, ttyACM1 = UART3).

The board was never corrupted. Every burst we chased was made on this host.

1. The short version

ModemManager is running on this machine and probes /dev/ttyACM0 and /dev/ttyACM1 every single time the CH342 enumerates — which, because the board resets constantly during debugging, is every single time we go to capture a boot. During the probe it:

takes the port with TIOCEXCL, so our own open() is refused for the duration;
reprograms the CH342’s UART to 57600 baud — its default — while the K230 is talking at 115200;
pulses the line at baud 0 (“flashing” the port);
writes AT six times, three seconds apart — an 18-second window;
then writes a binary QCDM/DIAG frame;
gives up (“unhandled port type”) and releases the port.

That single cause produces both symptoms we saw:

RX garbage. For 18+ seconds the CH342 samples a 115200 line at 57600. The mis-framed bytes accumulate in the chip’s internal receive FIFO. When we finally open the port and pyserial sets 115200, the FIFO drains first — garbage, then perfectly clean text. A periodic source (an idle prompt, a repeated line) undersampled 2:1 produces a short repeating byte pattern, which is exactly the shape of the 0xF4 0xA4 0xF4 0xA4 … burst; it is not noise.
TX garbage into U-Boot. ModemManager’s AT bytes are transmitted at 57600 into a receiver running at 115200. U-Boot decodes them as junk characters, keeps the printable ones in its command-line buffer, and echoes them back at us. That is where Unknown command 'kKBBBBkBBBkrun' comes from: the junk was already sitting in U-Boot’s line editor before we sent run, and what we “captured” was U-Boot’s echo of it.

There is a second, smaller contributor that survives even if ModemManager is removed: cdc_acm programs 9600 baud into every CDC device at probe time and does not touch it again until the first open(). So the very first open after each enumeration always has some mis-sampled data behind it.

2. Evidence

2.1 ModemManager is probing this exact device (primary, this host)

journalctl -u ModemManager on solomon, unedited:

Sep 21 11:32:33  <msg> [ttyACM0/probe] probe step: start
Sep 21 11:32:33  <msg> [ttyACM0/probe] probe step: AT open port
Sep 21 11:32:33  <msg> [ttyACM0/probe] probe step: AT
Sep 21 11:32:51  <msg> [ttyACM0/probe] probe step: AT close port
Sep 21 11:32:51  <msg> [ttyACM0/probe] probe step: QCDM
Sep 21 11:32:51  <msg> [ttyACM0/probe] probe step: done
Sep 21 11:32:51  <msg> [device …/usb3/3-1] creating modem with plugin 'generic' and '2' ports (1a86:55d2)
Sep 21 11:32:51  <wrn> [plugin/generic] could not grab port ttyACM0: Cannot add port 'tty/ttyACM0', unhandled port type

The same block repeats at 11:06:22, 11:32:33 and 11:39:58 — once per re-enumeration. Note the vendor/product: (1a86:55d2), our CH342. Note the duration: 18 seconds, 11:32:33 → 11:32:51.

It happens because /usr/lib/udev/rules.d/80-mm-candidate.rules says, with no vendor filtering at all:

SUBSYSTEM=="tty", ENV{ID_MM_CANDIDATE}="1"

and ModemManager.service on this host starts with a bare ExecStart=/usr/bin/ModemManager — no --filter-policy=strict.

2.2 What ModemManager does to the port (primary, MM 1.24.2 source)

Installed version is 1.24.2 (mmcli --version), so these quotes are from modemmanager 1.24.2-2 on sources.debian.org — the matching release.

src/mm-port-serial.c:

g_param_spec_uint (MM_PORT_SERIAL_BAUD, "Baud", "Baud rate",
                   0, G_MAXUINT, 57600, G_PARAM_READWRITE)

→ 57600 is the default baud when nothing sets ID_MM_TTY_BAUDRATE (nothing does here). Corroborated by a string in the shipped binary:

$ strings -a /usr/bin/ModemManager | grep 57600
baudrate invalid: %u; defaulting to 57600

Same file:

if (ioctl (self->priv->fd, TIOCEXCL) < 0)     /* exclusive access */
…
tcflush (self->priv->fd, TCIOFLUSH);          /* on open and on close */
…
stbuf.c_cflag |= (bits | parity | stopbits | CLOCAL | CREAD);

and mm_port_serial_flash() drops the port to B0 for flash_time and then restores the speed. The binary carries the matching help text:

$ strings -a /usr/bin/ModemManager | grep -i flash
Flashing the port (0 baud for a short period) is allowed.

src/mm-port-probe.c:

static const MMPortProbeAtCommand at_probing[] = {
    { "AT",  3, mm_port_probe_response_processor_is_at },
    …
};

six tries by default at a 3 s timeout each = the 18 s the journal shows, then a QCDM version-info frame (0x7E-delimited HDLC). The binary also contains ATE1, +GCAP, +CGMI, +CGMM for the follow-up probes.

TIOCEXCL is why capture-boot.py’s except: time.sleep(0.2) retry loop exists at all, and why the comment there — “udev has not finished with it yet” — is wrong. It is not udev. It is ModemManager holding the port for 18 seconds while writing AT at the wrong baud into U-Boot.

2.3 What cdc_acm does on probe and on open (primary, kernel source)

From drivers/usb/class/cdc-acm.c (mainline):

	/* in acm_probe() */
	acm->line.dwDTERate = cpu_to_le32(9600);
	acm->line.bDataBits = 8;
	acm_set_line(acm, &acm->line);

	acm_tty_driver->init_termios.c_cflag = B9600 | CS8 | CREAD |
							HUPCL | CLOCAL;

static int acm_port_activate(struct tty_port *port, struct tty_struct *tty)
{
	…
	acm_tty_set_termios(tty, NULL);     /* SET_LINE_CODING */
	clear_bit(ACM_THROTTLED, &acm->flags);
	retval = acm_submit_read_urbs(acm, GFP_KERNEL);   /* only now */

So: the chip is programmed to 9600 the moment it enumerates, and stays there until somebody opens the tty. Read URBs are only submitted at open, which means that until then nothing drains the chip’s bulk IN endpoint and whatever the chip decoded piles up in its own FIFO.

Two further details from the same file that matter:

	if (C_BAUD(tty) == B0) {
		newline.dwDTERate = acm->line.dwDTERate;
		newctrl &= ~USB_CDC_CTRL_DTR;
	}

— this is what ModemManager’s “flash” hits: B0 makes cdc_acm drop DTR.

	{ USB_DEVICE(0x1a86, 0x55d3), .driver_info = MISSING_CAP_BRK, },

— WCH’s CH343 (55d3) is already quirked in-tree for lying about BREAK support. Our 55d2 is not in the table. Mentioned only to show that WCH CDC parts are known to need special handling; it is not the cause here.

2.4 What pyserial does on open (primary, installed source)

/usr/lib/python3.14/site-packages/serial/serialposix.py, Serial.open(), in order:

os.open(port, O_RDWR|O_NOCTTY|O_NONBLOCK) — this is where the kernel runs acm_port_activate and starts reading at whatever baud is currently programmed, and where tty_port_block_til_ready() raises DTR/RTS;
self._reconfigure_port(force_update=True) — the tcsetattr that finally sends SET_LINE_CODING for 115200;
_update_dtr_state() / _update_rts_state();
self._reset_input_buffer() → termios.tcflush(fd, TCIFLUSH).

Three consequences, all load-bearing:

pyserial already flushes input on open, and it does not help. tcflush discards what has reached the tty buffer at that instant. The chip’s FIFO and the in-flight bulk IN URBs are downstream of it and arrive afterwards. This is precisely why “it already calls reset_input_buffer()” has not saved us — the flush is simply too early.
You cannot hold DTR/RTS low across open(). The kernel raises them in step 1; pyserial’s .dtr = False is applied in step 3, after the fact. The dtr/rts setters only call the ioctl if self.is_open, so setting them before open just records the wish. dsrdtr=True / rtscts=True merely make pyserial skip its own ioctl in step 3 — they do not stop the kernel.
exclusive=True is flock(), not TIOCEXCL: fcntl.flock(self.fd, fcntl.LOCK_EX | fcntl.LOCK_NB). It is advisory. It will keep a second copy of our script (or tio, which also uses flock) out, but it cannot keep ModemManager out — ModemManager does not take an flock, it takes TIOCEXCL, and TIOCEXCL beats us, not the other way round.

The kernel-side DTR/RTS raise is tty_port_block_til_ready() → tty_port_raise_dtr_rts() → acm_port_dtr_rts(port, true) → CDC SET_CONTROL_LINE_STATE with DTR|RTS. It is unconditional; there is no userspace opt-out short of patching the kernel. (https://qsantos.fr/2025/05/03/linux-always-toggles-dtr-rts/)

2.5 DTR/RTS are a dead end on this board (primary, the schematic)

repo/schematic/T-Display K230_V1.0_NEW.pdf, U11, is a CH342K — the 10-pin part. Its entire net list on this board is:

UD+ , UD- , GND , VDD5 , VIO , V3 ,
TXD0_CH342 -> UART0_RXD      RXD0_CH342 <- UART0_TXD
TXD1_CH342 -> UART3_RXD      RXD1_CH342 <- UART3_TXD

There is no DTR, RTS, CTS, DSR, DCD or RI pin on the package, and none on the board. Nothing the host does to modem control lines can physically reach the K230. Every hardware theory that routed through DTR — auto-reset, boot-strapping, a BREAK on the target — is ruled out by the schematic. The garbage is data, and only data.

2.6 A wrong-baud receiver reproduces the shape of what we saw

Written for this investigation (kept out of the repo; it touches no hardware): build the physical 8N1 waveform for a byte stream at one baud, then sample it the way a UART receiver at another baud would — hunt for a start bit, sample eight data bits at the mid-point, resynchronise.

ModemManager writing AT\r at 57600 into a 115200 receiver:

TX=57600 RX=115200
  raw: 06 90 60 66 1e 98 00 06 66 e6 80 06 98 60 66 1e 98 00 06 66 e6 80 …
  the printable subset U-Boot's line editor would keep and echo: b'`ff`ff`ff'

An idle K230# prompt at 115200 sampled by a receiver at 57600:

TX=115200 RX=57600
  raw: 94 48 c8 5a 1a 98 5a 1a 68 5a 1a 98 5a 1a 98 5a 1a 98 5a 1a 98 …

Both reproduce the two things that made the real bursts look like hardware faults: they are short, repeating, low-entropy patterns, and the host→board direction yields a run of ordinary printable ASCII, because U-Boot throws away the non-printables and keeps and echoes the rest.

This is a shape match, not a byte-for-byte reconstruction of kKBBBBkBBBk / 0xF4 0xA4 — we do not know exactly what the board was printing at that instant, nor which of the two wrong-baud epochs (9600 from cdc_acm probe, 57600 from ModemManager) each burst came from. Treat the exact byte mapping as unverified; treat the mechanism as established.

3. What was considered and ruled out

Hypothesis	Verdict
Opening asserts DTR/RTS and the CH342 answers with data or a BREAK	Ruled out. CH342K has no DTR/RTS pins (§2.5). The kernel does raise the CDC control-line bits, but they go nowhere.
`HUPCL` on close causes a transition on the next open	Not the cause. `cdc_acm`’s `init_termios` does carry `HUPCL`, and the hangup does drop DTR — but with no DTR pin it cannot reach the board. `stty -hupcl` is not worth doing here.
The chip is reprogrammed after open, so in-flight bytes decode at the wrong rate	Confirmed, and it is the core of it (§2.3, §2.4) — but the window is far wider than “in flight”: it is the whole interval from enumeration to our `tcsetattr`, which ModemManager stretches to 18+ seconds.
`reset_input_buffer()` at open fixes it	No. pyserial already does exactly that and it fires too early (§2.4). It works only after a settle delay.
`exclusive=True` fixes it	No. It is an advisory `flock`, and ModemManager uses `TIOCEXCL` (§2.4). Still worth having to stop two copies of our own tooling fighting.
Swap `cdc_acm` for WCH’s vendor driver	Helps a little, does not fix it. `WCHSoftGroup/ch343ser_linux` covers CH342 and sets `ch343->line.dwDTERate = 115200` at probe rather than 9600, so it removes the `cdc_acm` 9600 epoch. But its device is still a `tty`, so `80-mm-candidate.rules` still tags it and ModemManager still probes it (§2.1). An out-of-tree module to rebuild on every kernel bump, for part of the problem. Not recommended.
A WCH errata / CH342 firmware bug	No evidence found, and none needed — the host-side explanation is complete and each step is observable in the journal.
The board is crashing	No. This is the thing that cost us the most time.

4. The fix

4.1 Prevention: stop ModemManager touching the board (do this first)

tools/99-tdisplay-k230-no-modemmanager.rules, installed to /etc/udev/rules.d/:

SUBSYSTEM=="usb", ATTRS{idVendor}=="1a86", ATTRS{idProduct}=="55d2", ENV{ID_MM_DEVICE_IGNORE}="1"

sudo install -m0644 tools/99-tdisplay-k230-no-modemmanager.rules /etc/udev/rules.d/
sudo udevadm control --reload
# then unplug and replug the console cable

ID_MM_DEVICE_IGNORE is honoured under every filter policy including strict, and tagging the usb_device (not the tty) covers both ports at once. (https://modemmanager.org/docs/modemmanager/port-and-device-detection/)

This removes the 18-second TIOCEXCL lockout, the AT bytes going into U-Boot, the B0 flash, and the 57600 epoch. It does not remove the cdc_acm 9600-at-probe epoch, so keep §4.2 as well.

systemctl mask ModemManager would also work and is the right move on a machine with no WWAN hardware, but the udev rule is narrower and survives reinstalls.

4.2 Mitigation in `tools/capture-boot.py`

Complete prevention is not available: even with ModemManager gone, cdc_acm leaves the chip at 9600 from enumeration until the first open, so the first open always has some mis-sampled data queued behind it. The script therefore has to discard deterministically rather than hope. It now:

waits --settle seconds (default 0.30) after opening before doing anything else, then calls reset_input_buffer() / reset_output_buffer();
does not throw the settle-window bytes away silently — it writes them to the log inside an explicit --- discarded N bytes … --- marker, so a future reader can never again mistake them for the board misbehaving;
clears the --expect match buffer at that point, so a prompt can only ever be matched in post-flush data and --send can never fire off garbage;
sends \x15 (kill-line: CTL_CH('u') in U-Boot’s cread_line, VKILL in a canonical tty) immediately before each --send, so anything already sitting in the target’s line editor is discarded instead of being prefixed onto our command. --no-line-kill turns this off;
takes an advisory flock (--no-exclusive to opt out) and says so in the log when the port is busy, instead of retrying in silence;
checks at startup whether ModemManager is running without the udev rule in place, and prints the one-line fix.

--hammer, --kick, --expect/--send, --out, --seconds, --append and --quiet-exit are unchanged. --hammer and --kick now run after the settle+flush, which costs 0.3 s of the autoboot window and in exchange stops the hammer’s own keystrokes from being mixed into the discarded region. With the udev rule installed, --settle 0.05 is enough.

5. Confidence

Established from primary sources:

ModemManager probes 1a86:55d2 on this host for 18 s per enumeration, holds it with TIOCEXCL, uses 57600, flashes at B0, writes AT and a QCDM frame (journal from this machine + MM 1.24.2 source + strings in the installed binary).
cdc_acm programs 9600 at probe, sets line coding before submitting read URBs at open, and has HUPCL in init_termios (mainline source).
pyserial’s open order, the too-early tcflush, exclusive being flock, and DTR/RTS being applied only after open (installed 3.5 source).
The kernel raises DTR/RTS unconditionally on open and there is no userspace opt-out.
The CH342K on this board has no modem-control pins (the schematic in-tree).
WCH’s ch343ser_linux sets 115200 at probe instead of 9600.

Inference, consistent with everything above but not directly measured:

That each specific burst we saw (kKBBBBkBBBk, 0xF4 0xA4 …) is byte-for-byte the product of one particular wrong-baud epoch. The shape is reproduced by simulation; the exact bytes are not, because the board’s output at that instant is unknown.
That kKBBBBkBBBk reaching U-Boot’s command buffer is specifically ModemManager’s AT traffic rather than the tail of some other write. It is the only writer on the port at that time and it writes at the wrong baud, so it is the strong candidate — but we did not capture it with a bus analyser.
That the CH342’s receive FIFO, rather than an in-flight URB, is where the mis-decoded bytes wait. WCH documents “independent transmit-receive buffers” but the datasheet PDF would not render for a size figure.

The one way to close both inferences — once the board is free — is to install the udev rule, replug, and capture with --settle 0. If the burst disappears entirely, ModemManager was the whole story; if a short burst remains, that residue is the cdc_acm 9600 epoch. Neither test may be run while a capture is live.

Sources

journalctl -u ModemManager on solomon, 2026-09-21 (this host)
strings -a /usr/bin/ModemManager, mmcli --version → 1.24.2 (this host)
/usr/lib/udev/rules.d/80-mm-candidate.rules, /usr/lib/systemd/system/ModemManager.service (this host)
/usr/lib/python3.14/site-packages/serial/serialposix.py (pyserial 3.5, this host)
repo/schematic/T-Display K230_V1.0_NEW.pdf (in-tree)
drivers/usb/class/cdc-acm.c
ModemManager 1.24.2 src/mm-port-serial.c
ModemManager 1.24.2 src/mm-port-probe.c
ModemManager: port and device detection
ModemManager: modem filter
Downtown Doug Brown, “Fix for USB serial port being opened by ModemManager at startup”
Quentin Santos, “Linux always toggles DTR & RTS”
WCHSoftGroup/ch343ser_linux
CH342 datasheet (v1E)