The hang and the dark panel are one bug: the DSI PHY never reports ready

Root cause, established on hardware 2026-09-21 after four wrong hypotheses.

The trace

Captured by appending softlockup_panic=1 to the kernel command line at the U-Boot prompt. That was necessary: the ordinary soft-lockup path on this single-hart board prints its header and then nothing — no modules, no registers, no trace — because the watchdog interrupt preempts the stuck task on the only CPU and then blocks on something that task holds. The panic path prints fine.

watchdog: BUG: soft lockup - CPU#0 stuck for 26s! [kworker/0:5:44]
Modules linked in:
CPU: 0 PID: 44 Comm: kworker/0:5 Tainted: G    B    6.6.36 #1-NixOS
Hardware name: LILYGO T-Display-K230 (DT)
Workqueue: events output_poll_execute
epc : k230_dsi_config_4lan_phy+0x562/0x620
 a4 : 0000000000001fbd
status: 0000000200000120 badaddr: 0000000000000000 cause: 8000000000000005

[<..>] k230_dsi_config_4lan_phy+0x562/0x620
[<..>] canaan_dsi_encoder_enable+0x216/0x532
[<..>] drm_atomic_helper_commit_modeset_enables+0x194/0x1b4
[<..>] drm_atomic_helper_commit_tail_rpm+0x54/0xa2
[<..>] commit_tail+0x86/0x160
[<..>] drm_atomic_helper_commit+0x13c/0x170
[<..>] drm_atomic_commit+0x86/0xb6
[<..>] drm_client_modeset_commit_atomic+0x1ec/0x228
[<..>] drm_client_modeset_commit_locked+0x50/0x124
[<..>] drm_client_modeset_commit+0x2e/0x4e
[<..>] __drm_fb_helper_restore_fbdev_mode_unlocked+0x86/0xc6
[<..>] drm_fb_helper_hotplug_event+0xda/0xe8

cause 0x8000000000000005 has bit 63 set: a supervisor timer interrupt, i.e. the watchdog firing while the CPU spins. Not a fault. badaddr is 0 because nothing faulted.

The code

drivers/gpu/drm/canaan/canaan_phy.c, twice:

	while (readl(dsi->base + PHY_STATUS) != 0x1fbd)
		;

Bare infinite loops, body a single semicolon. No timeout, no iteration cap, no sleep. a4 in the register dump holds 0x1fbd, the value being waited for. The D-PHY on this board never reports it.

The vendor did bound the PHY_TST_CTRL1 loop immediately above these with count >= 1000; break, so the omission looks accidental rather than deliberate.

Why this is also the dark panel

canaan_dsi_encoder_enable() never returns, so the modeset never completes and drm_panel_prepare() is never called — which is what would have written the RM69A10 DCS init sequence, including sleep-out (0x11) and display-on (0x29). The dev_info added at the top of canaan_panel_prepare() never printed on any boot, and now we know why: execution never gets there.

This retires the earlier reading in panel-dark.md that the panel was being skipped. It is not skipped; the pipeline stalls one stage earlier.

Why it only appeared once fb0 existed

The bottom of the trace is drm_fb_helper_hotplug_event. The fbdev helper performs the modeset. Before the 16 bpp fix, fbdev emulation failed (XR24 not advertised by the planes), no framebuffer was created, nothing ever drove a full modeset, and so nothing hung — the panel was simply dark. Fixing fbdev did not cause the bug; it exposed one that was always there.

Prime suspect for why the PHY never reports ready

canaan_dsi.c:383 calls k230_dsi_config_4lan_phy() unconditionally, and only afterwards sets the real lane count at line 403 via canaan_dsi_set_lan_num(dsi, device->lanes) (which accepts 1, 2 or 4).

This panel is panel-dsi-lane = <2>. If 0x1fbd encodes “all four lanes ready”, the two absent lanes can never assert their bits and the wait can never succeed. Unverified — there is no register documentation in the tree, and the bit layout of PHY_STATUS is unknown.

What was changed

Both loops are bounded and now report what the PHY actually says:

{ int _w = 0; u32 _s = 0;
  while ((_s = readl(dsi->base + PHY_STATUS)) != 0x1fbd && _w++ < 2000)
          usleep_range(100, 200);
  if (_s != 0x1fbd)
          dev_err(dsi->dev, "PHY_STATUS 0x%x != 0x1fbd after %d tries, continuing\n", _s, _w); }

This is deliberately a diagnostic, not a fix. It converts an unkillable hang into a logged value. The observed PHY_STATUS is the evidence needed to decide the real fix — whether the expected value should depend on lane count, and if so what it should be for two lanes.

Four hypotheses this replaces

Every one of those cost a build and a flash, because the kernel could not print a backtrace: k230_defconfig sets CONFIG_SOFTLOCKUP_DETECTOR=y with no FRAME_POINTER, so dump_stack() emitted nothing. The lesson is to fix the instrument before forming the fifth theory, not the first.

Resolved: the display power domain was the cause

Pinning K230_PM_DOMAIN_DISP at probe — pm_runtime_get_sync() in canaan_drm_probe() rather than only in canaan_drm_open() — fixed the whole SoC-side chain. On the next boot, every symptom above is gone:

[ 3.055734] canaan-mipi-dsi 90850000.dsi: Attached device universal
[ 3.136970] [drm] Initialized canaan-drm 1.0.0 20230501
[ 3.162288] [drm] fb0: canaan-drmdrmfb frame buffer device
[ 3.179270] [drm:canaan_drm_bind] Canaan K230 DRM driver register successfully
[ 3.299730] canaan-panel-dsi 90850000.dsi.0: canaan_panel_prepare: entered, init_set_v1_flag=1

• No PHY_STATUS 0xffffffff. The PHY now reaches 0x1fbd, so the bounded wait succeeds instead of timing out 2001 times.
• No failed to write dcs cmd: -110. All 20 init commands are written.
• canaan_panel_prepare: entered appears for the first time on any boot.
• No soft lockup; the board boots to a login prompt.

This also retires the guess that 0x1fbd was a four-lane encoding an unreachable for a two-lane panel. It was reachable all along; the block was simply unpowered.

The panel is still dark, and it is now a panel-side problem

State read off the running board:

crtc[46]: canaan_crtc
	enable=1
	active=1
plane[33]: plane-1  crtc=canaan_crtc  crtc-pos=568x1232+0+0
	format=RG16 little-endian   size=568x1232  pitch[0]=1136
	dma_addr=0x000000001e100000
connector[48]: DSI-1
gpio-534 (dsi_reset)      out hi
gpio-537 (backlight_gpio) out hi

dd if=/dev/urandom of=/dev/fb0 wrote 1400832 bytes — the whole framebuffer — and a webcam pointed at the board shows no change. Note that writing a pattern was necessary to test this at all: an all-zero framebuffer on a working panel is indistinguishable from a dark one.

So every stage the SoC controls is verified good: domain powered, PHY locked, DCS accepted, CRTC active, plane bound to a real buffer at the right geometry and format. What remains is between the DSI output and the glass.

Remaining suspects, in the order worth testing:

1. The init sequence content. Transcribed from LilyGO’s RT-Smart rm69a10.c into panel-init-sequence. It is accepted by the DSI controller, which says nothing about whether it is correct for this panel. docs/evidence/rm69a10-init-sequence.md has the per-command provenance.
2. Panel supply rails. canaan_panel_prepare() has its power_on assertion commented out by the vendor exactly like the reset pulses were; the GPIO reads high, but whether GPIO25 is the AMOLED’s VCI/ELVDD enable or something else is unverified.
3. vth_line. Ours is 9, the vendor reference uses 10. Unlikely to cause total darkness, but it is a known divergence (docs/evidence/dts-divergence.md).
4. DCS read-back is unimplemented. canaan_dsi_dcs_read() is a stub returning 1, so the panel’s ID registers cannot be read to confirm it is responding at all. Implementing it would turn “the panel is silent” into a measurement.

Also ruled out: the framebuffer pixel format

The DSI wire format is set to RGB888 (dsi->format = MIPI_DSI_FMT_RGB888, panel-canaan-universal.c), while the fbdev fix made the framebuffer RGB565. That mismatch looked like a candidate, so it was tested directly without a rebuild:

modetest -M canaan-drm -s 48:568x1232@AR24

ARGB8888, held open, no error reported. A webcam frame taken while it was driving the panel is indistinguishable from the one taken with the RGB565 framebuffer: still dark. The format is not the problem.

Also ruled out: the panel power GPIO

canaan_panel_prepare() has its power_on assertion commented out, which looked like the reset-pulse bug repeating. It is not: probe does gpiod_direction_output(ctx->power_on, 1) unconditionally (panel-canaan-universal.c:338), and /sys/kernel/debug/gpio confirms gpio-537 (backlight_gpio) out hi on the running board.

Where this leaves it

Everything the SoC drives is verified correct, and three panel-side theories are dead. The two things still untested both point the same way:

• The init sequence is accepted, not validated. The DSI controller ACKing 20 writes says the bus works, not that the bytes are right for this panel.
• canaan_dsi_dcs_read() is a stub (// TODO, returns 1), so the panel cannot be asked anything — not its ID, not its power mode, not whether it received the sequence.

Implementing DCS read is therefore the highest-value next step. It converts “the panel is silent” from an inference into a measurement, and would distinguish a panel that is not responding at all from one that is responding but not lighting.

Also ruled out: vth_line

Set to 10, matching every Canaan board dts in the vendor tree, replacing an ungrounded 9. Boot is clean — Attached device universal, canaan_panel_prepare: entered, init_set_v1_flag=1, no PHY error, no DCS error, reaches a login prompt — and a webcam frame taken with 1400832 bytes of urandom in /dev/fb0 is still dark. The change is kept because 9 had no source, but it is not the fix.

Everything cheap is now exhausted

Verified correct, each on hardware or by mechanical comparison:

Every stage that can be observed from the SoC side reports success, and the panel emits nothing.

The one instrument still missing

canaan_dsi_dcs_read() is a stub:

static int canaan_dsi_dcs_read(struct canaan_dsi *dsi,
			       const struct mipi_dsi_msg *msg)
{
	// TODO
	return 1;
}

So the panel cannot be asked anything. Reading DCS 0x04 (RDDID) or 0x0A (RDDPM, power mode) would distinguish, for the first time, between:

• the panel is not receiving the sequence at all, and
• the panel is receiving it, acknowledging it, and still not lighting.

Every remaining hypothesis — wrong panel variant, a missing supply the board provides elsewhere, a DSI timing parameter the controller accepts but the panel rejects — is untestable without that. Six theories have now died here, and the pattern is consistent: the ones that died cheaply were the ones where an instrument existed.

Implementing DCS read is the next step, not another guess.

The measurement: the panel does not answer

With canaan_dsi_dcs_read() implemented and a caller added to canaan_panel_prepare(), the board reports:

[3.270] canaan-panel-dsi 90850000.dsi.0: canaan_panel_prepare: entered, init_set_v1_flag=1
[3.489] canaan-panel-dsi 90850000.dsi.0: RDDID (0x04) failed: -22
[3.539] canaan-mipi-dsi  90850000.dsi:   DCS read 0x0a: panel did not answer (status 0x50015)
[3.548] canaan-panel-dsi 90850000.dsi.0: RDDPM (0x0A) failed: -110

0x50015 against the driver’s own bits (canaan_dsi.c:37-42):

The command went out; nothing came back; the wait timed out at -110.

Why this changes the reading of every earlier result

“All 20 DCS writes accepted, no errors” has been treated as evidence the panel receives the init sequence. It is not. A DSI write reports only that the controller’s own FIFO drained — the panel never acknowledges it. A read is the first operation on this link that requires the panel to respond, and the panel is silent.

That single fact is consistent with every hypothesis that died here:

• the init sequence is byte-identical to the vendor’s — irrelevant if it is not arriving
• the delays are longer than the vendor’s — same
• LP-before-HS ordering is correct — same
• RG16 vs AR24 made no difference — same
• the CRTC is active=1 with a bound framebuffer — same

None of them were wrong about what they measured. They were all measuring the SoC side of a link whose far end is not responding.

Next suspects, now much narrower

1. Panel supply. backlight_gpio/power_on is asserted, but that is one GPIO; an AMOLED needs VCI and ELVDD, and nothing in the device tree or driver models them. If the panel is unpowered it cannot answer.
2. The physical DSI link. Lane mapping, polarity, or a connector issue. The PHY reports lock (PHY_STATUS == 0x1fbd) but that is the SoC’s own PHY, not proof of a working link to the panel.
3. The part is not an RM69A10 with these addresses.

A bug of mine in the above

RDDID (0x04) failed: -22 is EINVAL, not a timeout: the request asked for 3 bytes and canaan_dsi_transfer() only routes MIPI_DSI_DCS_READ when msg->rx_len == 1, falling through to default: -EINVAL. The RDDPM read used one byte and exercised the new code correctly, so the conclusion stands, but the RDDID call should either request one byte or the dispatcher should be widened.

Leading hypothesis: the PHY is configured for 4 lanes, the panel has 2

Found by reading LilyGO’s RT-Smart driver for this exact panel against the Linux driver.

RT-Smart sets the PHY to 2 lanes for the RM69A10, in rm69a10_set_phy_freq():

mipi_phy_attr.phy_lan_num = K_DSI_2LAN;
connector_set_phy_freq(&mipi_phy_attr);

And it is the only vendor panel that does. Across mpp/kernel/connector/src/:

The Linux driver has no 2-lane path at all. canaan_phy.c exports exactly one PHY configuration function, k230_dsi_config_4lan_phy(), and canaan_dsi.c:383 calls it unconditionally:

k230_dsi_config_4lan_phy(dsi, m - 2, n - 1, voc, 0x96);

canaan_dsi_set_lan_num(dsi, device->lanes) at line 403 sets the lane count in the DSI controller, but nothing tells the D-PHY. The driver was evidently developed against the 4-lane panels; ours is the exception.

This fits the measurement better than anything else so far. The SoC’s own PHY reports lock (PHY_STATUS == 0x1fbd) because both PHY instances are configured and happy — but the data is being striped across four lanes into a panel wired for two. The panel would see malformed traffic, respond to nothing, and never light. Writes would still “succeed”, because a DSI write only reports the controller’s FIFO draining.

It also explains why 0x1fbd is reachable: it is the all-PHYs-ready value for the 4-lane configuration the driver always programs.

Why this is not a quick patch

k230_dsi_phy0_config() and k230_dsi_phy1_config() interleave writes to the second PHY instance (dsi->base + 0x400 + ...) throughout, and there is no register documentation in the tree for TXDPHY_PLL_CFG0/1, TXDPHY_CFG0/1 or the PHY_STATUS bit layout. Producing a correct 2-lane configuration means working out which of those writes belong to the second lane pair and what the ready mask becomes — reverse engineering, not an edit.

connector_set_phy_freq(), which would show the vendor’s own mapping from phy_lan_num to registers, is declared in the RT-Smart tree but implemented outside it, so the delta cannot simply be read off.

Confirmed against the schematic: the panel is physically 2-lane

repo/schematic/T-Display K230_V1.0_NEW.pdf, the LCD FPC connector (J101x group) carries exactly:

DSI_CLK_P / DSI_CLK_N
DSI_D0_P  / DSI_D0_N
DSI_D1_P  / DSI_D1_N
LCD_RST, LCD_VDD

DSI_D2_* and DSI_D3_* appear only at the SoC pinout (balls Y7, Y11 and neighbours) and are never routed to the panel connector.

So panel-dsi-lane = <2> is correct and matches both the schematic and RT-Smart. The defect is that the Linux driver programs the D-PHY for four lanes regardless.

This also rules out the cheap experiment that suggested itself here earlier — setting panel-dsi-lane = <4> to see if the 2 was ours to blame. The board says 2. That test would have been a wasted build.

Suggested work from here

Implement the 2-lane PHY path, using k230_dsi_config_4lan_phy() as the template and gating the second-instance writes (dsi->base + 0x400 + ...) on lane count. Expect the 0x1fbd ready mask to need changing too: it is the all-ready value for a four-lane configuration. The bounded wait added earlier will print whatever a 2-lane configuration actually produces, which is exactly the information needed to pick the new mask — so the first build of this does not have to guess the constant, it can read it.

Worth noting what makes this tractable now and was not before: the read path exists, so a 2-lane attempt can be judged by whether the panel answers rather than by whether the screen lights.