Testing the NUCLEO-F412ZG with Hubris and Humility

The setup: a NUCLEO-F412ZG, an ST development board with a Cortex-M4, and Hubris, Oxide Computer’s embedded OS, which doesn’t support it out of the box. The kernel didn’t change by a single line. The board had to be added, three drivers adapted, the probe firmware updated, and Humility, the debugger, fixed twice.

# Hubris, Humility: what are they? Who’s behind them?

Hubris is a small OS written in Rust that Oxide uses in its servers for the service processor. Each task lives in its own memory regions, protected by the MPU. Tasks, their priorities and their access to peripherals are all declared at build time, in an app.toml. Humility reads the board’s memory through the probe and decodes it using the debug info: task states, stacks, registers, dumps.

The main author is Cliff Biffle. His talk at the Open Source Firmware Conference 2021 explains why Oxide wrote its own OS. This talk is truly amazing, what a beautiful piece of engineering!

Bryan Cantrill, co-founder and CTO of Oxide, and Adam Leventhal also deserve a mention among the big names behind Hubris. In his talk on debugging Docker in production, at DockerCon 2015, Cantrill makes a simple case: a crashed system doesn’t get rebooted before it has been debugged, either live or from an image of its memory, the core dump. Humility does both: humility tasks reads the running board, and humility dump saves its full state to a file you examine later, with the board unplugged.

# What had to be patched

I paste a mission prompt into Claude: get Hubris running on this board, plugged into the Mac over USB. The closest target in the repository is a Discovery board with an STM32F407. That same evening, Hubris runs at 100 MHz on the NUCLEO.

What was patched, layer by layer:

• The kernel: nothing, not one line.
• The board and the app: pure additions. A 2-line board file (the chip name for probe-rs), a chip description (1 MiB of flash, 256 KiB of RAM, peripheral addresses), and the app itself, with the 100 MHz clock configuration. The branch adds 621 lines, documentation included.
• Three shared drivers, about 150 lines: stm32fx-rcc selects the F412 variant of the register library, user-leds drives the Nucleo’s LEDs on PB0, PB7 and PB14, and stm32fx-usart moves to USART3 (PD8/PD9), the ST-LINK’s virtual serial port, clocked at 50 MHz. The tricky part: two chip variants of the stm32f4 crate can’t coexist in the same task, and two of these drivers forced the F407 in their Cargo.toml. That choice became a feature. The Discovery still builds identically, rechecked at every step.
• The probe: the ST-LINK firmware was too old and had to be updated.
• Humility, the debugger: two fixes, detailed below. The first one repairs Humility against the latest Hubris, whatever the board. The second fixes a wrong measurement that shows up on this board.

# The mission, in slices

The prompt’s north star fits in one sentence: “humility tasks montre toutes les tâches Healthy sur la NUCLEO-F412ZG, les LEDs utilisateur clignotent sous le contrôle de pong, et ping écrit sur le port COM virtuel du ST-LINK.” [humility tasks shows every task Healthy on the NUCLEO-F412ZG, the user LEDs blink under pong’s control, and ping writes to the ST-LINK’s virtual COM port.] Five slices, each one has to end on hardware that works: preflight, reference build, first boot on the 16 MHz internal oscillator, UART, then 100 MHz.

First stop in slice 0. The board’s ST-LINK firmware is V2J25; probe-rs, which Humility builds on, requires J26. The prompt said to stop and ask. Claude offers to patch it automatically, I give it the zip downloaded from ST’s site, and that’s it.

Every task is Healthy. Claude can’t see the LEDs: it reads GPIOB’s output register several times in a row and sees it cycle between PB14, PB0 and nothing, in time with pong. The board runs at 100 MHz. Over 10 seconds, the Hubris clock counts 10,235 ms against 10.32 s of wall time, the gap being Humility’s connection latency.

# Where it broke: SavedState

Right before the first boot, humility tasks fails: “expected structure SavedState not found”.

To understand why, you need to know how Humility reads a task. When the kernel interrupts a task, it stores its registers (r4 to r11, the stack pointer, and so on) in a structure, the save field of that task’s Task entry. Humility reads the task table straight from the board’s RAM. To know at which byte each register sits, it needs the layout of that structure, which it finds in the debug info (DWARF) of the kernel binary. And it looked the structure up by name: SavedState.

On October 9, four days earlier, Hubris PR #2704 introduced an Arch trait so the kernel can target several architectures. The ARMv7-M structure was renamed ArmMSavedState. Humility can no longer find it, and neither humility tasks nor reading a task’s registers works, whatever the board.

Two ways out. Claude first pins the branch right before that commit and keeps going, which is enough for this board. It’s in DECISIONS.md, and it unblocks things. Then the fix, on the savedstate-by-member branch of my Humility fork: instead of looking the type up by name, Humility takes the type of the save field of Task, which DWARF provides directly. The name can change; the field stays. Eleven lines added, four removed, in the two places that read this structure: humility tasks and reading the registers. The fix works with both the old kernel and the new one, tested on the board in both cases.

# The second fix: stackmargin showed zeros

humility stackmargin tells you, for each task, how much stack it has never used. The idea: when a task starts, the kernel fills its whole stack with a pattern, 0xbaddcafe. Humility reads the stack from the bottom and counts the words still intact: that’s the margin.

On the board, it shows a margin of 0 for jefe, the supervisor, with a 152-byte stack, while its app.toml declares 2200.

A Cortex-M4’s MPU only knows regions whose size is a power of 2. Since April (PR #2458), Hubris splits a task’s RAM into several contiguous regions to waste less: jefe’s 2560 bytes become 1 KiB + 1 KiB + 512 bytes instead of a single 4 KiB region. A stack can therefore span several regions. The kernel was fixed in May (PR #2492) to paint all of it. Humility, though, only read the region holding the top of the stack: here the last one, which holds only 152 bytes of stack, all of them used. Hence the margin of 0.

           STACKBASE  STACKSIZE  MAXDEPTH  MARGIN
before jefe 0x20001c00       152       152       0
after  jefe 0x20001400      2200       272    1928

The fix, on the stackmargin-multi-region branch, walks down the contiguous regions that belong to the same task, as the kernel does, before measuring. Humility’s regression test confirmed it too: on a dump from a Sidecar, a real Oxide switch, the expected snapshot contained three bogus “MARGIN 0”. After the fix, every stack size matches, to the byte, the one declared in that Sidecar’s app.toml.

# Where I’m at

The nucleo-f412zg branch has five commits, with a DECISIONS.md and a README. It’s pinned before PR #2704; with the first fix, Humility also keeps up with the recent kernel. Both fixes are on my Humility fork. The kernel test suite on the F412, the optional slice 5, isn’t done. And “all Healthy” is only half true: ping deliberately faults every 100 replies from pong, as in Oxide’s demos.

A lead for later, not done yet: an I2C or SPI sensor behind a driver task. In Hubris, a driver is a task like any other: it alone has access to the peripheral’s registers, declared in the app.toml, and the other tasks talk to it through messages. The repository’s I2C driver targets the STM32H7 and G0 (plus a separate driver for the LPC55), SPI only the STM32H7 and the LPC55: nothing for the F4 family, it would have to be written for this board.

To be continued…