storagebits · #agents

Testing the NUCLEO-F412ZG with Hubris and Humility

2026-10-18T18:05:17-04:00

The setup: a NUCLEO-F412ZG, an ST development board with a Cortex-M4, and Hubris, Oxide Computer’s embedded OS, which doesn’t support it out of the box. The kernel didn’t change by a single line. The board had to be added, three drivers adapted, the probe firmware updated, and Humility, the debugger, fixed twice.

The NUCLEO-F412ZG, powered through the USB of the ST-LINK, the probe built into the bottom of the board. Above it, the three user LEDs driven by the pong task.

# Hubris, Humility: what are they? Who’s behind them?

Hubris is a small OS written in Rust that Oxide uses in its servers for the service processor. Each task lives in its own memory regions, protected by the MPU. Tasks, their priorities and their access to peripherals are all declared at build time, in an app.toml. Humility reads the board’s memory through the probe and decodes it using the debug info: task states, stacks, registers, dumps.

The main author is Cliff Biffle. His talk at the Open Source Firmware Conference 2021 explains why Oxide wrote its own OS. This talk is truly amazing, what a beautiful piece of engineering!

“On Hubris and Humility: developing an OS for robustness in Rust”, Cliff L. Biffle, Open Source Firmware Conference 2021, 43 min.

Bryan Cantrill, co-founder and CTO of Oxide, and Adam Leventhal also deserve a mention among the big names behind Hubris. In his talk on debugging Docker in production, at DockerCon 2015, Cantrill makes a simple case: a crashed system doesn’t get rebooted before it has been debugged, either live or from an image of its memory, the core dump. Humility does both: humility tasks reads the running board, and humility dump saves its full state to a file you examine later, with the board unplugged.

“Running Aground: Debugging Docker in Production”, Bryan Cantrill, DockerCon 2015. Docker took down the original video in 2018; this is the copy Cantrill reposted.

# What had to be patched

I paste a mission prompt into Claude: get Hubris running on this board, plugged into the Mac over USB. The closest target in the repository is a Discovery board with an STM32F407. That same evening, Hubris runs at 100 MHz on the NUCLEO.

What was patched, layer by layer:

The kernel: nothing, not one line.
The board and the app: pure additions. A 2-line board file (the chip name for probe-rs), a chip description (1 MiB of flash, 256 KiB of RAM, peripheral addresses), and the app itself, with the 100 MHz clock configuration. The branch adds 621 lines, documentation included.
Three shared drivers, about 150 lines: stm32fx-rcc selects the F412 variant of the register library, user-leds drives the Nucleo’s LEDs on PB0, PB7 and PB14, and stm32fx-usart moves to USART3 (PD8/PD9), the ST-LINK’s virtual serial port, clocked at 50 MHz. The tricky part: two chip variants of the stm32f4 crate can’t coexist in the same task, and two of these drivers forced the F407 in their Cargo.toml. That choice became a feature. The Discovery still builds identically, rechecked at every step.
The probe: the ST-LINK firmware was too old and had to be updated.
Humility, the debugger: two fixes, detailed below. The first one repairs Humility against the latest Hubris, whatever the board. The second fixes a wrong measurement that shows up on this board.

# The mission, in slices

The prompt’s north star fits in one sentence: “humility tasks montre toutes les tâches Healthy sur la NUCLEO-F412ZG, les LEDs utilisateur clignotent sous le contrôle de pong, et ping écrit sur le port COM virtuel du ST-LINK.” [humility tasks shows every task Healthy on the NUCLEO-F412ZG, the user LEDs blink under pong’s control, and ping writes to the ST-LINK’s virtual COM port.] Five slices, each one has to end on hardware that works: preflight, reference build, first boot on the 16 MHz internal oscillator, UART, then 100 MHz.

First stop in slice 0. The board’s ST-LINK firmware is V2J25; probe-rs, which Humility builds on, requires J26. The prompt said to stop and ask. Claude offers to patch it automatically, I give it the zip downloaded from ST’s site, and that’s it.

Every task is Healthy. Claude can’t see the LEDs: it reads GPIOB’s output register several times in a row and sees it cycle between PB14, PB0 and nothing, in time with pong. The board runs at 100 MHz. Over 10 seconds, the Hubris clock counts 10,235 ms against 10.32 s of wall time, the gap being Humility’s connection latency.

Slice 4: from 16 MHz on the HSI to 100 MHz. Every constant is justified by a section of RM0402 in the code, and read back from the registers once flashed.

# Where it broke: SavedState

Right before the first boot, humility tasks fails: “expected structure SavedState not found”.

To understand why, you need to know how Humility reads a task. When the kernel interrupts a task, it stores its registers (r4 to r11, the stack pointer, and so on) in a structure, the save field of that task’s Task entry. Humility reads the task table straight from the board’s RAM. To know at which byte each register sits, it needs the layout of that structure, which it finds in the debug info (DWARF) of the kernel binary. And it looked the structure up by name: SavedState.

On October 9, four days earlier, Hubris PR #2704 introduced an Arch trait so the kernel can target several architectures. The ARMv7-M structure was renamed ArmMSavedState. Humility can no longer find it, and neither humility tasks nor reading a task’s registers works, whatever the board.

Two ways out. Claude first pins the branch right before that commit and keeps going, which is enough for this board. It’s in DECISIONS.md, and it unblocks things. Then the fix, on the savedstate-by-member branch of my Humility fork: instead of looking the type up by name, Humility takes the type of the save field of Task, which DWARF provides directly. The name can change; the field stays. Eleven lines added, four removed, in the two places that read this structure: humility tasks and reading the registers. The fix works with both the old kernel and the new one, tested on the board in both cases.

# The second fix: stackmargin showed zeros

humility stackmargin tells you, for each task, how much stack it has never used. The idea: when a task starts, the kernel fills its whole stack with a pattern, 0xbaddcafe. Humility reads the stack from the bottom and counts the words still intact: that’s the margin.

On the board, it shows a margin of 0 for jefe, the supervisor, with a 152-byte stack, while its app.toml declares 2200.

A Cortex-M4’s MPU only knows regions whose size is a power of 2. Since April (PR #2458), Hubris splits a task’s RAM into several contiguous regions to waste less: jefe’s 2560 bytes become 1 KiB + 1 KiB + 512 bytes instead of a single 4 KiB region. A stack can therefore span several regions. The kernel was fixed in May (PR #2492) to paint all of it. Humility, though, only read the region holding the top of the stack: here the last one, which holds only 152 bytes of stack, all of them used. Hence the margin of 0.

Read on the board (humility map, readmem). Humility only looked at the region holding the top of the stack. The kernel, for its part, paints the whole stack with 0xbaddcafe across all three regions.

           STACKBASE  STACKSIZE  MAXDEPTH  MARGIN
before jefe 0x20001c00       152       152       0
after  jefe 0x20001400      2200       272    1928

The fix, on the stackmargin-multi-region branch, walks down the contiguous regions that belong to the same task, as the kernel does, before measuring. Humility’s regression test confirmed it too: on a dump from a Sidecar, a real Oxide switch, the expected snapshot contained three bogus “MARGIN 0”. After the fix, every stack size matches, to the byte, the one declared in that Sidecar’s app.toml.

# Where I’m at

The nucleo-f412zg branch has five commits, with a DECISIONS.md and a README. It’s pinned before PR #2704; with the first fix, Humility also keeps up with the recent kernel. Both fixes are on my Humility fork. The kernel test suite on the F412, the optional slice 5, isn’t done. And “all Healthy” is only half true: ping deliberately faults every 100 replies from pong, as in Oxide’s demos.

A lead for later, not done yet: an I2C or SPI sensor behind a driver task. In Hubris, a driver is a task like any other: it alone has access to the peripheral’s registers, declared in the app.toml, and the other tasks talk to it through messages. The repository’s I2C driver targets the STM32H7 and G0 (plus a separate driver for the LPC55), SPI only the STM32H7 and the LPC55: nothing for the F4 family, it would have to be written for this board.

To be continued…

A 2020 watch, a complete firmware, three days

2026-10-16T00:00:00+00:00

I bought a LilyGO T-Watch 2020 six years ago, because it was cheap and ideal for testing and tinkering. Developing for it, on the other hand, took a real investment: C/C++, a framework to learn, some embedded know-how. It ended up in a corner, with other unfinished side projects.

With AI, all those brakes come off: within a few minutes, I had a complete firmware, written from scratch. Without AI, I would not have taken it out of the drawer: building anything on it took too much effort.

What devices are sleeping at your place? Imagine what you could make of them, in record time.

The post about the four days already told the story of the first flash. Here, I stay with the watch.

# What I wanted

An aviation chronograph dial and 7-segment displays in the style of “time circuits”. Then the weather, the buses and the iPhone notifications. The full list of apps is in the repository’s README.

My method comes down to two moves. I simply lay out the features I want. Then I immediately ask for a first end-to-end version, to get a short feedback loop. The result becomes convincing very quickly.

I steered the development in natural language, by stating intentions. Reading the generated code, I also learned a lot, about the language as much as about the frameworks and embedded concepts. The teaching value is huge!

# The beginning

First commit at 3:22 am. The mission prompt set two rules: back up the original 16 MB of flash before any write, and never touch the eFuses or the core voltage.

The backup nearly failed. At 921,600 baud, the USB port vanished in the middle of a read. The agent started over at 230,400 baud, in 1 MB blocks checked one by one, because above that the cable introduced errors. At 3:46 am, the full backup was verified and its SHA-256 hash recorded. At 3:48 am, the first verified flash on the watch.

I was a bit afraid of bricking the watch. I didn’t know where the fault came from; probably the cable. The agent took care of everything, down to trying a slow, chunked flash to work around that faulty cable (!). Its questions and its methods were relevant and precise, and that is what gave me confidence.

At 4:09 am, the aviator dial was on screen, with a second hand ticking at 8 Hz.

The aviator dial, captured on the real watch over the serial port. Capture already published in the previous post.

# The moment it broke

Plugged into USB, the watch held up. On battery, it crashed every single time after a few seconds. During the morning, I feared it was bricked, and I was pushing the agent to hurry before the next crash. I had to power it off and on again three times.

There were two causes, found one after the other. First, the processor’s watchdog stayed armed during light sleep. It is now suspended before going to sleep and re-armed on wake-up. Second, a press on the button froze the watch. The power chip’s interrupt line stays low until it is read, and the interrupt kept firing endlessly. GPIO interrupts are now disabled during sleep, and I saw that the watch held up very well.

# A decision: stop depending on the cable

The USB drop-outs kept coming back during flashes. There were two options: wait for the port to come back, or make the cable optional. The agent proposed a “workshop mode”: when the watch is powered over USB and Wi-Fi is configured, it stays connected, accepts over-the-air updates (OTA) and exposes a TCP console. Wi-Fi left on all the time is normally an anti-pattern. On mains power, the battery is not at stake, and the option can be turned off.

Since the cable kept failing, the agent switched to Wi-Fi updates (OTA) on its own. Magic! Later, the drop-outs also showed up while the processor was in its ROM bootloader: so the problem is hardware, not software. A safe mode followed: after three crashes in a row within the first minute, Bluetooth does not start, so the watch can always be repaired remotely.

The real watch on the diagnostic screen, the first screen flashed. Photo already published in the previous post.

# Where I stand

The cable problem (B2 in BLOCKERS.md) is still open: the micro-USB connection remains fragile. The double tap that wakes the screen is still a bit hit-and-miss, but it works. The wrist gesture to wake the watch still needs tuning. Battery life over a full day has not been measured yet.

What impressed me most is how quickly the agent flashed the watch on its own and showed me concrete results. I can’t wait to try it on other devices. In fact, I’ll soon be taking my micro:bits out again: I have an idea for a project!

Twelve minutes from idea to TV

2026-10-12T00:00:00+00:00

Thursday, October 8, 8:34 pm. I send a mission prompt to an agent. At 8:46 pm the time shows up, huge, on my TV, in a typeface the agent picked.

Four days later: six projects, 159 commits, some 240 messages from me, a good share of them under five words. “go T8”. “vas-y présente la tranche 3” [go ahead, show me slice 3]. “c’est fait, reteste” [done, test again].

The gap between intent and something real has become ridiculous. Twelve minutes for the Almanak app. Sixteen for Lukarne. The bottleneck now is me: my sleep, my cables, my quotas.

One row per project, Thursday 7 pm to Monday 7 pm. The number on the right counts commits. Hover the circles for exact times.

Each row is a project, each tick a commit. The hollow circle is my first message to the agent. The filled one, raised above the row, is the first concrete thing: the app running on the TV, the firmware on the watch, two-factor auth live on the homelab gateway. The grey bands are nights. There are commits in them. Not always because I was awake.

# Thursday night: three agents in 45 minutes

On the evening of the 8th, within 45 minutes:

8:34 pm: the Almanak mission goes out.
8:36 pm: I restart shot2cart, a project begun in September with another tool. “inspect project, it comes from an other AI conversation, inspect and we pursue”.
9:07 pm: I download from claude.ai the CLAUDE.md for my homelab, which I’m redesigning with an LLM. I had just written it with another instance while the first one was building Almanak.
9:19 pm: “je viens de créer CLAUDE.md pour toi, c’est parti !” [just created CLAUDE.md for you, let’s go!]

Nobody waits for anybody. While one agent compiles, I write the next one’s mission with a third.

# Almanak, the spark

The north star fit in one sentence:

A screen you want to leave on. When I walk past my TV, I know in 2 seconds: the time, the date, the weather now and later, and how many minutes until my next bus. It’s beautiful like a design object, not like an admin dashboard.

And one rule I kept for every project after it: no slice is done until the agent has seen it running on the Shield. It takes a capture over adb, reads it, critiques it. Legibility at 3 m, overscan, overflow. And for every visual slice, two variants; keep the better one.

Almanak on the Shield on the first night: the dark theme at 10:17 pm, the paper theme at 9:38 pm.

Seven slices between 8:46 and 10:32 pm. For the clock, the agent compared an editorial serif with a Swiss sans-serif and kept Fraunces, because at 3 m, Inter at weight 200 “starts to look like a wire”. It’s right.

Best bug: the bus schedule sync returned HTTP 441 on the Shield and worked fine from the Mac. The agent dug through dumpsys connectivity and found the VPN running on the Shield: the STM site (Montreal’s transit agency) rejects VPN exit IPs. It switched to a mirror of the same feed, byte for byte, and documented split tunnelling in the README. The prompt never mentioned a VPN.

To be honest: my definition of “done” asked for 24 hours of screensaver without a memory leak. The actual measurement covers about an hour. Done, then. In quotes.

# Lukarne, my favourite

Next day, 5:58 pm, second mission. A window on the world: spin a globe with the remote, dive into a region, watch a wall of live webcams, zap from one camera to the next. The sentence that matters in the prompt:

If an interaction isn’t smooth with the remote, it has failed, even if it is technically correct.

And a quieter one that explains why it all went so fast: “I’ve already built another Google TV app with you on this same Shield. Before anything else, find in my environment […] how we connect to it […]. Reuse what works.” Every project leaves behind an adb address, a deploy script, a DECISIONS.md format. The next one starts from there.

6:14 pm: first live stream fullscreen on the TV, HLS and YouTube, left/right to switch. 16 minutes. By 8:54 pm, V1 was there: an OpenGL day/night globe at 60 frames per second (3 ms per frame), 1,640 cameras in 34 regions (19 more since Monday), a 3×3 wall whose focused tile goes live within two seconds, and a “surprise me” button that sends you somewhere it’s daytime. Two 22-minute soak tests, zero crashes.

Lukarne: Europe at night, Venice selected, 16 live webcams, St Mark’s Basin in the corner.

The easiest way is to watch it run:

78 seconds on the Shield, no sound: the globe over to California, the dive into Pollock Pines, the wall of live highway cameras, fullscreen, next camera, climbing back out, then “surprise me”.

The agent shot this video on Monday evening, for this post. It drove the Shield with a virtual remote (adb shell input keyevent) and recorded the screen with screenrecord. The first take was in French, over Europe, where it was 1 am: everything was black. It reshot in English, over California, in daylight.

A place’s wall, then fullscreen. Left/right jumps to the neighbouring camera without going back to the wall.

“surprise me” landed on this, mid-shoot. Explore.org’s aurora camera in Fairbanks, 3:09 pm local time: probably a replay of the night between two live shots.

Then I used it, and complained. Saturday, 11:20 pm: “je suis assez surpris du peu de flux video , bcp d’images fixes qui apportent peu” [I’m pretty surprised how few video feeds there are, lots of stills that don’t add much]. 11:34 pm, commit: live first, stills optional. Fourteen minutes.

Overnight from Sunday to Monday, at 12:08 am: “parfait , je vais dormir là” [perfect, I’m off to bed]. And two requests: “les vignettes pixelisée : on peut trouver un moyen de charger ne serait ce qu 1 frame propre plutôt que la vignette degeueu ?” [the pixelated thumbnails: can we find a way to load even 1 clean frame instead of the gross thumbnail?], and for the last zoom level, an “effet plongeant genre zoom google earth” [diving effect, like a Google Earth zoom]. At 12:26 am, the thumbnails are real frames grabbed on the device during playback. At 12:48 am, you dive into a place with a ×30 zoom, the lights come on, and you climb back out. At 12:59 am, a 10-minute soak test on top of all that. I was asleep.

Monday afternoon, I asked for a demo mode, so the TV travels on its own. 41 minutes later:

Demo mode: random cameras where it's daytime, never the same region twice, with a fade through a title card in between.

Settings, one remote away: language (English, French, Spanish), units, stills or live only, corner player, demo time per camera, and credits for every source.

Still missing: live coverage outside North America is thin. On day one, the agent burned the whole YouTube search quota while two bugs hid the results. Now a public repository republishes the live catalogue every day, and the app downloads it behind its logo. And there’s a bug I can’t always reproduce: a YouTube webcam that freezes when you chain two of them. I reported it at 3:46 pm, a fix landed at 4:31 pm. Still to be checked on the TV.

# DropboksTV, one Sunday afternoon

A Dropbox client for the TV, built for the couch: photos and videos first, read-only, no backend. Sunday, 3:18 pm: “read prompt-dropbokstv-claude-code.md and go”. 4:02 pm: phone pairing works on the TV. 6:28 pm: eighth and final slice, with search, offline mode, the screensaver and 128 translated strings per language.

The trickiest part is pairing. Signing in to Dropbox from a TV, without a keyboard, should be a code you type on your phone. Except Dropbox’s API doesn’t offer that flow. The agent checked the SDK and wrote it down in its decision log. So it built something else:

the TV shows a QR code pointing to a page served by the TV itself, on the home Wi-Fi;
the phone opens that page, which sends it to authorise DropboksTV on Dropbox;
Dropbox shows a code, you paste it on the phone, and the TV exchanges it for a token (PKCE: the secret never leaves the TV, no backend);
the session expires after 10 minutes and the TV renews the QR on its own. A typo doesn’t burn the session. And if the phone can’t see the TV, a button switches to the direct Dropbox link.

The TV’s little HTTP server (150 lines) was lifted from Almanak.

Pairing

And then there’s the viewer. In three slices, between 4:25 and 6:07 pm, the agent wrote a viewer that opens pretty much everything:

photos, with ×2 and ×3 zoom you pan with the arrows, and a slideshow with fades or Ken Burns;
videos: H.264, HEVC up to 4K, resume where you left off, .srt subtitles;
PDFs, page by page, with zoom;
Word and PowerPoint, which Dropbox converts to PDF, and spreadsheets, scaled to 170% so they’re readable from the couch;
formatted Markdown, code with line numbers and syntax colours;
music, MP3 and FLAC, with cover art;
and for everything else, a QR code that sends the file to your phone.

The video player took 16 minutes. Documents, music and send-to-phone, 27. Insane.

In order: zoomed photo, video, PDF, spreadsheet, Markdown, code, music, send to phone (QR blurred). The agent made the test files itself and dropped them into a dedicated Dropbox folder. Screenshots taken during development, in French.

# Kronograf, or the return of hardware

Monday, 3:19 am, a 2020 LilyGO T-Watch plugged in over USB, and a mission: an aviation chronograph face, plus 7-segment “time circuit” displays. At 3:48 am, first verified flash on the watch, after a full backup of the original flash. That was rule no. 2 in the prompt: nothing gets written to the watch without a safety net. The cable dropped out during the first backup, and the agent refused to flash until it had a complete one.

The real watch, Monday at 6:11 pm: the diagnostics screen, the first one flashed. Green: chips detected; grey: what the V1 doesn’t have.

Left, a capture of the real watch over the serial port. Right, the “time circuits” face in the simulator, which compiles exactly the same UI.

This is where the delay goes back up, and it’s not the agent’s fault. It’s no longer code against an emulator, it’s a tired micro-USB connector. My messages that morning: “elle plante systematiquement au bout de quelque secondes” [it crashes every time after a few seconds], “va vite sinon elle plante” [hurry or it crashes], “j’espère qu’elle n’est pas brikée” [I hope it isn’t bricked]. The problem was light sleep, with an interrupt storm when the button woke it up. At 11:17 am: “ce semble tres bien tenir maintenant” [seems to be holding up fine now]. At 11:18 am: “go pour la météo” [go for the weather]. At 11:31 am, the watch shows the weather and the next bus in real time.

Tonight it has 33 commits, alarms, a stopwatch, a step counter, Wi-Fi updates and iPhone notifications over Bluetooth, which still don’t hold the connection well. The double tap is “un peu alétoire , comme le mouvement du poignet … mais ca passe” [a bit random, like the wrist gesture… but it’ll do].

# The homelab, and the shot2cart app

The longest job doesn’t show. Sixty commits and 276 messages to rebuild the home server from the ground up: an inventory, backups with real restore tests, remote access with passkeys for the family, alerts, a home page with an animated map of the infrastructure and another of the local network.

My favourite message of the whole run, on the first night: “ignore cet audit ; je ne compte que sur toi maintenant” [ignore that audit; I’m counting only on you now]. The most honest one, on Sunday: “Nest dans Home Assistant c’est payant ??!!” [Nest in Home Assistant costs money??!!]. Yep, $5, thanks Google.

shot2cart is the counter-example. The app’s pitch: a photo of a shopping list, and the cart opens at the store in under 15 seconds, without typing a thing. First attempt in September, with another tool and a spec-driven workflow: three days, two commits. On Thursday night, I restarted it on that base. Sunday, 8:13 pm, I archived everything: “forget about previous session, I archived the previous project and to start a new one with better prompt”. 11:01 pm: photo, analysis by a vision model, store links item by item. 12:04 am: “I’m going to sleep […] please go as far as you can in full autonomy”. By morning there was streaming detection, review cards and automated test flows.

Same idea, same developer. The difference between September and this Sunday is the quality of the models, Opus 5.5 in particular. A beast!!!

# The delay, measured

Type	What I asked for	Time to commit
FEAT	“ajoute la langue espagnole à l’application” [add Spanish to the app] (Lukarne)	5 min
BUG	Neighbours 2,500 km away: “je suis sur Londres mais la Patagonie est listée” [I’m on London but Patagonia is listed]	5 min
BUG	The logo that only shows for half a second at launch	2 min
FEAT	More live, fewer stills	14 min
FEAT	The “Google Earth style” diving zoom, requested before going to bed	40 min
FEAT	A demo mode that chains random webcams	41 min
BUG	The recent video that always opens a video from 2012 (DropboksTV)	2 min
FEAT	“go pour la météo” [go for the weather] on the watch	13 min

What doesn’t go faster: a micro-USB cable that comes loose, a YouTube quota that resets at 3 am, an API key the provider’s portal rejects, my subscription. Sunday, 4:46 pm: “il me reste que 10% pour notre session en cours” [I’ve only got 10% left for our current session]. And my sleep, which I ended up building into the plan: “je vais dormir la , avance au maximum de ton autonomie” [I’m going to sleep, go as far as you can on your own].

What I don’t know yet: how many of these six projects I’ll still be using in a month. Time to see it through now, ship what can be shipped, feed other agents.

This post was made the same way. One message from me on Monday late afternoon, in a jumble, typos included: “génère moi un article complet […] mettre en avant le délai ridicule que c’est devenu entre l’intention et le concret” [write me a complete article […] highlight how ridiculous the gap between intent and reality has become]. Lay it all out, transparently and honestly. The blog itself was online 45 minutes after my first message, Monday at 4 am.

Anyway, have a good weekend!