cross-build.txt

Cross-build evidence — image/cross-build
========================================

Host:     solomon, x86_64-linux, 32 cores, 125 GiB RAM
Nix:      nix (Nix) 2.35.2
Date:     2026-09-20
nixpkgs:  github:NixOS/nixpkgs/nixos-unstable (see flake.lock for the pin)

Task 1.1 — the cross toolchain resolves
---------------------------------------
$ nix build --no-link --print-out-paths .#checks.x86_64-linux.cross-hello
/nix/store/8dlbjqlyh82xc1h02klzrgfbi9pw2sv4-hello-riscv64-unknown-linux-gnu-2.12.3

$ file /nix/store/8dlbjqlyh82xc1h02klzrgfbi9pw2sv4-hello-riscv64-unknown-linux-gnu-2.12.3/bin/hello
ELF 64-bit LSB pie executable, UCB RISC-V, RVC, double-float ABI, version 1 (SYSV), dynamically linked, interpreter /nix/store/syg6xw4x296w82267qnvrd3nn0cp6j31-glibc-riscv64-unknown-linux-gnu-2.42-84/lib/ld-linux-riscv64-lp64d.so.1, for GNU/Linux 4.15.0, not stripped

Task 1.2 — what a cross-build costs on this host
------------------------------------------------
cross-hello, cold (toolchain fetched from cache.nixos.org): 1m15s wall.
Almost all of that was substituting the riscv64 cross toolchain, which is
itself an x86_64 package and therefore IS cached. What is not cached is
anything built FOR riscv64.

Warm rebuild (everything substituted or already built): 19s wall.

Derivations that would not cross-compile
-----------------------------------------
(filled in by task 2.3)

Cache coverage — measured, and it corrects an assumption
---------------------------------------------------------
The proposal assumed riscv64 has no binary cache at all. That is true of
NATIVE riscv64-linux builds and false of CROSS-compiled ones, because a
pkgsCross.riscv64 derivation is built on x86_64 and upstream Hydra builds many
of them.

Measured on the k230 closure build, 2026-09-20:

  substituted from cache.nixos.org : 636 paths
    of which riscv64 outputs       : 174
  built locally                    : 164

So the cross path gets substantial cache coverage. Examples of riscv64 paths
that were substituted, not compiled:

  acl-riscv64-unknown-linux-gnu-2.4.0-man
  attr-riscv64-unknown-linux-gnu-2.6.0-man
  bash-interactive-riscv64-unknown-linux-gnu-5.3p15-man

image/cross-build's cache requirement was rewritten to say this rather than
the thing we guessed.

Stage-1 seam (task 4.1)
-----------------------
$ nix eval --json .#stage1
  vendored = true, builtFromSource = false
$ grep -rniE 'u-?boot|opensbi' flake.nix nix/
  no matches outside explanatory comments
Nothing in the flake compiles U-Boot or OpenSBI.

Emulation fallback — available in principle, not enabled in practice
---------------------------------------------------------------------
Nothing in this build needed emulation, and nothing COULD have been emulated:

  $ nix config show extra-platforms
  i686-linux x86_64-v1-linux x86_64-v2-linux x86_64-v3-linux

riscv64-linux is absent, so Nix will not build riscv64-linux derivations
natively on this host regardless of binfmt. The OS-level binfmt handler IS
registered -- the cross-built hello runs directly on the build host -- but
that is a property of the OS, not of Nix.

So image/cross-build's "emulation remains available as a fallback" is true
only after a one-line change:

  nix.settings.extra-platforms = [ "riscv64-linux" ];   # NixOS
  extra-platforms = riscv64-linux                       # /etc/nix/nix.conf

That change has NOT been made, because nothing has needed it. Recorded here
so the fallback is understood as one config line away rather than as
something already in place.

A minimal NixOS closure needs TWO cross toolchains
---------------------------------------------------
Unexpected, and the largest single cost in the first build.

The glibc cross toolchain (riscv64-unknown-linux-gnu-gcc 15.3.0) substitutes
from cache.nixos.org in seconds. But NixOS's initrd wants a statically linked
busybox, which is built against musl -- and no musl riscv64 toolchain is
cached, so the whole bootstrap is compiled locally:

  riscv64-unknown-linux-musl-nolibc-gcc-15.3.0
  musl-static-riscv64-unknown-linux-musl-1.2.6
  riscv64-unknown-linux-musl-gcc-15.3.0
  riscv64-unknown-linux-musl-binutils-wrapper-2.46

That is a second GCC bootstrap, compiled in parallel with the kernel, and it
is why "minimal NixOS" is not a short build here even with good cache
coverage on the glibc side.

It is a one-time cost per nixpkgs pin -- the results land in the local store
and subsequent builds reuse them. Recorded so the first build's duration is
not mistaken for the steady-state cost, and so anyone trimming the closure
knows the initrd is where the expensive part hides.

A note on parallelism: -j 32 was the wrong flag
------------------------------------------------
The closure build was launched with `nix build -j 32`. That is max-jobs --
up to 32 concurrent DERIVATIONS -- and each derivation separately defaults
to using every core. Measured during the build:

  load average : 101
  compiler processes : 156
  cores : 32

So the machine was roughly 3x oversubscribed. Work still completes and it
was left running rather than restarted (45 minutes of GCC bootstrap already
invested), but it is not the fast configuration.

For a rebuild, prefer letting nix pick, or set both explicitly so the
product is bounded:

  nix build -j 8 --cores 4     # 8 derivations x 4 cores = 32

Recorded because "the build is slow" and "the build is oversubscribed" look
identical from outside, and the next person timing this should not inherit
a bad number as the baseline.

TASK 2.2 — the closure built
-----------------------------
$ nix build .#nixosConfigurations.k230.config.system.build.toplevel
/nix/store/lzg30bab6kqkgsa54jaql1ixfxgizxv3-nixos-system-nixos-26.11.20260919.20b1ddd

  EXIT=0   ELAPSED=4413s (73m33s)   2026-09-20

Final tallies for the hardware closure:

  derivations built locally : 353
  paths substituted         : ~970
  closure size              : 1.2 GiB
  store paths in closure    : 474

Remember the timing is from an oversubscribed run (-j 32, load 101). It is
an upper bound, not a clean baseline.

TASK 2.3 — nothing needed emulation
------------------------------------
The list is empty, and stated explicitly as the task requires: no derivation
refused to cross-compile, and none could have been emulated anyway, because
nix's extra-platforms does not include riscv64-linux on this host. Every one
of the 353 local builds was a cross build or a native host tool.

Minimality (system/nixos-config)
---------------------------------
Checked against the built closure rather than asserted:

  NetworkManager  0 paths       xserver  0 paths
  ModemManager    0 paths       mesa     0 paths

The QEMU variant IS heavier -- netboot-minimal imports installation-device,
which brings NetworkManager, ModemManager and a python3 environment. That is
the test harness, not the product, and it never reaches the board. It is also
why tools/qemu-k230.sh gives the guest 6G: a 1.2 GiB closure has to be
resident in a ramdisk there, where on hardware it is an ext4 root on the card.