ThemeliOS

ThemeliOS (from Greek θεμέλιο — “foundation”) is an experimental capability-based microkernel operating system written in Rust. It is designed from the ground up to do one thing well: run container workloads securely.

What is ThemeliOS?

ThemeliOS is a from-scratch kernel — it does not use or build on top of Linux. It implements its own memory management, process scheduling, inter-process communication, and security model.

The long-term vision is a minimal, immutable OS that:

• Boots on virtual machines and bare metal
• Runs OCI-compatible container images
• Serves as a Kubernetes/K3s worker node
• Provides hardware-enforced isolation between containers via capabilities
• Has no SSH, no shell, and no way to “log in” — all management is via API

Why build a new kernel?

Existing container OSes (Bottlerocket, Talos Linux, Flatcar) all use the Linux kernel with a stripped-down userspace. This is practical, but it inherits Linux’s security model — namespaces and cgroups are opt-in isolation bolted onto a kernel designed for general-purpose computing.

ThemeliOS takes the opposite approach: isolation is the default. The capability-based security model means a process has zero access to anything unless explicitly granted. There’s nothing to escape from because there’s no ambient authority to escalate to.

Project status

ThemeliOS is in early development. See the Milestones page for the current roadmap.

License

MIT — Copyright (c) 2026 Rudi MK

Development Setup

This guide walks through setting up a development environment for ThemeliOS on macOS or Linux.

Prerequisites

1. Rust nightly toolchain

ThemeliOS requires Rust nightly because the kernel uses unstable features (#![no_std], #![no_main], inline assembly, custom allocators).

The project pins the exact toolchain via rust-toolchain.toml, so you just need rustup installed — it will automatically download the correct nightly version.

Install rustup (if you don’t have it):

curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh

After cloning the repo, the first cargo command will automatically install the pinned nightly toolchain plus the bare-metal targets (x86_64-unknown-none, aarch64-unknown-none).

You can verify with:

rustup show

You should see a nightly toolchain with the x86_64-unknown-none and aarch64-unknown-none targets listed.

2. QEMU

QEMU emulates the hardware that ThemeliOS runs on. You need qemu-system-x86_64 for the primary amd64 target and optionally qemu-system-aarch64 for arm64.

macOS (Homebrew):

brew install qemu

This installs all QEMU system emulators.

3. xorriso

xorriso creates bootable ISO images. The build pipeline uses it to package the kernel with the Limine bootloader into a hybrid BIOS+UEFI ISO.

macOS (Homebrew):

brew install xorriso

Ubuntu/Debian:

sudo apt install xorriso

Fedora:

sudo dnf install xorriso

4. C compiler (for Limine CLI tool)

The first cargo xtask run downloads and builds the Limine bootloader’s CLI tool, which is a small C program. This requires a C compiler.

• macOS: Xcode Command Line Tools (xcode-select --install)
• Linux: gcc or clang (usually pre-installed)

Ubuntu/Debian:

sudo apt install qemu-system-x86 qemu-system-arm

Fedora:

sudo dnf install qemu-system-x86 qemu-system-aarch64

Arch Linux:

sudo pacman -S qemu-full

Verify installation:

qemu-system-x86_64 --version
qemu-system-aarch64 --version

3. mdbook (optional, for building documentation)

cargo install mdbook

5. Filesystem image tools (squashfs, e2fsprogs)

Phase 3 (storage) builds the disk images ThemeliOS boots from using two host tools, invoked by cargo xtask image:

• mksquashfs (from squashfs-tools) — builds the compressed, read-only SquashFS root image.
• mkfs.ext2 (from e2fsprogs) — formats the read-write ext2 data volume.

macOS (Homebrew):

brew install squashfs e2fsprogs

Note: e2fsprogs is keg-only on macOS (Apple ships conflicting versions), so Homebrew does not symlink mkfs.ext2 onto your PATH. xtask handles this automatically — it looks for mkfs.ext2 on PATH and falls back to /opt/homebrew/opt/e2fsprogs/sbin/mkfs.ext2 (and the Intel /usr/local/opt/... location). You do not need to edit your PATH.

Ubuntu/Debian:

sudo apt install squashfs-tools e2fsprogs

Fedora:

sudo dnf install squashfs-tools e2fsprogs

Arch Linux:

sudo pacman -S squashfs-tools e2fsprogs

Installing everything at once (macOS)

The repo ships a Brewfile that declares every macOS host dependency (QEMU, xorriso, squashfs, e2fsprogs). From the repo root:

brew bundle

Building and running

All build and run commands go through the xtask tool. You never need to invoke cargo build for the kernel directly.

Build the kernel

cargo xtask build

This cross-compiles the kernel for x86_64-unknown-none (the default target).

For arm64:

cargo xtask build --arch arm64

Run in QEMU

cargo xtask run

This builds the kernel, creates a bootable ISO, and launches it in QEMU in headless mode — serial output is piped to your terminal, but no graphical window opens. Press Ctrl+A, X to exit QEMU.

For arm64 (not yet implemented):

cargo xtask run --arch arm64

Build ISO only (without launching QEMU)

cargo xtask iso

This builds the kernel and creates a bootable ISO at target/themelios.iso without launching QEMU. Useful when you want to run QEMU manually with custom flags.

Run with QEMU display window

To see the QEMU graphical window (shows the Limine bootloader screen and any framebuffer output):

cargo xtask run --display

This does everything cargo xtask run does but opens a QEMU window instead of running headless. Serial output still goes to your terminal.

Build documentation

cargo xtask docs

This builds both the mdbook (to docs/book/) and the rustdoc API docs.

Shorthand alias

The workspace defines a cargo xt alias, so these also work:

cargo xt build
cargo xt run
cargo xt docs

Project layout

themelios/
├── kernel/          # The kernel crate (#![no_std], bare-metal)
│   └── src/
│       ├── main.rs  # Kernel entry point, module declarations
│       ├── arch/    # Architecture-specific (x86_64, aarch64)
│       ├── mm/      # Memory management
│       ├── sched/   # Scheduler
│       ├── cap/     # Capability system
│       ├── ipc/     # Inter-process communication
│       ├── drivers/ # Device drivers (VirtIO, serial, etc.)
│       ├── fs/      # Filesystem
│       └── net/     # Networking
├── xtask/           # Build tooling (runs on host)
├── docs/            # mdbook documentation
├── .cargo/          # Cargo configuration
└── CLAUDE.md        # Project documentation for AI assistants

IDE setup

VS Code

Install the rust-analyzer extension. It should pick up the workspace configuration automatically.

If rust-analyzer struggles with the #![no_std] kernel crate, you may need to add this to .vscode/settings.json:

{
    "rust-analyzer.cargo.target": "x86_64-unknown-none",
    "rust-analyzer.cargo.buildScripts.enable": true
}

Other editors

Any editor with rust-analyzer LSP support should work. The key setting is ensuring the target is set to x86_64-unknown-none for the kernel crate.

Troubleshooting

“can’t find crate for core”

This means the bare-metal target isn’t installed. Run:

rustup target add x86_64-unknown-none aarch64-unknown-none

Or let rust-toolchain.toml handle it by running any cargo command in the project.

“error: -Zbuild-std is unstable”

You need to be on the nightly toolchain. Check with rustup show — the project’s rust-toolchain.toml should select nightly automatically.

QEMU not found

Make sure QEMU is installed and on your $PATH. See the QEMU installation section above.

Bootloader

ThemeliOS uses the Limine bootloader. This page explains why, how it works, and how it fits into the build pipeline.

Why Limine?

We evaluated several options for booting ThemeliOS:

Limine was chosen because:

1. Multi-architecture: Supports x86_64 and aarch64 (and RISC-V, LoongArch). We need both for our cloud targets.
2. Multi-firmware: Works on both BIOS (legacy) and UEFI (modern). Cloud platforms use UEFI; QEMU defaults to BIOS.
3. Higher-half kernel: Limine sets up page tables that map our kernel at 0xffffffff80000000, which is the standard layout for 64-bit kernels.
4. Clean protocol: The Limine boot protocol gives us a memory map, framebuffer, and other boot info without writing any assembly.
5. Active maintenance: Regular releases, good documentation.

Cloud compatibility

Limine’s UEFI support means ThemeliOS can boot on:

• AWS EC2 (Nitro): UEFI supported on most instance types
• GCP Compute Engine: UEFI supported
• Azure Gen2 VMs: UEFI
• Bare metal: UEFI is standard on modern server hardware
• QEMU/KVM: Both BIOS (default) and UEFI (via OVMF)

The same kernel binary works on all platforms — only the bootloader firmware interface differs, and Limine handles that.

How it works

Boot sequence

1. Firmware (BIOS or UEFI) loads the Limine bootloader from the boot media
2. Limine reads limine.conf to find the kernel path and boot protocol
3. Limine loads the kernel ELF into memory at the addresses specified in the linker script
4. Limine sets up:
   • 64-bit long mode (x86_64) or EL1 (aarch64)
   • 4-level page tables with identity + higher-half mappings
   • A valid stack
5. Limine scans the kernel’s .requests ELF section for boot protocol requests
6. Limine fills in the requests (memory map, framebuffer, etc.)
7. Limine jumps to the kernel entry point (kmain)

Boot protocol requests

The kernel communicates with Limine through static data structures placed in a special ELF section. These are “requests” — the kernel declares what boot information it needs, and Limine fills in the responses.

#![allow(unused)]
fn main() {
// Placed in the .requests ELF section via the linker script
#[used]
#[link_section = ".requests"]
static BASE_REVISION: BaseRevision = BaseRevision::new();
}

The linker script places these between start/end markers so Limine knows where to scan:

.data : {
    ...
    KEEP(*(.requests_start_marker))
    KEEP(*(.requests))
    KEEP(*(.requests_end_marker))
}

Configuration file

limine.conf (in the project root) uses the v8 format:

timeout: 0

/ThemeliOS
    protocol: limine
    kernel_path: boot():/boot/themelios

• timeout: 0 — boot immediately without showing a menu
• /ThemeliOS — defines a boot entry
• protocol: limine — use the Limine protocol (not Linux or Multiboot)
• kernel_path: boot():/boot/themelios — load the kernel from the boot volume

Linker script

The linker script (kernel/linker-x86_64.ld) controls the kernel’s memory layout:

• Entry point: ENTRY(kmain) — tells the ELF where execution begins
• Load address: 0xffffffff80000000 — the higher-half virtual address
• Sections: .text (code), .rodata (constants), .data (mutable data + Limine requests), .bss (zeroed data)

The kernel must be compiled with -Crelocation-model=static to produce a non-PIE executable with fixed addresses that match the linker script.

Build pipeline

The cargo xtask run command handles the full pipeline:

1. Cross-compile the kernel for x86_64-unknown-none
2. Download Limine (one-time: git clone of the v8.x-binary branch to target/limine/)
3. Build Limine CLI (one-time: make compiles limine.c)
4. Create ISO via xorriso:
   • Copies kernel, Limine files, and limine.conf into an ISO directory structure
   • Creates a hybrid BIOS+UEFI bootable ISO
   • Installs BIOS boot sectors via limine bios-install
5. Launch QEMU with the ISO attached as a CD-ROM

Limine version

• Bootloader: v8.x (binary distribution from v8.x-binary branch)
• Rust crate: limine = "0.5" (boot protocol structures)

The bootloader binaries are cached in target/limine/ and not committed to git.

Architecture Overview

ThemeliOS is a capability-based microkernel. This page explains the high-level design and the reasoning behind key architectural decisions.

Microkernel vs monolithic

In a monolithic kernel (like Linux), drivers, filesystems, and networking all run inside the kernel with full hardware access. A bug in any driver can crash or compromise the entire system.

In a microkernel, only the absolute minimum runs in kernel space:

Everything else runs as isolated userspace processes that communicate via IPC. A buggy driver crashes its own process, not the kernel.

Why microkernel for ThemeliOS? Since we’re building an OS specifically for running untrusted container workloads, minimizing the trusted computing base (the code that can compromise the whole system) is critical. The smaller the kernel, the smaller the attack surface.

Capability-based security

ThemeliOS does not use Linux-style permissions (UID/GID, filesystem permissions) or Linux-style isolation (namespaces, cgroups). Instead, it uses capabilities.

What is a capability?

A capability is an unforgeable token that grants its holder specific permissions on a specific resource. For example:

• “Read and write to memory region 0x1000–0x2000”
• “Send messages to IPC endpoint #42”
• “Access VirtIO block device at MMIO address 0xFE00”

Key properties

1. No ambient authority: A newly created process has zero capabilities. It can’t do anything until its parent grants it capabilities.

2. Unforgeable: Capabilities are managed by the kernel. Userspace can’t create them or guess valid ones.

3. Transferable: Capabilities can be passed between processes via IPC, enabling controlled delegation.

4. Revocable: A capability can be revoked, immediately cutting off access.

Why not namespaces?

Linux namespaces are “isolation after the fact” — processes start with broad access and namespaces restrict what they can see. Capabilities are “isolation by default” — processes start with nothing and are explicitly granted only what they need.

For a container OS, this means a compromised container literally cannot access resources it wasn’t given capabilities for. There’s no kernel interface to probe, no /proc to read, no syscall to escalate through — the authority simply doesn’t exist.

Inspiration

• seL4: Formally verified capability microkernel. ThemeliOS borrows its capability model.
• Fuchsia/Zircon: Google’s capability-based OS. Demonstrates the model works at scale.

Memory model

ThemeliOS uses hardware-enforced memory isolation:

• Each process runs in its own virtual address space (page tables enforced by the MMU).
• The kernel has its own address space that userspace cannot access.
• Shared memory between processes requires explicit capabilities from both sides.

Physical memory management

A frame allocator tracks free physical memory pages (4 KiB). Frames are allocated to:

• Process page tables
• Kernel heap
• Shared memory regions
• DMA buffers for device drivers

Virtual memory layout

The virtual address space layout will be defined per-architecture, but the general structure is:

0x0000_0000_0000_0000  ┌──────────────────────┐
                        │   Userspace           │
                        │   (per-process)       │
0x0000_7FFF_FFFF_FFFF  └──────────────────────┘
                        ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┐
                          Non-canonical hole
                        └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘
0xFFFF_8000_0000_0000  ┌──────────────────────┐
                        │   Kernel space        │
                        │   (shared, all procs) │
0xFFFF_FFFF_FFFF_FFFF  └──────────────────────┘

(This is the x86_64 layout; aarch64 is similar but with different conventions.)

IPC

Inter-process communication is the backbone of the microkernel. Since drivers, filesystems, and networking all run in userspace, every system operation involves IPC.

Synchronous message passing

The primary mechanism: a client sends a message to a server and blocks until it gets a reply. This is used for request/response patterns like “read this file” or “send this network packet.”

Performance consideration

IPC overhead is the classic criticism of microkernels. ThemeliOS will address this by:

• Keeping messages small (pointers to shared memory for bulk data)
• Using register-based fast-path for small messages
• Careful cache-aware scheduling of communicating processes

Immutability

The OS root filesystem is read-only. The entire OS image is a single artifact that is booted as-is.

• Updates: Swap the entire image. No package managers, no apt-get, no partial updates.
• Configuration: Injected at boot time via cloud-init-style metadata or the management API.
• Ephemeral state: Container images and runtime state live on a RAM-backed ephemeral layer that is lost on reboot.

This model treats nodes as cattle: if a node is unhealthy, replace it with a fresh one. No debugging on the node, no SSHing in, no manual fixes.

Target platforms

ThemeliOS is designed to run as a virtual machine, with bare-metal support as a secondary goal.

Capability System

This document details the design of ThemeliOS’s capability system — the core security mechanism of the kernel.

Status: Design phase. Implementation begins in Phase 2.

Overview

In ThemeliOS, every resource is accessed through capabilities. A capability is a kernel-managed, unforgeable token that encodes:

1. Which resource (identified by a kernel object ID)
2. What operations are permitted (a bitmask of rights)

Capability types

Capability spaces

Each process has a capability space (CSpace) — a table mapping local capability slots to kernel objects. A process refers to its capabilities by slot index, not by object ID. The kernel translates slot indices to objects on each syscall.

Process A's CSpace:
  Slot 0 → MemoryCap(region=0x1000, rights=RW)
  Slot 1 → EndpointCap(endpoint=#7, rights=Send)
  Slot 2 → (empty)
  Slot 3 → ThreadCap(thread=#12, rights=Start|Stop)

Process B's CSpace:
  Slot 0 → EndpointCap(endpoint=#7, rights=Receive)
  Slot 1 → MemoryCap(region=0x2000, rights=R)

Process A can send to endpoint #7 (slot 1), and Process B can receive from it (slot 0). Neither can access the other’s memory — they’d need explicit capabilities for that.

Capability operations

Grant

A parent process can grant a capability to a child process, optionally with reduced rights:

Parent has: MemoryCap(region=X, rights=RWX)
Parent grants child: MemoryCap(region=X, rights=R)

The child gets read-only access. Rights can only be reduced, never elevated.

Transfer via IPC

Capabilities can be attached to IPC messages. This is how services delegate access:

FileServer receives "open /config" request
FileServer replies with MemoryCap(region=file_data, rights=R)
Client now has read access to the file's memory region

Revoke

The kernel (or a process with the appropriate meta-capability) can revoke a capability, immediately invalidating it. Any future use of the revoked slot returns an error.

Container mapping

In ThemeliOS, a “container” is a group of processes sharing a common set of capabilities. The container’s capability set defines its sandbox:

• Memory: Only the memory regions granted to it
• Network: Only the network endpoints it has capabilities for
• Filesystem: Only the filesystem views it’s been granted
• IPC: Only the services it has endpoint capabilities for

A container cannot discover or access anything outside its capability set. Unlike Linux containers (where a kernel exploit can escape the namespace), escaping a capability sandbox requires forging a kernel object — which is impossible without a kernel memory corruption bug.

Comparison with Linux isolation

Memory Management

This document describes ThemeliOS’s memory management subsystem design.

Status: Design phase. Implementation begins in Phase 1.

Overview

The memory management (MM) subsystem is responsible for:

1. Physical frame allocation — tracking which 4 KiB pages of physical RAM are free or in use
2. Virtual memory — creating and managing page tables for each process
3. Kernel heap — providing dynamic allocation (alloc-style) for kernel data structures

Physical memory

Boot-time discovery

The bootloader provides a memory map describing which physical address ranges are usable RAM, reserved by firmware, or used for MMIO. The frame allocator uses this map to initialize its free list.

Frame allocator

The frame allocator hands out 4 KiB physical memory frames. Initial implementation will use a bitmap allocator:

• One bit per physical frame (1 = allocated, 0 = free)
• Simple, predictable, easy to implement
• For 4 GiB of RAM: bitmap is 128 KiB (manageable)

Later optimization: replace with a buddy allocator for efficient allocation of contiguous multi-frame regions (needed for DMA buffers, large pages).

Capability integration

Physical frames are resources protected by capabilities. When a process requests memory:

1. Kernel allocates a frame from the free pool
2. Kernel creates a MemoryCap for that frame
3. Kernel inserts the capability into the process’s CSpace
4. Process can now map the frame into its address space using the capability

A process cannot access physical memory it doesn’t have a capability for — the page tables are configured to reflect capability permissions.

Virtual memory

Address space layout (x86_64)

 Lower half (user space, per-process):
   0x0000_0000_0000_0000 - 0x0000_7FFF_FFFF_FFFF

 Upper half (kernel space, shared across all processes):
   0xFFFF_8000_0000_0000 - 0xFFFF_FFFF_FFFF_FFFF
     ├── Physical memory direct map
     ├── Kernel code and data
     ├── Kernel heap
     └── Per-CPU data

Page tables

x86_64 uses 4-level page tables (PML4 → PDPT → PD → PT), each with 512 entries. Each entry is 8 bytes and can point to:

• The next level table
• A large page (2 MiB at PD level, 1 GiB at PDPT level)
• A 4 KiB page (at PT level)

The kernel manages page tables for each process. When a context switch occurs, the CPU’s CR3 register is loaded with the new process’s PML4 physical address, instantly switching the entire address space.

aarch64 differences

aarch64 uses a similar 4-level translation table scheme but with different register names (TTBR0/TTBR1 instead of CR3) and different table entry formats. The architecture abstraction layer hides these differences from the rest of the kernel.

Kernel heap

The kernel needs dynamic allocation for data structures like:

• Process control blocks
• Capability tables
• IPC message buffers
• Driver state

We’ll use the linked_list_allocator crate initially (a simple free-list allocator suitable for #![no_std] kernels), backed by physical frames allocated from the frame allocator.

The kernel heap lives in the upper-half virtual address space and is shared across all contexts (but only accessible from kernel mode).

Memory safety

Rust’s ownership model provides compile-time guarantees against:

• Use-after-free: The compiler prevents using a frame after it’s been freed
• Double-free: The compiler prevents freeing a frame twice
• Data races: Shared mutable access requires synchronization (Mutex, RefCell)

The unsafe keyword is required for raw pointer operations (hardware register access, page table manipulation) — these are confined to small, well-documented blocks.

Storage Architecture

This document describes ThemeliOS’s storage stack — the block driver, the filesystem servers, and the capability-guarded syscalls that connect a userspace process to a file on disk. It reflects the system as built in Phase 3.

Status: Implemented in Phase 3.

The core idea: a hybrid microkernel

Filesystem parsers are one of the most exploited pieces of code in a monolithic kernel. SquashFS decompression, ext2 metadata walking, directory parsing — all of it consumes untrusted bytes from a disk that an attacker may control, and all of it historically runs with full kernel privilege. A single bug becomes a kernel compromise.

ThemeliOS splits the storage stack across the privilege boundary:

• The block driver stays in the kernel (ring 0). It is thin (~500 lines), talks only to trusted, emulated VirtIO hardware, and exposes a single abstraction: read and write fixed-size blocks.
• Every filesystem runs in userspace (ring 3), as a separate server process with its own address space and its own capabilities. SquashFS, ext2, and the overlay each parse untrusted on-disk data entirely outside the kernel.

A corrupt or malicious disk image can, at worst, crash the filesystem server that parses it. It cannot touch kernel memory, cannot read another process’s data, and cannot bypass a capability check — because the code doing the parsing never had those privileges to begin with.

┌─────────────────────────────────────────────────────────────┐
│                       Ring 0 (Kernel)                        │
│                                                              │
│   PCI scan ─▶ VirtIO-blk driver ─▶ BlockDevice trait         │
│                                        │                     │
│                                   Block server               │
│                                   (kernel task on an         │
│                                    IPC endpoint)             │
│                                        ▲                     │
│              VFS dispatch              │ IPC + shared memory  │
│         (routes SYS_OPEN/READ/… )      │                     │
│                   │                    │                     │
├───────────────────┼────────────────────┼─────────────────────┤
│                   │  Ring 3 (Userspace)│                     │
│                   ▼                    ▼                     │
│   ┌────────────┐   ┌────────────────┐   ┌────────────────┐   │
│   │  overlay   │──▶│    squashfs    │   │      ext2      │   │
│   │  server    │   │    server      │   │     server     │   │
│   │ (RAM upper │   │  (read-only    │   │  (read-write   │   │
│   │  + lower)  │   │   root)        │   │   data volume) │   │
│   └────────────┘   └────────────────┘   └────────────────┘   │
│      mount "/"                              mount "/data"    │
└─────────────────────────────────────────────────────────────┘

Everything above the block driver is a message. The kernel is a router, not a filesystem implementor — it validates capabilities and forwards IPC, but never parses a superblock or an inode.

The ring-0 block path

PCI enumeration

VirtIO devices on QEMU’s Q35 machine present as PCI devices. At boot the kernel scans PCI configuration space (via the 0xCF8/0xCFC I/O ports on x86_64), identifies devices by vendor and class, and reads their BAR (Base Address Register) regions. VirtIO devices carry vendor ID 0x1AF4; a block device has PCI class 0x01. The scan records every device so a driver can bind to it later.

VirtIO transport and the virtqueue

VirtIO defines a standard transport (how to find config registers, negotiate features, and set up queues) shared by all device types. ThemeliOS implements the modern (VirtIO 1.0+) PCI transport: it walks the device’s vendor capabilities to locate the common/notify/ISR/device-config regions, maps them as uncached MMIO, and runs the initialization handshake (reset → ACKNOWLEDGE → DRIVER → FEATURES_OK → DRIVER_OK).

Data moves through a split virtqueue: a descriptor table plus an available ring (driver → device: “process these buffers”) and a used ring (device → driver: “I finished these”). ThemeliOS polls the used ring for completion rather than taking an interrupt — the spec permits it, and it is simpler and deterministic. Building the transport as a shared layer means the Phase 4 VirtIO-net driver inherits it for free.

The BlockDevice trait

The transport is device-type-agnostic; block semantics live behind a trait:

#![allow(unused)]
fn main() {
pub trait BlockDevice: Send + Sync {
    fn read_blocks(&self, start_lba: u64, buf: &mut [u8]) -> Result<(), BlockError>;
    fn write_blocks(&self, start_lba: u64, buf: &[u8]) -> Result<(), BlockError>;
    fn block_size(&self) -> u32;   // typically 512
    fn block_count(&self) -> u64;
    fn flush(&self) -> Result<(), BlockError>;
}
}

VirtioBlk is the first implementation. A block request is a three-descriptor chain — a header (request type + sector), the data buffer, and a one-byte status. Because heap buffers are not guaranteed to be physically contiguous (which DMA requires), the driver copies through a physically-contiguous bounce buffer, chunking large transfers at 64 KiB. The trait is the seam that lets Phase 8 add NVMe or VirtIO-SCSI drivers with zero changes to any filesystem code.

The block server

Ring-3 filesystem servers cannot call BlockDevice methods directly — they have no access to MMIO or DMA. The block server bridges the gap. It is a kernel task that listens on an IPC endpoint, receives block requests, performs the I/O through the trait, and replies with a status:

Request (client → server), in the four IPC message words:
  word0 = operation   (0 = READ, 1 = WRITE, 2 = FLUSH)
  word1 = start LBA
  word2 = block count
  word3 = byte offset into the shared region

Reply (server → client):
  word0 = status      (0 = OK, 1 = ERROR)
  word1 = error code

Modeling the device as an IPC service (rather than a dedicated syscall) keeps the kernel’s syscall surface small: a filesystem server can touch storage only if it holds both an endpoint capability to the block server and a shared-memory capability for the data buffer. With neither, it cannot reach the disk at all.

One block server runs per disk. At boot, ThemeliOS starts two — one for the SquashFS root disk and one for the ext2 data disk — each with its own endpoint, device, and shared region, so a request on one never touches the other’s state.

Moving data: shared memory

IPC messages are four words — far too small for a 512-byte block, let alone a 128 KiB SquashFS block. Bulk data travels through a shared memory region instead. The kernel allocates contiguous physical frames and maps them into both participants’ address spaces; the IPC message only names a byte offset into that region.

A new capability type, CapType::SharedMemory { phys_base, size, owner_pid }, governs these regions. The kernel reaches them through the HHDM (the higher-half direct map of physical memory) for its own DMA; each server sees them as ordinary user-writable, non-executable pages.

Two shared regions participate in a typical read:

• A block region between the block server and a filesystem server, carrying raw disk blocks.
• A client region between a filesystem server and its client (the kernel’s VFS layer, or another server), carrying paths and file data.

The request/reply protocol serializes access to each window — a client waits for the reply before reusing the buffer — so no locking is needed on the region itself.

The ring-3 server framework

Each filesystem server is a separate no_std Rust crate compiled to a flat binary (no ELF headers — the kernel has no ELF parser, since ELF parsing is itself attack surface). The binaries are embedded into the kernel image with include_bytes!() and loaded into fresh user pages at spawn time. A shared linker script fixes their load address.

spawn_server() creates a process with an isolated address space, copies the binary into user code pages, maps stack and heap pages, maps the shared regions, grants the configured capabilities, and starts the server in ring 3 at its entry point. Servers link against libthemelios, a small userspace library providing:

• Syscall wrappers (ipc_send, ipc_receive, ipc_call, yield_now, exit, …)
• A global heap allocator over the server’s fixed heap region
• A panic handler that reports the error and exits — a server panic is contained, never fatal to the kernel
• The shared filesystem and block protocol types (fs_proto)

The filesystem servers

All three servers speak the same request/reply protocol on their IPC endpoint. Paths and bulk data pass through the client shared region; the message words carry the opcode, handles, offsets, and lengths.

FS_OPEN     [OP_OPEN,    path_off, path_len, flags]   → [status, fd]
FS_READ     [OP_READ,    fd, buf_off, buf_len]        → [status, bytes_read]
FS_WRITE    [OP_WRITE,   fd, buf_off, buf_len]        → [status, bytes_written]
FS_CLOSE    [OP_CLOSE,   fd]                          → [status]
FS_STAT     [OP_STAT,    path_off, path_len, …]       → [status, size, is_dir]
FS_READDIR  [OP_READDIR, fd, max_entries]             → [status, entry_count]
FS_CREATE / FS_MKDIR / FS_UNLINK                      → [status, …]

SquashFS server — the read-only root

SquashFS is a compressed, read-only format — the natural choice for an immutable root image. The server reads the superblock (magic 0x73717368), then walks compressed metadata blocks (each an 8 KiB-max block prefixed with a length header, inflated with miniz_oxide’s pure-Rust zlib), parses inodes (basic and extended directory/file forms), lists directories, reads file data blocks, and unpacks fragments — the packed tails of small files. Writes are rejected with ReadOnlyFs.

Overlay server — the ephemeral writable layer

The immutable-root model needs a place for runtime writes to go without touching the read-only image. The overlay server provides an overlayfs-style merged view: a RAM-backed upper layer stacked over the SquashFS lower layer.

• Reads check the upper layer first; on a miss (and no whiteout) they forward to the SquashFS server via IPC.
• Writes to a lower-layer file trigger copy-up: the file is read from SquashFS, copied into RAM, and modified there (up to 1 MiB per file, 8 MiB total budget).
• Deletes of lower-layer files write a whiteout marker that hides the name.
• Directory listings merge upper and lower entries, with the upper winning and whiteouts removed.

The upper layer is pure RAM, so it evaporates on reboot — exactly the ephemeral semantics a cattle-not-pets node wants. This is the same layering model container runtimes use to stack image layers, which is why Phase 5’s container storage comes largely for free.

ext2 server — the persistent data volume

Containers need real persistent volumes. ext2 is the simplest Linux-compatible on-disk filesystem — ext4 without the journal or extents — so it is easy to implement correctly and readable by standard host tools. The server parses the superblock (magic 0xEF53) and block group descriptors, reads and writes inodes (12 direct block pointers plus one single-indirect pointer), walks linear directories, and allocates blocks and inodes via the on-disk bitmaps, keeping the free counts consistent. It works with 1 KiB blocks and 256-byte inodes.

Volumes are formatted on the host with mkfs.ext2, never by the kernel. Power-loss durability is out of scope for Phase 3 (no journal), which is acceptable for the QEMU test target. After the kernel test suite writes to an image, e2fsck -fn reports it clean — bitmaps, link counts, and directory structure all consistent.

VFS dispatch, capabilities, and syscalls

The kernel ties the servers together through a small VFS layer and two new capability types:

• CapType::Filesystem { mount_id } — the right to open paths on a mount (READ and/or WRITE).
• CapType::FileDescriptor { fd, mount_id } — a per-process handle to an open file, returned by open.

A mount table maps mount IDs to filesystem-server endpoints. Phase 3 mounts two: / → the overlay server, and /data → the ext2 server.

Six syscalls (numbers 8–13) expose storage to userspace: SYS_OPEN, SYS_READ, SYS_WRITE, SYS_CLOSE, SYS_STAT, SYS_READDIR. Each one:

1. Checks the caller’s capability (a process with no Filesystem capability gets PermissionDenied; a read-only capability cannot write).
2. Resolves the target mount and forwards the request to that server via IPC.
3. Copies data between the user buffer and the shared region, validating every user pointer page-by-page in the caller’s own address space (with a transfer size cap) so a bad pointer returns an error instead of faulting the kernel.
4. Records the operation in the audit log (AuditOp::FsAccess) with the PID, operation, and result.

The kernel never interprets filesystem bytes. It checks a capability, copies bounded buffers, and routes a message — nothing more.

A read, end to end

Following cat /version from the shell shows every layer cooperating:

1. The shell calls SYS_OPEN("/version"). The kernel checks the caller’s Filesystem capability for mount /, then sends FS_OPEN to the overlay server, writing the path into the client shared region.
2. The overlay finds no /version in its RAM upper layer and no whiteout, so it forwards FS_OPEN to the SquashFS server.
3. The SquashFS server resolves the inode. To read the on-disk bytes it sends a block request to its block server naming an offset in the block shared region.
4. The block server calls VirtioBlk::read_blocks, which posts a descriptor chain to the virtqueue and polls for completion. The blocks land in the shared region.
5. The SquashFS server inflates the metadata/data, fills in the file, and replies up the chain. The overlay returns a file descriptor; the kernel mints a FileDescriptor capability and hands the shell an fd.
6. SYS_READ repeats the forward-and-copy path, and the kernel copies the file bytes into the shell’s buffer. The shell prints THEMELIOS_ROOT.

Two ring-3 servers, one kernel block server, one hardware round-trip — and not a single byte of filesystem structure parsed inside the kernel.

Boot sequence

At boot, after PCI and the heap are up, boot_storage():

1. Probes each VirtIO block device, classifying it by on-disk magic (SquashFS vs. ext2 vs. an unknown scratch disk).
2. Starts a block server for the SquashFS disk and spawns the SquashFS server over it.
3. Spawns the overlay server with the SquashFS server as its lower layer and registers it as mount /.
4. Starts a second block server for the ext2 disk, spawns the ext2 server, and registers it as mount /data.
5. Prints the mount table to the serial console.

The debug shell then exposes mount, ls, cat, stat, write, and mkdir for interactive inspection of the live stack.

Why it matters for containers

The choices here are not just about Phase 3 — each one pays off later:

Running the parsers in ring 3 is the throughline: a hostile container image is untrusted input, and the component that unpacks it should never hold kernel privilege.

Network Architecture

This document describes ThemeliOS’s network stack — the VirtIO-net driver, the kernel net service, the ring-3 TCP/IP stack, and the capability-guarded socket API that connects a userspace process to the outside world. It reflects the system as built in Phase 4.

Status: Implemented in Phase 4 (amd64; arm64-ready by design).

The core idea: a hybrid microkernel

TCP/IP stacks are, alongside filesystem parsers, the most exploited code in a monolithic kernel. IP fragment reassembly, TCP option parsing, out-of-order segment handling, DHCP option walking — all of it consumes untrusted bytes off the wire, and all of it historically runs with full kernel privilege. A single bug becomes a kernel compromise.

ThemeliOS splits the network stack across the privilege boundary, exactly as the storage stack splits filesystems:

• The VirtIO-net driver stays in the kernel (ring 0). It is thin, talks only to trusted, emulated VirtIO hardware, does the DMA, and exposes a single abstraction: send this Ethernet frame / here is a received one.
• The entire TCP/IP stack runs in userspace (ring 3), in a single net server process built on smoltcp. Ethernet, ARP, IPv4, ICMP, UDP, TCP, and the DHCPv4 client all parse untrusted network data entirely outside the kernel.

A malformed or malicious packet can, at worst, crash the net server that parses it. It cannot touch kernel memory, cannot read another process’s data, and cannot bypass a socket capability check — because the code doing the parsing never had those privileges to begin with.

┌───────────────────────────────────────────────────────────────┐
│                        Ring 0 (Kernel)                        │
│                                                               │
│   PCI scan ─▶ VirtIO-net driver ─▶ NetDevice trait            │
│                                        │                      │
│                                  Net service                  │
│                                  (kernel task on an           │
│                                   IPC endpoint; drains        │
│                                   RX, forwards TX)            │
│                                        ▲                      │
│         Socket dispatch                │ IPC + shared memory   │
│    (routes SYS_SOCKET/SENDTO/…)        │  (pull-based frames)  │
│                   │                    │                      │
├───────────────────┼─────────────────────┼──────────────────────┤
│                   │  Ring 3 (Userspace) │                      │
│                   ▼                    ▼                      │
│              ┌──────────────────────────────────┐             │
│              │            net-server            │             │
│              │  smoltcp: Ethernet/ARP/IPv4/     │             │
│              │  ICMP/UDP/TCP + DHCPv4 client    │             │
│              │  Device-over-IPC frame transport │             │
│              │  Socket table (UDP/TCP/ICMP)     │             │
│              └──────────────────────────────────┘             │
└───────────────────────────────────────────────────────────────┘

Everything above the driver is a message. The kernel is a router, not a protocol implementor — it validates capabilities and moves bytes, but never parses an Ethernet header, an IP option, or a TCP segment.

The frame bridge: pull-based by necessity

Ring-3 servers cannot touch the NIC directly, so the kernel net service bridges the in-kernel driver to the ring-3 stack over IPC and shared memory.

The subtlety is that received frames arrive unsolicited at the NIC, but the kernel’s IPC is synchronous rendezvous only — there is no non-blocking send and no notification primitive, so a kernel task cannot push a frame to a ring-3 server that is not currently parked in receive. A single task that both pushed RX and served TX would deadlock.

The bridge is therefore pull-based: the ring-3 net server always initiates and the kernel service always replies. This is deadlock-free by construction — there is never an unsolicited kernel→ring-3 send.

The net server’s event loop, once per iteration:

1. MSG_POLL — “give me the next RX frame and the current time.” The service pops one frame from its RX ring (or reports none) and returns the monotonic clock (SYS_UPTIME_MS, for smoltcp’s timers).
2. Feed any frame to smoltcp and run iface.poll().
3. For each frame smoltcp wants to send, MSG_TX_FRAME — the service transmits it via the driver.
4. Service the DHCP client and any pending socket request, then yield.

Frame bytes travel through shared memory regions (an RX ring and a single TX slot); the four IPC words carry only lengths and status. The service drains the NIC’s RX virtqueue into the RX ring on each MSG_POLL; overflow drops the oldest frame and bumps a counter surfaced by ifconfig.

Polled RX, no interrupts. The VirtIO transport disables MSI-X and there is no PCI INTx/ISR path, so RX is polled — the net server’s continuous poll loop plus the driver’s pre-posted buffer ring absorb bursts. Interrupt-driven RX is deferred (it would need INTx discovery or an IO-APIC + MSI-X).

The clock

smoltcp’s poll loop needs a monotonic clock for retransmit and DHCP timers. The kernel already maintains a 100 Hz tick; it is exposed as SYS_UPTIME_MS (ticks × 10), which the net server reads on every poll. No periodic kernel→server tick message is needed — the server pulls the time whenever it polls.

Addressing: DHCP

On a live boot the net server runs smoltcp’s dhcpv4::Socket and applies each lease to the interface (address, prefix, default route). It reports every acquired, renewed, or lost lease to the kernel over a MSG_CONFIG message so the ifconfig shell command can display the live configuration — the kernel only records what the server tells it. DNS server addresses from the offer are captured for display but there is no resolver in Phase 4.

DHCP is gated behind a boot-argument flag (NET_ARG_DHCP, packed alongside the NIC’s MAC in arg1): the live node sets it, while the deterministic static-IP round-trip tests spawn the server without it and keep a fixed 10.0.2.15.

The socket API: capability-checked, kernel-routed

Sockets follow the same pattern as the VFS: a new CapType::Socket, with two roles.

• The network authority (socket_id == SOCKET_FACTORY) is the right to create sockets. SYS_SOCKET requires the caller to present it — a process without it cannot make a socket at all.
• A per-socket capability is minted by SYS_SOCKET (or by accept, for an inbound connection) and consumed by the send/recv/close syscalls. WRITE grants send, READ grants receive. Closing a socket revokes its capability, exactly as closing a file descriptor does.

The data path mirrors the VFS too: the payload region is shared between the kernel and the net server, not the client process. The kernel copies validated user bytes into it and reads results out; a client never shares memory with the server directly, and the kernel mediates and audit-logs every transfer (AuditOp::NetAccess). The kernel never parses packet data — it checks a capability, forwards an OP_SOCK_* request, and moves bytes.

Transports

Sockets are non-blocking. With no socket readiness wait/wake mechanism yet, a caller that wants to block busy-polls with yield (the same pattern the storage boot path uses). A readiness/wake primitive is deferred.

Shell commands

The debug shell exposes the stack for inspection and manual testing. These run in the kernel and use kernel-internal socket helpers (the kernel is trusted); userspace must go through the capability-checked syscalls.

The sockets listing is itself a round-trip: the kernel asks the net server to serialise its socket table into the shared region (OP_SOCK_LIST) and decodes the entries — the socket state lives entirely in the ring-3 server.

The arm64 seam

Phase 4 runs and is tested only on amd64, but the stack is designed to port unchanged:

• The NetDevice trait is the bus/arch seam. VirtIO-net implements it today; the Phase 7 arm64 port implements device discovery for virtio-mmio/ECAM behind the same trait without touching the stack or the socket API.
• The TCP/IP stack is architecture-independent. To keep it honest, CI compiles smoltcp alone — with the net server’s exact feature set — for aarch64-unknown-none (cargo xtask arm64-gate, the servers/smoltcp-gate crate). If a dependency ever pulled in amd64-only or std code, that job goes red long before the arm64 port would trip over it. This is the same “compile the library bare-metal” check Phase 3 used for miniz_oxide.

The net server binary itself builds only for x86_64 in Phase 4 because its libthemelios syscall wrappers are raw x86 syscall assembly; the aarch64 build of the kernel and servers is Phase 7 work.

What is contained, and what is deferred

Contained by the ring-3 boundary: the whole smoltcp stack, all protocol parsing, and the DHCP client. A bug there is a net-server crash, not a kernel compromise — the same containment already accepted for miniz_oxide in the SquashFS server.

Deferred (with rationale):

• Interrupt-driven RX — needs INTx/ISR or IO-APIC + MSI-X; polled RX suffices for now.
• A DNS resolver — DNS server addresses are captured and displayed, but name resolution is out of scope for Phase 4.
• Socket readiness wait/wake — callers busy-poll with yield; a blocking wrapper and a wake primitive come later.
• Per-client payload regions — a single kernel↔server region serves one request at a time in practice, matching the FS servers.
• A net-server crash-isolation test — the ring-3 containment is structural (identical to the FS servers) and exercised implicitly whenever a server is spawned; a dedicated fault-injection test is deferred rather than adding an artificial panic path.

Where the code lives

Container Runtime

This document describes ThemeliOS’s container runtime — how a container image becomes a running, isolated process. It reflects the system as built in Phase 5.

Status: Implemented in Phase 5 (amd64). Core pipeline complete; a real static-musl image over a live registry, container exec, and moving the image parser into a ring-3 server are documented deferrals (see the end of this chapter).

What a container is here

A container is not a virtual machine and not a Linux namespace. It is an ordinary ThemeliOS process with three things arranged so it believes it is running on Linux, inside its own root filesystem, with no access to anything it wasn’t given:

1. a Linux syscall personality — its syscall instructions are answered by a Linux-ABI table, not the native ThemeliOS one;
2. a rootfs mount — every path it opens resolves inside one filesystem image, with no way to name anything outside it;
3. an empty capability space — it holds no capabilities at all, so every privileged operation (opening a socket, signalling another process) is denied by construction.

The isolation boundary is the capability system, not a namespace abstraction layered on top of a shared kernel. A container can do exactly what its capabilities permit — which, by default, is nothing beyond its own rootfs and writing to its stdout/stderr. This is the whole reason the capability microkernel exists: container isolation falls out of the capability model rather than being bolted on.

The pipeline: image → rootfs → process

Running a container (container::create then container::start) is a straight line from image bytes to a ring-3 task:

 image bundle ─► unpack ─► assemble rootfs ─► load entrypoint ELF ─► ring-3 Linux process
   (OCI/tar)     (oci)     (VFS writes)        (elf loader)          (personality = Linux)

1. Unpack (oci::unpack / oci::unpack_registry). The image — either a local docker save bundle or a registry manifest + blobs — is parsed into a flat file list plus a runtime config (entrypoint, cmd, env, workdir). Layers are applied in order, with OCI whiteouts (.wh.*) resolved. See Images and layers below.
2. Assemble the rootfs. The unpacked files are written onto a writable mount (the ext2 data volume from the storage stack) via the ordinary VFS syscalls — the container runtime has no special filesystem access.
3. Load the entrypoint. The entrypoint ELF is read out of the assembled rootfs (a VfsByteSource feeding the ELF loader) and mapped into a fresh address space with W^X segment permissions and a System V initial stack (argc/argv/envp/auxv).
4. Enter ring 3. The process is marked Personality::Linux, given its rootfs mount and initial cwd, and spawned. From its first instruction it is a Linux program that cannot tell it isn’t on Linux.

The Linux syscall personality

A process flagged Personality::Linux has its syscall entries routed to linux::syscall::dispatch instead of the native table. This matters because the two ABIs collide — native SYS_SEND is 1, but Linux write is also 1 — so the personality flag, checked on every syscall, is what keeps them apart.

The implemented subset is what a small static binary needs to start, run, and exit: write/writev, openat/read/close/lseek/fstat/getdents64/ getcwd/chdir/readlinkat (the filesystem set, Phase 5.2), brk/mmap (anonymous), arch_prctl (TLS via %fs), clone(CLONE_THREAD)/futex/ set_tid_address (threads, Phase 5.3), clock_gettime, getrandom, exit/exit_group, and the process-control calls below. Unimplemented numbers return -ENOSYS.

Per-thread TLS is real: arch_prctl(SET_FS) records an %fs base that the scheduler restores on every context switch, so thread-local storage works across preemption.

Capability isolation — how a container is contained

Two boundaries make a container safe, and Phase 5.7 made both enforced and tested rather than incidental.

Filesystem: one mount, .. clamped at the root

Every filesystem syscall resolves its path against the process’s single rootfs mount. The path resolver clamps .. at the root: ../../../../etc/passwd normalizes back to /etc/passwd inside the container’s own mount — there is no host root for it to escape to. A container therefore cannot name, let alone open, any file outside its image.

This is verified positively, not vacuously. The test_container_isolation integration test runs a probe (servers/isolation-smoke) as a container /init that opens /only, then opens ../../../../only, and asserts the second call succeeds and returns bytes identical to the first — proving the clamp is live on the real syscall path, not merely that some out-of-tree path happens to miss. (A bare “escape returns -ENOENT” assertion would prove nothing: with a single mount and no host root, the miss happens whether the clamp works or not.)

Per-container confinement (Phase 6.1b). Multiple containers share one writable mount (a per-container mount is infeasible here — mounts need a physical disk and are never freed), so each container is instead confined to a /c/<id> subdirectory. Its rootfs_base is prepended to every already-..-clamped path at a single choke point (linux::fs::host_path), so the container’s / is /c/<id> and it can name nothing outside that subtree — not a sibling container’s files, not the mount root. Because untrusted image paths are just as dangerous (the ext2 server honors .., so a layer member ../../host_secret would escape at assembly time), every image path is run through the same clamp before it is written. test_container_confinement proves both halves: a malicious ../../evil is clamped into the base (never reaching the mount root, and a root /host_secret is left intact), and a confined probe reads its own file but cannot open that root /host_secret. (One caveat carried forward: the guarantee is proven for a single running container; serializing the kernel↔fs-server forwarding region is a prerequisite before the management API runs multiple containers concurrently.)

Everything else: no capability, no access

A container is created with an empty capability space. Ambient authority does not exist in ThemeliOS, so holding no capability means being able to do nothing privileged. The sharpest example is the network: a container that calls socket(AF_INET, SOCK_DGRAM, 0) receives -EPERM. It holds no SOCKET_FACTORY capability, and the Linux socket() ABI carries no handle by which it could present one — so the denial is a checked, real Linux errno. The isolation probe asserts exactly this -EPERM.

The result: opening a network socket, signalling another process, or touching another container’s filesystem are all denied at the capability layer, uniformly, by the same mechanism that governs every other resource in the system.

Images and layers

Two on-disk formats are understood, both parsed by the dependency-light oci module (alloc-only, no serde):

• docker save bundles (Phase 5.4): an outer tar containing manifest.json, an image config JSON, and one or more uncompressed layer tars.
• Registry images (Phase 5.6): a Docker Registry HTTP API v2 manifest naming a config blob and gzip-compressed layer blobs by sha256: digest. Every blob is digest-verified before use — a blob whose contents don’t hash to the digest the manifest names is rejected (DigestMismatch) before it is ever parsed or inflated.

Because these parsers consume untrusted image bytes, they fail closed on hostile input: bounded gzip inflation (a decompression bomb is capped, not allowed to exhaust the heap), bounded JSON nesting (a deeply-nested manifest cannot overflow the kernel stack), and no arithmetic panics on adversarial lengths.

Lifecycle

• run (shell) launches the demo container: unpack → assemble → load → run, then waits for the exit status and prints it.
• stop <pid> (shell) force-terminates a running container — container::terminate, the minimal SIGKILL equivalent. It verifies the target actually is a container before tearing it down, so it cannot be used to destroy a kernel service. Teardown marks all of the container’s tasks dead before freeing its address space, closing a use-after-free window in which a timer tick could otherwise switch into a task whose page tables had just been freed.
• exit status is captured by exit_group and readable by the launcher; this is the “wait” primitive the runtime uses.

kill(2) from inside a container may only signal itself (there is no cross-process signal capability): a fatal self-signal routes to exit_group with the conventional 128 + signo status; signalling any other pid returns -EPERM. wait4(2) returns -ECHILD — there is no parent/child process linkage.

Testing

The runtime is exercised deterministically, with no external toolchain, by hand-crafted Linux-ABI probe ELFs run as container entrypoints, each reporting a result code to a kernel-mapped page:

• test_container_run — a full unpack → assemble → load → run → exit round-trip.
• test_container_isolation — the enforced-isolation test described above (positive read, live .. clamp, absent-path miss, socket() → -EPERM).
• test_oci_unpack, test_sha256, test_registry_pull, test_registry_hardening — the image/registry pipeline, including digest verification and the fail-closed hardening for bombs, deep JSON, and bad lengths.

Deferred

The following are documented, deliberate deferrals — the core pipeline and its isolation guarantees do not depend on them:

• A real static-musl image over a live registry. The runtime has been driven with synthetic probe ELFs as /init and a mock registry transport; the live TCP Connection (through a slirp guestfwd to a host registry:2) and a real busybox image are a thin, well-scoped follow-up.
• exec into a running container — a second process sharing an existing container’s rootfs. Needs process-group semantics not yet required.
• Real wait4 and signal-handler delivery. These need parent/child PID tracking and a per-process signal-disposition table respectively; rt_sigaction is currently an accepted no-op.
• Moving the image parser into a ring-3 oci-server. The oci module (tar/ JSON/gzip/sha256) currently runs in the kernel for the deterministic tests. Relocating it — so untrusted image bytes never parse in ring 0, exactly as the filesystem and network stacks already do — is the standing containment hardening for this subsystem.

Management API

ThemeliOS is managed entirely through an external HTTP API — there is no SSH, no shell, no interactive login. A node is driven the way a container host is driven: a control-plane process listens on a TCP port and speaks a subset of the Docker Engine API, so existing tooling and habits transfer. This chapter describes that control plane: where it runs, how it is authorized, the ABI it drives into the kernel, and the two layers of authentication that gate it.

The api-server is a ring-3 process

The management API is served by api-server, an ordinary userspace (ring-3) process — not kernel code. This is a direct consequence of the microkernel design: parsing untrusted HTTP off a socket is exactly the kind of complex, attack-exposed work that does not belong in the kernel. The kernel exposes a narrow, capability-checked seam; everything above it — HTTP framing, request routing, JSON, authentication policy — lives in the api-server, where a bug is a crashed process, not a compromised kernel.

The full request path is:

accept → HTTP parse → authenticate → route → management ABI (SYS_MGMT) → JSON → reply

The first four stages are ring-3 code; only the management ABI crosses into the kernel, and only after the request has been authenticated.

Fault-freedom is mandatory

There is one sharp constraint on a ring-3 server: a user-mode fault halts the whole kernel (the IDT halts on faults taken from ring 3, rather than killing just the faulting task — a deliberate fail-stop for this stage of the project). A page fault, a panic, an out-of-bounds slice, an arithmetic overflow, or an unbounded recursion in the api-server is therefore a node-wide denial of service, reachable by anyone who can send it a request.

The api-server is written defensively against this:

• The request buffer is bounded to MAX_REQUEST (64 KiB) before it grows, so a hostile client cannot make it allocate without limit.
• Every syscall return value is checked; no unwrap/expect on socket data.
• Every loop is bounded (a large spin cap, yielding on WouldBlock) so a slow or stalled peer cannot wedge the single core.
• The HTTP parser and JSON parser are None-on-malformed, never panicking: bounded header counts, bounded body size, checked_add on lengths, and a recursion-depth guard on the JSON parser (a deeply-nested [[[[… body cannot overflow the stack).
• The accepted socket is closed after every request; libthemelios installs an allocation-error handler so an OOM exits the process cleanly instead of aborting.

Two layers of authorization

Access to the management API is gated at two independent layers, one in the kernel and one in the api-server.

Layer 1 — the Management capability (kernel)

The kernel does not know about HTTP, tokens, or clients. It knows one thing: the authority to drive the management ABI is a capability. CapType::Management is a coarse, fieldless sentinel capability — holding it grants every management operation; not holding it denies every one — exactly analogous to the SOCKET_FACTORY authority for networking. It is minted only to the trusted, kernel-spawned api-server (via ServerConfig::grant_management) and never placed in a container’s capability space. A container is created with an empty CSpace, so it can never even name the management ABI, let alone call it.

This is the microkernel’s answer to ambient authority: without a capability model, any process could enumerate or stop every container on the node. Here, that power is an unforgeable token held by exactly one process.

Layer 2 — bearer-token authentication (api-server)

The Management capability answers “may this process drive the ABI?” It says nothing about “may this remote client make this API call?” That second question is application-level policy, and it lives in the api-server.

Every route except the GET /_ping and GET /version health/version probes requires an HTTP Authorization: Bearer <token> header whose token matches the one the kernel provisioned to the api-server via boot-info. The token is provisioned only to the control plane (the sole grant_management server), so only it holds the node secret in its address space; it rides in the boot-info page rather than being baked into the binary image, modelling a per-node secret handed over at spawn.

Enforcement details:

• A missing or wrong token is rejected with 401 Unauthorized before any management op runs — including on unknown paths, so an unauthenticated client cannot even enumerate which routes exist.
• A wrong token is 401, not 403: per RFC 9110 a correct token would work, so the request is “unauthorized”, never “forbidden”. 403 is reserved for an authenticated-but-unauthorized principal, a distinction a single all-or-nothing token does not have.
• The Bearer scheme is matched case-insensitively; the token itself is compared exactly.
• Authentication outcomes are audited on the same ABI as operations: successful calls audit as ApiAccess (the management op), and rejections go through a dedicated SYS_MGMT audit verb that records a distinct ApiAuthReject event — so a failed auth attempt is as visible in the audit log as a successful op.

Not transport security. Bearer auth over plaintext HTTP gates who can drive the API, but does nothing against a wire sniffer — the token travels in cleartext. Transport confidentiality (TLS/mTLS) is deferred; until it lands, the API must not be exposed on an untrusted network. For this reason the token compare is a plain byte compare, not constant-time: a timing oracle would reveal nothing the cleartext transport does not already give away.

The SYS_MGMT ABI

The seam between the ring-3 api-server and the kernel is a single syscall, SYS_MGMT, op-multiplexed on a verb selector in RDI — so the whole growing ABI costs one syscall number instead of one per verb. Op-specific arguments ride the remaining registers; the return value is the verb’s success value (a capability handle or a byte count) with bit 63 clear, or a high-bit-set MgmtError code.

Every verb is capability-checked against the caller’s Management cap and audited inside the kernel mgmt module before it touches any backing service — the fail-closed property. Each op returns owned bytes (a Vec<u8> of compact Engine-API JSON, or a freshly minted capability handle), so the api-server consumes the result with no shared-lifetime hazard across the ring boundary.

MgmtError is a stable, numbered space (PermissionDenied, NotFound, InvalidState, InvalidArgument, CreateFailed, ServerUnavailable, NoResources, BufferTooSmall) that the api-server maps to Docker-style HTTP statuses (404/409/400/500). A read verb whose JSON exceeds the caller’s output buffer fails closed as BufferTooSmall rather than truncating.

The kernel-accept shim

The listener is opened through the ABI, not by the ring-3 server binding a socket directly. LISTEN runs the trusted kernel socket path (open → bind → listen) and mints a per-listener Socket capability parented to the management handle, so it is revoked together with the management grant. The api-server then accepts on that handle through the ordinary socket ABI — there is no separate management accept. This keeps the privileged bind/listen inside the kernel while the untrusted accept-loop lives in ring 3.

The Engine API subset

The api-server implements a hard subset of the Docker Engine API — enough to list, create, start, stop, and inspect containers and read their logs:

A Docker /v1.NN API-version prefix on the path is stripped before routing. Container logs are captured into a per-container RAM ring buffer as the container writes to stdout/stderr (fd 1/2 through the Linux write/writev path), keyed by container id so the log survives the process; LOGS reads back a bounded tail.

Testing

The management API is proven by test_api_server, in three phases:

1. Fail-closed control — the api-server spawned without the Management grant has its LISTEN denied (PermissionDenied) before any NIC access, and reports DENIED. This proves the capability gate.
2. Routing / auth / JSON self-test — a deterministic, in-process run (no network) drives a fixed set of requests through the router and asserts the exact HTTP statuses [200, 401, 401, 200, 400, 500, 409]. Each status is impossible for the catch-all 404, so observing it proves the specific arm ran: GET/POST routing, the 401/200 auth contrast on one route, the untrusted request-body JSON parse, Image extraction, and the create/start write verbs — all without depending on the timing-sensitive inbound-TCP path.
3. Live inbound smoke — a single authenticated GET /containers/json sent over a host-to-guest port forward, proving the accept → parse → authenticate → route → reply path (and that the Authorization header) round-trips over real TCP.

The in-process self-test exists because the immature ring-3 net server can deliver stale RX data across sequential connections on one listener, which makes a multi-connection, content-asserting wire test flaky; proving the content of the routing and auth logic in-process removes that dependency, leaving the wire path a single-connection smoke.

Deferrals

The management API is functionally complete for its core (list/create/start/stop/ inspect/logs + auth), with several capabilities explicitly deferred:

• TLS / mTLS — transport confidentiality and client-certificate auth. Until it lands, the API is a plaintext, app-token-gated interface not to be exposed on an untrusted network.
• exec and interactive streaming — docker exec and websocket-based bidirectional streams for interactive sessions.
• A live docker CLI / multi-request curl mutation sequence end to end — blocked on the net-server’s stale-RX-across-connections behavior and on POST create needing a /data mount and a real image provisioned at boot. The container-creation success path is covered in-kernel by the container-runtime tests; the API layer’s create/start/logs verbs are proven at the ABI and routing levels.
• Networks and image management endpoints beyond the container lifecycle subset.

Milestones

ThemeliOS development is organized into phases. Each phase builds on the previous one and produces a working, testable artifact.

Phase 0 — Boot (Complete)

Goal: Get the kernel booting on QEMU and printing to the serial console.

Deliverables:

• Bootloader integration (Limine or UEFI)
• Architecture-specific early init (x86_64 first)
• Serial console output (16550 UART on x86_64)
• “Hello from ThemeliOS” printed on boot
• cargo xtask run boots the kernel in QEMU end-to-end

Phase 1 — Kernel basics (Complete)

Goal: A kernel that can manage memory and schedule tasks. x86_64 only — aarch64 is deferred to Phase 7.

Deliverables:

• Physical frame allocator (bitmap-based)
• Kernel heap allocator
• Interrupt handling (GDT, IDT, 8259 PIC on x86_64)
• Timer-driven preemptive scheduler (round-robin)
• Basic kernel shell over serial (for debugging, will be removed later)
• Automated test infrastructure (isa-debug-exit, cargo xtask test, GitHub Actions CI)

Phase 2 — Isolation (Complete)

Goal: Implement the capability system and process isolation.

Deliverables:

• Custom page tables replacing Limine’s (required for per-process address spaces)
• Capability types and capability space (CSpace)
• Process creation with isolated address spaces
• Capability grant, transfer, and revocation
• Synchronous IPC (message passing between processes)
• Audit logging (tamper-evident record of capability usage for compliance and security)
• Reclaim bootloader-reclaimable memory (safe once we own GDT, page tables, and stack)
• First userspace process (init)

Phase 3 — Storage (Complete)

Goal: Read from a virtual disk and present a filesystem, using a hybrid microkernel design — a thin in-kernel block driver with all filesystem parsing in userspace servers. See Storage Architecture for the full design.

Deliverables:

• PCI enumeration and a VirtIO (modern PCI transport + split virtqueue) layer
• BlockDevice trait and a VirtIO-blk driver
• Kernel-side block server exposing the disk to userspace over IPC + shared memory
• CapType::SharedMemory for bulk data transfer across the privilege boundary
• Userspace server framework (flat-binary embedding, spawn_server) and the libthemelios support library
• SquashFS server (compressed read-only root) running in ring 3
• Overlay server (RAM upper + SquashFS lower, copy-up, whiteouts) — the ephemeral writable layer
• ext2 server (read-write persistent data volume) running in ring 3
• VFS dispatch with Filesystem / FileDescriptor capabilities and the filesystem syscalls (open, read, write, close, stat, readdir)
• Audit logging for filesystem operations
• cargo xtask image tooling to build the SquashFS root and ext2 data images
• Boot integration (mounts / and /data) and debug-shell commands (mount, ls, cat, stat, write, mkdir)

Phase 4 — Networking (Complete)

Goal: TCP/IP connectivity.

The whole TCP/IP stack (smoltcp) runs in a ring-3 net server; a thin VirtIO-net driver stays in the kernel and frames cross via a pull-based IPC bridge. Sockets are capability-checked and kernel-routed. See the Network Architecture doc for the full design.

Deliverables (all delivered):

• VirtIO network driver + NetDevice trait (the arm64/bus seam)
• Kernel net service (pull-based frame bridge) and SYS_UPTIME_MS clock
• Ring-3 smoltcp stack: Ethernet, ARP, IPv4, ICMP
• DHCPv4 client (address, gateway, DNS captured for display)
• Capability-checked socket API (CapType::Socket): UDP, TCP, and ICMP
• Shell: ifconfig, sockets, ping, udpsend, tcpconnect
• Boot integration (NIC + service + net server, DHCP-configured, at boot)
• Integration tests (driver, service, ARP/ICMP, DHCP, UDP echo, socket caps, socket listing, TCP client + server) — 35 tests, reliably green
• aarch64 smoltcp compile gate in CI (cargo xtask arm64-gate)

Phase 5 — Containers (Complete (core; real-image busybox, live registry transport, ring-3 oci-server deferred))

Goal: Run OCI container images as capability-isolated processes. See the Container Runtime chapter for the full design.

Delivered:

• ELF64 loader + exec — load a static ELF and enter ring 3 (Phase 5.0)
• Linux syscall personality — a per-process Linux-ABI table (write/writev/brk/ mmap/arch_prctl/clock_gettime/getrandom/exit_group/…), routed by a personality flag so it doesn’t collide with the native ABI (Phase 5.1)
• Linux filesystem syscalls over the VFS, rooted at a single rootfs mount with a ..-clamping path resolver (Phase 5.2)
• Linux threads: clone(CLONE_THREAD), futex WAIT/WAKE, per-thread %fs TLS restored across context switches (Phase 5.3)
• OCI image unpacking: docker save bundles → flat rootfs + config, layers and whiteouts applied (Phase 5.4)
• Container runtime: unpack → assemble rootfs → load the entrypoint from that rootfs → run it as a Linux process; exit-status capture (Phase 5.5)
• Registry pull: Docker Registry HTTP API v2, gzip layers, sha256 digest-verified before use, with fail-closed parsers (Phase 5.6)
• Enforced capability isolation + lifecycle: socket() → -EPERM (no capability), live ..-clamp proof, container teardown (stop), honest kill/wait4 errnos, and a test_container_isolation that proves the boundary on the live syscall path (Phase 5.7)

Deferred (documented — see the Container Runtime chapter): real static-musl image over a live registry transport; container exec; real wait4/signal-handler delivery; relocating the OCI image parser into a ring-3 oci-server; PTYs; per-container resource limits; registry auth/TLS + cloud credential helpers.

Phase 6 — Management (Complete — core)

Goal: Docker-compatible management API for the node.

A ring-3 api-server holds a Management sentinel capability, opens an inbound-TCP listener through the kernel-accept shim, and serves a subset of the Docker Engine API behind two layers of authorization: the kernel capability (which process may drive the ABI) and an app-layer bearer token (which client may call the API). Untrusted HTTP/JSON parsing stays in ring 3, fail-closed against a node-halting fault; every container mutation crosses into the kernel through the capability-checked, audited SYS_MGMT ABI. See the Management API chapter for the design.

Delivered:

• Docker Engine API subset — _ping, version, info, container list/inspect/ create/start/stop/logs (with /v1.NN version-prefix stripping)
• Capability-gated management ABI (SYS_MGMT) driven only by the trusted control plane
• App-layer bearer-token authentication (401 fail-closed; auth outcomes audited)
• Per-container RAM-ring log capture (docker logs)
• No SSH — the API is the only management interface
• Momus-audited untrusted-input surface (no reachable kernel panic, no auth bypass)

Deferred (documented):

• TLS client-certificate + transport security (mTLS/HTTPS) — the API is a plaintext, token-gated interface until it lands
• Interactive exec and bidirectional streaming for sessions (websocket)
• A live docker CLI / multi-request curl mutation sequence end to end (blocked on net-server RX recycling + a /data mount at boot)
• Broader Engine API surface (networks, volumes, images, events, stats)
• Configuration injection at boot time beyond ServerBootInfo

Phase 7 — aarch64 port (Not started)

Goal: Port all Phase 0 and Phase 1 functionality to aarch64 (ARM64), enabling ThemeliOS to run on ARM-based hardware and cloud instances (e.g., AWS Graviton).

Deliverables:

• aarch64 boot via Limine (UEFI on ARM)
• PL011 UART serial driver for debug output
• GIC (Generic Interrupt Controller) initialization and exception handling
• ARM generic timer for scheduler preemption
• Physical frame allocator (same bitmap design, architecture-independent)
• Kernel heap (architecture-independent, just works)
• Scheduler and context switch for aarch64 (different register set, different calling convention)
• Serial debug shell (architecture-independent, just works)
• cargo xtask run --arch aarch64 boots and passes all tests
• Automated tests on aarch64 QEMU in CI

Phase 8 — Hyperscaler support (Not started)

Goal: Boot and run on AWS, GCP, and Azure.

Deliverables:

• Instance metadata service (IMDS) clients for all three providers
• Cloud-aware configuration injection at boot time
• Machine image tooling (cargo xtask image --cloud aws/gcp/azure)
• AMI creation for AWS (raw disk import via aws ec2 import-image)
• GCP image creation (raw disk tarball + gcloud compute images create)
• Azure VHD image creation
• UEFI Secure Boot chain verification and kernel image signing
• Measured boot (TPM support)
• Boot validation on each provider’s compute instances
• GitHub Actions workflow to build downloadable QEMU ISOs (x86_64, aarch64)
• GitHub Actions workflows to build and publish cloud-specific machine images

Phase 9 — Testing and benchmarks (Not started)

Goal: Comprehensive test suite and performance benchmarks to validate the OS works correctly end-to-end.

Deliverables:

• CI infrastructure (GitHub Actions with QEMU, isa-debug-exit device for pass/fail exit codes)
• Boot smoke tests (kernel boots, reaches known-good state, no panic)
• Kernel unit tests (allocator, scheduler, capability enforcement tested in isolation)
• Kernel integration tests (spawn process + grant capability + IPC message + verify result)
• Security and isolation tests (capability violations, unauthorized memory access, process escape attempts — all must fail cleanly)
• Container runtime tests with standard images (alpine, busybox, nginx)
• Custom test images (memory stress, network connectivity, filesystem I/O, multi-process isolation)
• Container lifecycle tests (create, start, stop, restart, destroy, exec)
• Multi-container isolation validation
• Container networking tests
• Resource limit enforcement tests
• Cloud validation tests (boot on each hyperscaler, IMDS, networking, container workloads)
• Benchmarks: boot time, context switch latency, IPC throughput, memory allocation speed, container cold-start time
• Benchmark history tracking for regression detection

Phase 10 — Kubernetes (Not started)

Goal: Full drop-in K8s/K3s/RKE2 worker node. Any pod that runs on an Ubuntu or Flatcar node must run identically on ThemeliOS.

Deliverables:

• Full Linux syscall coverage for real-world K8s workloads (databases, language runtimes, service meshes, logging agents, init systems)
• CRI (Container Runtime Interface) gRPC API implementation
• CNI (Container Network Interface) plugin support (Flannel, Calico, Cilium)
• CSI (Container Storage Interface) driver support for persistent volumes
• Pod semantics (groups of containers sharing network and storage namespaces)
• kubelet (standard binary or compatible custom implementation)
• kube-proxy equivalent for service networking and load balancing
• Node registration, capacity reporting, and health conditions
• kubectl exec -it with full interactive shell support
• kubectl logs, kubectl cp, kubectl port-forward
• Pod resource management (CPU/memory requests and limits, QoS classes)
• DNS resolution for K8s service discovery

Phase 11 — GPU support (Not started)

Goal: GPU passthrough and accelerator support for containerized workloads across all major cloud providers.

Deliverables:

• VFIO/IOMMU support for GPU device passthrough to containers
• NVIDIA driver ioctl compatibility in the syscall layer
• K8s device plugin API support for GPU resource scheduling
• GPU resource requests and limits in pod specs
• Validation on AWS GPU instances (P/G series)
• Validation on GCP GPU instances (A2/G2 series)
• Validation on Azure GPU instances (NC/ND series)
• Cloud-specific accelerator support (AWS Inferentia/Trainium, GCP TPU, Azure AMD GPUs)

Phase 12 — Production operations (Not started)

Goal: Day-2 operational tooling for running ThemeliOS nodes in production.

Deliverables:

• Metrics export in Prometheus format (node-exporter compatible)
• Log forwarding to external collectors (CloudWatch, Stackdriver, Fluentd)
• Health endpoints for load balancers and orchestrators
• Distributed tracing support for container workloads
• A/B partition scheme for whole-image OS updates
• Automatic rollback on failed updates
• Zero-downtime node upgrades (drain → swap image → rejoin cluster)
• OS update tooling (cargo xtask image --update or equivalent)
• Update coordination with K8s (respect PodDisruptionBudgets during upgrades)