Chapter 23: AI/ML Policy Framework¶
Companion to Chapter 22: AI/ML and Accelerators. This chapter contains §23.1: AI/ML Policy Framework — Closed-Loop Kernel Intelligence. See 22-accelerators.md for §22.1–§22.6 (hardware accelerator framework).
Phase 5c. Companion to Chapter 22: AI/ML and Accelerators (Section 22.1). This chapter specifies the closed-loop kernel intelligence framework: ML models that observe kernel telemetry, predict workload behavior, and adjust scheduling, memory, and I/O policies in real time. All policy decisions are bounded by hard safety invariants — ML suggestions are advisory, never authoritative over safety-critical paths.
23.1 AI/ML Policy Framework: Closed-Loop Kernel Intelligence¶
Section 22.6 defines the in-kernel inference engine — a tiny integer-only model running on the hot path (~500-5000 cycles). Section 22.6 defines Tier 2 inference services — more powerful models in userspace. But these sections describe the inference plumbing, not the policy integration: what kernel knobs can ML actually adjust, how does the kernel emit telemetry so ML has data to learn from, and how do results from an external "big model" (LLM, large transformer, RL agent) flow back into the kernel to affect scheduler behavior until the next tuning cycle?
This section defines the complete closed-loop framework.
23.1.1 Design Principles¶
Advisory, not authoritative. Every ML-adjusted parameter has a heuristic fallback. If the ML layer crashes, misbehaves, or is absent, the kernel runs its built-in heuristics. ML improves average-case performance; it never replaces correctness.
Bounded parameters. Each tunable parameter has [min, max] bounds enforced at the
kernel enforcement point. An ML model cannot set eevdf_weight_scale = 1000000 any more
than a sysctl can — the kernel clamps it. This makes the parameter space safe even if
the model is adversarially manipulated.
Temporal decay. Parameters set by ML automatically revert to their defaults after
decay_period_ms milliseconds without a refresh. If the Tier 2 service crashes or
becomes unreachable, the kernel gradually returns to baseline behavior. No explicit
"ML service is down" signal is needed.
Deterministic replay. For bug reproduction, ml_policy=disabled on the kernel
command line disables all Tier 2 observation delivery and parameter updates. All
tunable parameters hold their default values. Two machines with identical hardware
and workload produce identical scheduling/memory/network behavior — the ML layer
introduces no non-determinism.
Workload-specific, not global. ML tuning is cgroup-scoped: the ML layer can set different parameters for a latency-sensitive web server cgroup and a batch analytics cgroup running on the same machine. Global parameters exist but are the minority.
Latency tiers for AI decisions:
| Tier | Latency | Mechanism | Examples |
|---|---|---|---|
| A | < 1 μs | Pure heuristic | Page fault handler, IRQ routing |
| B | 1–50 μs | In-kernel model (Section 22.6) | Page prefetch stride, I/O queue reorder |
| C | 50 μs–5 s | Tier 2 service round-trip | NUMA migration, compression selection, power budget |
| D | 5 s–5 min | Tier 2 + external "big model" | Scheduler workload characterization, full EAS recalibration, anomaly root-cause |
Tier classification for ML policy services — Unlike generic drivers, the ML policy services described in this chapter are architecturally constrained to run at effective Tier 2 (Ring 3 process). This is not a violation of the tier-agnostic driver model (Section 11.3); it is expressed through the manifest fields (Section 12.6): ML policy service manifests declare
minimum_tier = 2, maximum_tier = 2. The tier-selection algorithm in Section 11.3 then forceseffective_tier = 2on every architecture. The constraint exists because ML inference workloads require:
- Floating-point and SIMD execution (unavailable in Ring 0 kernel context, which runs with lazy FPU save)
- User-space ML frameworks (PyTorch, ONNX Runtime, XGBoost) that cannot be linked into kernel code
- GPU and accelerator access through user-space device libraries (CUDA, ROCm, OneAPI)
- Crash isolation strong enough to tolerate in-process model divergence without impacting kernel integrity
None of these are available in Core or in kernel-space isolation domains. The
minimum_tier = 2constraint is the manifest's way of declaring this hardware/software boundary.Subsequent uses of "Tier 2 policy service" in this chapter refer to services whose manifest enforces this constraint. This exception applies ONLY to ML policy services (and their equivalents — Section 22.6, Section 7.10); it is NOT a precedent for generic drivers.
23.1.2 Kernel Observation Bus¶
Every kernel subsystem that participates in ML tuning emits observations via a zero-cost macro. Observations are stored in per-CPU ring buffers and consumed asynchronously by Tier 2 policy services.
// umka-nucleus/src/ml/observation.rs
/// Identifies the emitting subsystem.
#[repr(u16)]
/// **Note**: Discriminant 0 is intentionally reserved (no subsystem uses it).
/// `ObservationRingSet.rings` is indexed by `subsystem as usize`, making
/// `rings[0]` a permanently null dead slot. This wastes 8 bytes per CPU
/// but avoids off-by-one index arithmetic on every `observe_kernel!` call.
pub enum SubsystemId {
Scheduler = 1,
MemoryManager = 2,
TcpStack = 3,
BlockIo = 4,
PowerManager = 5,
FmaHealth = 6,
NvmeDriver = 7,
NetworkDriver = 8,
IoScheduler = 9,
Gpu = 10,
Storage = 11,
Accel = 12,
VfsLayer = 13,
ContainerMgr = 14,
Kvm = 15,
// ID 16 reserved for a future subsystem.
// ParamId encoding uses bits [11:8] for the subsystem index (4 bits, max 16
// groups). SubsystemId is repr(u16) for wire compatibility, but ParamId
// limits the active range to 16 subsystems.
//
// **Overflow plan**: If all 16 groups are consumed and a new subsystem
// requires ML tuning, increment `PolicyRegisterRequest::version` to v2
// and widen the subsystem field to bits [13:8] (6 bits, max 64 groups).
// The ML policy service FD interface is versioned; both sides negotiate
// the version at registration time. The per-subsystem param field narrows
// to bits [5:0] (max 64 params per group), which is sufficient (current
// maximum usage is ~40 params in the Scheduler group). Alternatively,
// merge low-utilization groups (e.g., FmaHealth and PowerManager share
// similar telemetry patterns).
}
impl SubsystemId {
/// Derive the subsystem from a `ParamId` discriminant value.
/// ParamId encoding: bits [11:8] = subsystem index (4 bits, max 16 groups), bits [7:0] = param
/// within subsystem. E.g., ParamId 0x0001 = Scheduler param 1,
/// 0x0102 = MemoryManager param 2, 0x0200 = TcpStack param 0.
/// Returns `None` for out-of-range group indices. Callers must handle
/// the None case (dispatch_to_subsystem returns PolicyError::InvalidParam;
/// the registration path rejects the param).
pub const fn from_param_id(id: ParamId) -> Option<Self> {
// Extract upper byte and map to SubsystemId discriminant.
// ParamId discriminants use 0x00xx for Scheduler (SubsystemId=1),
// 0x01xx for MemoryManager (SubsystemId=2), etc.
// The +1 accounts for SubsystemId discriminants starting at 1, not 0.
// ParamId group byte 0x00 → SubsystemId 1 (Scheduler),
// group byte 0x01 → SubsystemId 2 (MemoryManager), etc.
let group = ((id as u32 >> 8) & 0x0F) as u16 + 1;
match group {
1 => Some(SubsystemId::Scheduler),
2 => Some(SubsystemId::MemoryManager),
3 => Some(SubsystemId::TcpStack),
4 => Some(SubsystemId::BlockIo),
5 => Some(SubsystemId::PowerManager),
6 => Some(SubsystemId::FmaHealth),
7 => Some(SubsystemId::NvmeDriver),
8 => Some(SubsystemId::NetworkDriver),
9 => Some(SubsystemId::IoScheduler),
10 => Some(SubsystemId::Gpu),
11 => Some(SubsystemId::Storage),
12 => Some(SubsystemId::Accel),
13 => Some(SubsystemId::VfsLayer),
14 => Some(SubsystemId::ContainerMgr),
15 => Some(SubsystemId::Kvm),
// Group 16 reserved for a future subsystem.
_ => None,
}
}
}
/// Compact observation emitted by a kernel subsystem.
/// 64 bytes total — fits in one cache line. align(64) enforces
/// cache-line alignment for per-CPU ring buffer allocations.
#[repr(C, align(64))]
pub struct KernelObservation {
pub timestamp_ns: u64, // Monotonic clock (TSC-derived)
pub subsystem: SubsystemId, // Source subsystem
pub obs_type: u16, // Subsystem-defined event type (see tables below)
pub cpu_id: u16, // CPU where the event occurred
pub _pad: u16,
pub cgroup_id: u64, // Originating cgroup (0 = kernel/no cgroup)
pub features: [i32; 10], // Up to 10 integer feature dimensions.
// **Truncation warning**: values wider than 32 bits (e.g.,
// nanosecond timestamps, byte counts) must be scaled before
// storing. Use `(value >> SHIFT) as i32` or `(value / SCALE) as i32`
// to fit in 32 bits. The 64-byte cache-line constraint prevents
// using i64 (would reduce to 5 features). Subsystems needing
// wide values should use relative deltas or per-unit scaling.
// Slot assignments are subsystem-specific and defined by
// `define_feature_extractor!` declarations below. Each
// `obs_type` has its own feature layout — see the per-subsystem
// observation type tables (Scheduler §6, Memory §4, TCP, Block I/O,
// etc.) for the authoritative slot-to-metric mapping. Unused
// trailing slots are zero-filled by `collect_features!`.
}
const_assert!(core::mem::size_of::<KernelObservation>() == 64);
/// Number of 64-bit words in one `KernelObservation`.
pub const OBSERVATION_WORDS: usize = core::mem::size_of::<KernelObservation>() / 8;
// The word view must be total: a partial trailing word could publish an
// uninitialized byte into a service-readable mapping.
const_assert!(core::mem::size_of::<KernelObservation>() % 8 == 0);
/// One observation ring slot, as the producer and every consumer actually
/// access it: an array of atomic words.
///
/// **Why the payload is atomic words and not a plain `KernelObservation`.**
/// The overwrite-oldest policy means the producer may write a slot while a
/// consumer is copying it — that overlap is intended and is what the
/// head-window check exists to detect. But detection is not enough on its own:
/// a plain (or `volatile`) 64-byte write overlapping a plain (or `volatile`)
/// 64-byte read is a DATA RACE, and `volatile` does not change that.
/// `read_volatile`/`write_volatile` constrain only whether the compiler may
/// elide, duplicate, or reorder the access — they make it neither indivisible
/// nor race-free, and a 64-byte access is several machine accesses on every
/// supported architecture. The race is undefined behaviour before any
/// after-the-fact check can run, and the compiler is entitled to assume it
/// cannot happen.
///
/// Word-wise atomic access removes the race at its root: every access to
/// shared memory is an atomic operation, so concurrent producer and consumer
/// accesses are well-defined by construction, and what the head-window check
/// then decides is only whether the (race-free) word set the consumer read is
/// a COHERENT observation or a mixture of two. That is a correctness question
/// about data, which a check can answer — not a soundness question, which it
/// cannot.
///
/// `Relaxed` on the words is sufficient and is the whole point: the ordering
/// that matters is carried by the producer's `Release` publish of `head` and
/// the consumer's `Acquire` loads of it around the copy.
///
/// Byte-identical to `KernelObservation` — same size, same alignment, same
/// field offsets — so the shared mapping layout, and every service that parses
/// it, is unchanged.
#[repr(C, align(64))]
pub struct ObservationSlot {
/// The observation, one naturally-aligned 64-bit word per element.
pub words: [AtomicU64; OBSERVATION_WORDS],
}
const_assert!(core::mem::size_of::<ObservationSlot>() == 64);
// The byte-identity the shared mapping depends on, asserted rather than
// asserted-by-comment: a future field added to one and not the other would
// silently reinterpret every slot a service reads.
const_assert!(
core::mem::size_of::<ObservationSlot>() == core::mem::size_of::<KernelObservation>()
);
const_assert!(
core::mem::align_of::<ObservationSlot>() == core::mem::align_of::<KernelObservation>()
);
impl ObservationSlot {
/// Producer side: publish `obs` into this slot, word by word, `Relaxed`.
///
/// The word view is a `#[repr(C)]` reinterpretation. `KernelObservation`
/// has no padding holes — every field is naturally aligned and the one gap
/// the layout would otherwise have is the explicit `_pad` field — so the
/// reinterpretation is total and no uninitialized byte is ever published
/// into a mapping a service can read.
#[inline]
pub fn store_obs(&self, obs: &KernelObservation) {
let words: &[u64; OBSERVATION_WORDS] = unsafe { core::mem::transmute(obs) };
for i in 0..OBSERVATION_WORDS {
self.words[i].store(words[i], Ordering::Relaxed);
}
}
/// Consumer side: copy this slot out, word by word, `Relaxed`. The caller
/// validates coherence with the head window before using the result.
#[inline]
pub fn load_obs(&self) -> KernelObservation {
let mut words = [0u64; OBSERVATION_WORDS];
for i in 0..OBSERVATION_WORDS {
words[i] = self.words[i].load(Ordering::Relaxed);
}
unsafe { core::mem::transmute(words) }
}
}
/// Per-(CPU, subsystem) observation ring — the KERNEL-PRIVATE descriptor.
/// Never mapped into any service address space.
///
/// **Single producer, many independent consumers.** The owning CPU is the sole
/// producer. Every subscriber to this subsystem owns its OWN cursor
/// (`ObservationCursor`) in the shared header and advances only that cursor, so
/// N services — plus the kernel-internal summary aggregator — read the same
/// published stream without contending for one shared read pointer. This is
/// what makes the framework's own shipped service set expressible: registration
/// admits up to `MAX_POLICY_SERVICES` services and several of them subscribe to
/// overlapping subsystems (`umka-ml-numa` and `umka-ml-compress` both consume
/// MemoryManager observations; `umka-ml-sched` and `umka-ml-intent` both consume
/// scheduler observations). Because the overflow policy is overwrite-oldest, the
/// producer never waits on any cursor, so adding consumers costs the producer
/// nothing: the hot path performs exactly one slot store, one head publish and
/// one cursor-lag probe regardless of subscriber count.
///
/// **Shared-mapping layout** — the same `[header][entries]` idiom the policy
/// update ring below uses:
///
/// ```text
/// offset 0 : ObservationRingShared (occupies one page, mapped RW)
/// offset PAGE_SIZE : [ObservationSlot; capacity] (mapped RO)
/// ```
///
/// `ObservationSlot` is byte-identical to `KernelObservation` (same size,
/// alignment and field offsets) — it names how the bytes are ACCESSED, not a
/// different layout — so a service parses the slot array exactly as before.
///
/// The header gets a whole page because the two protections must be separable:
/// subscribers WRITE their cursors, and nothing may write the slot array.
/// `PAGE_SIZE` is `arch::current::mm::PAGE_SIZE` ([Section 4.1](04-memory.md#boot-allocator)), so the
/// split holds unchanged on 4 KiB, 16 KiB and 64 KiB page architectures.
///
/// **Nothing in the shared header is trusted.** Every kernel-side index is
/// derived from `capacity_mask` and `head` in THIS struct, which no service can
/// reach; the header carries only a published mirror of `head` plus the
/// service-written cursors. A service that scribbles its own header corrupts
/// only its own view of its own ring — it can never steer a kernel store out of
/// the slot array.
///
/// **Capacity** is a system-wide runtime value, NOT a per-registration one:
/// one ring per (CPU, subsystem) is shared by every subscriber to that
/// subsystem, so there is no per-service capacity to honour. It is read from
/// the `umka.ml.ring_capacity` boot parameter and re-settable at runtime via
/// `/sys/kernel/umka/ml/ring_capacity`; it must be a power of two in
/// `[OBSERVATION_RING_MIN_CAPACITY, OBSERVATION_RING_MAX_CAPACITY]` and
/// defaults to `OBSERVATION_RING_DEFAULT_CAPACITY`. Discovering the capacity at
/// runtime rather than baking it into the type is what lets the same kernel
/// serve a 128-core server (where 4096-entry rings are cheap) and a
/// memory-constrained embedded or laptop target (where 256-entry rings are the
/// difference between usable and unaffordable).
///
/// **Allocation**: header page + `capacity * size_of::<KernelObservation>()`
/// bytes, allocated with `vmalloc` on the (cold, sleepable) registration path.
/// Nothing is stack-constructed — at the default capacity the slot array alone
/// is 256 KiB, far past the 8-16 KiB kernel stack.
#[repr(align(64))]
pub struct ObservationRing {
/// `capacity - 1`. TRUSTED: `head & capacity_mask` is the sole bound on
/// every kernel store into `slots`, which is why it lives here and not in
/// the service-writable header.
pub capacity_mask: u64,
/// Producer write counter — the authoritative copy, mirrored into the
/// shared header after each slot write. Written only by this ring's CPU
/// under preemption-disable; `AtomicU64` rather than a plain field because
/// the producer reaches the ring through a shared `&ObservationRing`.
/// **Longevity**: u64 wraps after ~18.4 quintillion observations — at 10M
/// observations/sec per CPU, ~58,000 years. Wrapping arithmetic is correct:
/// `head - tail` computes pending entries via unsigned modular subtraction
/// regardless of wrap.
pub head: AtomicU64,
/// Kernel VA of the slot array (`capacity_mask + 1` entries).
pub slots: *mut ObservationSlot,
/// Kernel VA of the shared header page.
pub shared: *mut ObservationRingShared,
/// Live subscriber count. Incremented when a registration claims a cursor
/// slot, decremented at deregistration; the ring is freed (after an RCU
/// grace period) when it reaches zero. A later registration for the same
/// subsystem CLAIMS A CURSOR on the existing ring — it never replaces the
/// `ObservationRingSet::rings` pointer, so no earlier subscriber can be
/// deprived of its stream by a peer's registration.
pub subscribers: AtomicU32,
}
// SAFETY: `slots`/`shared` are raw pointers to `vmalloc` allocations that are
// published (Release) before the ring pointer becomes visible and freed only
// after the last subscriber is gone AND an RCU grace period has elapsed. They
// are never reassigned while the ring is reachable, so sharing `&ObservationRing`
// across CPUs (the aggregation softirq and the FMA health check both read it)
// exposes no unsynchronized mutation.
unsafe impl Sync for ObservationRing {}
/// Shared observation-ring header — the first `PAGE_SIZE` bytes of the mapping
/// handed to subscribers, mapped READ-WRITE (the slot array that follows is
/// mapped READ-ONLY). UNTRUSTED by the kernel: see `ObservationRing`.
#[repr(C, align(64))]
pub struct ObservationRingShared {
/// Slot count, published for the subscriber's index arithmetic.
pub capacity: u64,
/// Mirror of `ObservationRing::head`, republished with `Release` after
/// every slot write. Subscribers load it with `Acquire`.
pub head: AtomicU64,
/// Observations lost to overwrite-oldest on this ring (aggregate,
/// diagnostic). Feeds the `OBSERVATION_DROP_ALERT_THRESHOLD` FMA alert.
pub overflow: AtomicU64,
/// Cached minimum over the live cursors — the producer's SINGLE-load
/// overflow probe, so the hot path never walks the cursor array and its
/// cost does not grow with subscriber count. Recomputed exactly by the
/// 100 ms aggregation softirq (which walks the cursors anyway) and refined
/// opportunistically by any subscriber that advances past it. Staleness
/// affects only the diagnostic counters, never a kernel index.
pub slowest_tail: AtomicU64,
/// Explicit padding: fields end at offset 32; the cursor array starts on
/// the next cache line so producer writes never false-share with cursors.
pub _pad0: [u8; 32],
/// One cursor per subscriber slot. Slot 0 is reserved for the
/// kernel-internal summary aggregator; slots 1.. are claimed by
/// registrations.
pub cursors: [ObservationCursor; RING_SUBSCRIBER_SLOTS],
}
// 8+8+8+8 = 32 header words + 32 pad = 64, + 5 * 64 cursors = 384 bytes,
// occupying the first page of the shared mapping.
const_assert!(core::mem::size_of::<ObservationRingShared>() == 384);
const_assert!(core::mem::size_of::<ObservationRingShared>() <= PAGE_SIZE);
/// One subscriber's read cursor. Cache-line isolated because subscribers on
/// different CPUs write their own cursors concurrently.
#[repr(C, align(64))]
pub struct ObservationCursor {
/// Owning subscriber: a `PolicyServiceRegistration::service_id`, or
/// `CURSOR_SLOT_FREE` when unclaimed, or `CURSOR_SLOT_AGGREGATOR` for the
/// summary softirq. Kernel-written, service-read.
pub owner: AtomicU64,
/// This subscriber's read pointer. Written ONLY by its owner; read by the
/// aggregation softirq when it recomputes `slowest_tail`. Untrusted — a
/// bogus value costs its owner accurate drop accounting and nothing else.
pub tail: AtomicU64,
/// Explicit padding to the 64-byte cache line.
pub _pad: [u8; 48],
}
const_assert!(core::mem::size_of::<ObservationCursor>() == 64);
/// Maximum number of concurrently registered Tier 2 policy services. This is a
/// hard bound, not a boot default: it sizes the per-ring cursor array, so the
/// admission check (`EBUSY`) and the cursor table can never disagree.
pub const MAX_POLICY_SERVICES: usize = 4;
/// Cursor slots per ring: one per admissible service plus one for the
/// kernel-internal summary aggregator.
pub const RING_SUBSCRIBER_SLOTS: usize = MAX_POLICY_SERVICES + 1;
/// `ObservationCursor::owner` value for an unclaimed slot.
pub const CURSOR_SLOT_FREE: u64 = 0;
/// `ObservationCursor::owner` value for the kernel-internal summary aggregator.
/// `service_id` allocation therefore starts at 2.
pub const CURSOR_SLOT_AGGREGATOR: u64 = 1;
/// Smallest, largest and default observation-ring capacity in entries. Powers
/// of two; the effective value is chosen at boot from `umka.ml.ring_capacity`.
pub const OBSERVATION_RING_MIN_CAPACITY: usize = 256;
pub const OBSERVATION_RING_MAX_CAPACITY: usize = 65536;
pub const OBSERVATION_RING_DEFAULT_CAPACITY: usize = 4096;
The observe_kernel! macro emits observations with zero overhead when no consumer
is registered (one byte static key, branch predicted-not-taken). It depends on the
collect_features! helper macro to pack feature arguments into the fixed-size
[i32; 10] features array:
/// Packs up to 10 feature expressions into a `[i32; 10]` array for use in
/// `KernelObservation::features`. Accepts 1–10 positional expressions; unused
/// slots are zero-filled at compile time. All expressions must be convertible
/// to `i32`.
///
/// **Zero runtime cost**: The macro expands to a fixed-size array literal.
/// No Option, no branching, no function calls — just direct assignment
/// of the provided values and literal 0s for unused slots.
///
/// # Example
/// ```
/// let arr = collect_features!(latency_ns as i32, runqueue_len, cpu_id as i32);
/// // Expands to: [latency_ns as i32, runqueue_len as i32, cpu_id as i32, 0, 0, 0, 0, 0, 0, 0]
/// ```
macro_rules! collect_features {
// Entry: accept 1–10 comma-separated expressions.
($($feat:expr),+ $(,)?) => {{
// Count provided features at compile time.
const _N: usize = <[()]>::len(&[$(collect_features!(@unit $feat)),+]);
const _: () = assert!(_N <= 10, "collect_features! accepts at most 10 features");
// Build the array: provided values first, then zero-pad to 10.
// Each feature is routed through `feature_narrow` (a compile-time
// width gate, see below) and then cast via `as i32`. Values wider than
// 32 bits (u64, usize) MUST be pre-scaled before passing to this macro;
// the gate rejects them at the call site. To pass a wide value
// intentionally, use explicit scaling: `(value >> SHIFT) as i32` or
// `(value / SCALE) as i32`.
{
let mut arr = [0i32; 10];
let mut _i = 0;
$(
arr[_i] = feature_narrow($feat) as i32;
_i += 1;
)+
arr
}
}};
(@unit $e:expr) => { () };
}
/// Compile-time feature-width gate for `collect_features!`: an identity
/// function that fails to instantiate when its argument's type is wider than
/// `i32`.
///
/// **Why a function and not a `const` item.** The check must be performed on
/// the feature expression's TYPE, never on its VALUE. A `const` item expanded
/// into the macro body (`const _: () = assert!(size_of_val(&$feat) <= 4);`)
/// cannot name the call site's locals — a `const` initializer may not reference
/// a non-constant value — so every ordinary `observe_kernel!` call site, which
/// passes runtime variables, would fail to compile. Routing the expression
/// through a generic function moves the assertion into a context where only the
/// inferred type `T` is needed: the inline `const` block below is evaluated once
/// per monomorphisation and reports at the offending call site.
///
/// **Zero runtime cost**: `#[inline(always)]` identity; it lowers to nothing.
#[inline(always)]
pub fn feature_narrow<T: Copy>(v: T) -> T {
const {
assert!(
core::mem::size_of::<T>() <= 4,
"collect_features!: feature wider than i32 — pre-scale before passing"
);
}
v
}
Implementation note: In the generated code the optional-slot pattern above is expressed via a counting macro that emits the correct number of
, 0zero-fillers, not viaOption. The pseudocode above conveys the intent; the actual expansion uses a standard Rust macro repetition counter pattern. The result is always a[i32; 10]with no run-time cost — the array literal is zero-cost (compiled to static data); feature slot assignments execute at runtime (same cost as a direct array write). The macro itself expands at compile time; the populated feature values are computed at the observation call site (runtime).
23.1.2.1 Typed Feature Extraction: define_feature_extractor!¶
While collect_features! packs ad-hoc expressions into the [i32; 10] observation
array at emit time, the Tier 2 policy service needs the inverse: a typed extractor
that unpacks the [i32; 10] array back into named f32 fields for inference input.
The define_feature_extractor! macro generates a per-subsystem collect() function
that converts a KernelObservation into a fixed-size [f32; N] feature vector with
named accessors.
/// Define a typed feature extractor for a subsystem's observation data.
///
/// Generates:
/// - A `struct {Name}Features` with named `f32` fields.
/// - A `fn collect(obs: &KernelObservation) -> [f32; N]` that extracts and
/// converts the `[i32; 10]` features array into `[f32; N]`.
/// - A `const FEATURE_NAMES: [&str; N]` for debug/logging.
///
/// The generated code is zero-allocation (stack-only) and suitable for use
/// in Tier 2 userspace inference pipelines.
///
/// # Example
/// ```
/// define_feature_extractor! {
/// /// Scheduler feature extractor for TaskWoke observations.
/// SchedWakeFeatures for SubsystemId::Scheduler {
/// latency_ns: [0], // features[0] → latency in nanoseconds
/// runqueue_len: [1], // features[1] → runqueue length at wakeup
/// prev_cpu: [2], // features[2] → CPU the task last ran on
/// target_cpu: [3], // features[3] → CPU selected for wakeup
/// }
/// }
///
/// // Generated API:
/// // struct SchedWakeFeatures { pub latency_ns: f32, pub runqueue_len: f32, ... }
/// // impl SchedWakeFeatures {
/// // pub fn collect(obs: &KernelObservation) -> [f32; 4] { ... }
/// // pub const FEATURE_NAMES: [&str; 4] = ["latency_ns", "runqueue_len", ...];
/// // pub fn from_obs(obs: &KernelObservation) -> Self { ... }
/// // }
/// ```
macro_rules! define_feature_extractor {
(
$(#[$meta:meta])*
$name:ident for $subsystem:expr {
$( $field:ident : [$idx:expr] ),+ $(,)?
}
) => {
$(#[$meta])*
pub struct $name {
$( pub $field: f32, )+
}
impl $name {
/// Number of features extracted by this extractor.
pub const N: usize = {
// Count fields at compile time.
let mut _n = 0usize;
$( let _ = stringify!($field); _n += 1; )+
_n
};
/// Human-readable names for each feature dimension.
/// Useful for logging, model metadata, and ONNX input naming.
pub const FEATURE_NAMES: [&'static str; Self::N] = [
$( stringify!($field), )+
];
/// The subsystem this extractor is designed for.
/// Used for debug assertions in the inference pipeline.
pub const SUBSYSTEM: SubsystemId = $subsystem;
/// Extract features from a `KernelObservation` into a fixed-size
/// `[f32; N]` array suitable for direct use as model input.
///
/// Each `features[idx]` is cast from `i32` to `f32`. The conversion
/// is lossless for values in the range `[-2^24, 2^24]` (±16M), which
/// covers all kernel observation values (latencies in ns are capped at
/// `i32::MAX` ≈ 2.1s; queue lengths, CPU IDs, and byte counts fit easily).
///
/// # Panics (debug only)
/// Debug-asserts that `obs.subsystem == Self::SUBSYSTEM`.
#[inline]
pub fn collect(obs: &KernelObservation) -> [f32; Self::N] {
debug_assert_eq!(
obs.subsystem, Self::SUBSYSTEM,
"feature extractor subsystem mismatch"
);
[ $( obs.features[$idx] as f32, )+ ]
}
/// Extract features into a named struct for ergonomic access.
#[inline]
pub fn from_obs(obs: &KernelObservation) -> Self {
debug_assert_eq!(
obs.subsystem, Self::SUBSYSTEM,
"feature extractor subsystem mismatch"
);
Self {
$( $field: obs.features[$idx] as f32, )+
}
}
}
};
}
// Concrete extractors for each subsystem's primary observation type.
define_feature_extractor! {
/// Scheduler wakeup features (SchedObs::TaskWoke).
SchedWakeFeatures for SubsystemId::Scheduler {
latency_ns: [0],
runqueue_len: [1],
prev_cpu: [2],
target_cpu: [3],
}
}
define_feature_extractor! {
/// Memory page fault features (MemObs::PageFault).
MemFaultFeatures for SubsystemId::MemoryManager {
fault_addr_page: [0],
fault_flags: [1],
alloc_latency: [2],
numa_node: [3],
lru_gen: [4],
}
}
define_feature_extractor! {
/// OOM victim selection features (MemObs::OomVictimSelection).
/// Emitted when the OOM killer is invoked, before victim selection.
OomVictimFeatures for SubsystemId::MemoryManager {
constraint_type: [0], // OOM trigger class: 0=Global, 1=CgroupLimit, 2=Global+MPOL_BIND nodemask (from OomContext.scope/.nodemask, [Section 4.5](04-memory.md#oom-killer))
candidate_count: [1], // Number of eligible tasks
victim_pid: [2], // Selected victim PID
victim_rss: [3], // Victim RSS in pages
victim_swap: [4], // Victim swap in pages
victim_score: [5], // Computed OOM score (0-2000)
victim_cgroup_id: [6], // Victim cgroup ID
free_pages: [7], // System free pages at OOM time
psi_stall_us: [8], // PSI memory stall last 10s (µs)
}
}
define_feature_extractor! {
/// OOM outcome features (MemObs::OomOutcome).
/// Emitted 5 seconds after OOM kill to record actual recovery metrics.
/// This is the ground-truth signal for closed-loop ML training.
OomOutcomeFeatures for SubsystemId::MemoryManager {
victim_pid: [0], // PID of killed task
pages_freed: [1], // Pages actually reclaimed
time_to_recovery_ms: [2], // ms until PSI stall < 1%
service_restart_ms: [3], // ms until replacement task appeared
cascading_kills: [4], // Additional OOM kills within 10s
oom_count_after: [5], // memory.events.oom 10s after
was_ml_adjusted: [6], // 1 if ML adjusted score, 0 if pure heuristic
baseline_score: [7], // Score without ML adjustment
}
}
define_feature_extractor! {
/// TCP stack features (TcpObs::SegmentArrival).
TcpSegmentFeatures for SubsystemId::TcpStack {
rtt_us: [0],
cwnd_segs: [1],
ssthresh_segs: [2],
in_flight: [3],
loss_count: [4],
}
}
define_feature_extractor! {
/// Block I/O features (BlockObs::RequestComplete).
BlockIoFeatures for SubsystemId::BlockIo {
latency_us: [0],
queue_depth: [1],
sector_count: [2],
is_write: [3],
}
}
The observe_kernel! macro emits observations with zero overhead when no consumer
is registered (one byte static key, branch predicted-not-taken):
Static key patching mechanism: Each observe_kernel! call site contains a NOP
instruction at compile time. When a Tier 2 policy service registers (or deregisters)
for a subsystem, the runtime patcher writes to OBSERVE_ENABLED[subsystem] and
rewrites each NOP to a short conditional jump (JMP) — or back to NOP. This is a
standard static branch / jump-label technique: the patching cost is paid once at
registration time, and every subsequent call-site check is a single correctly-predicted
branch with zero cache-miss overhead.
The static_key_enabled!(OBSERVE_ENABLED[subsystem]) check expands to the patched
branch instruction. Per-architecture patching details:
| Architecture | Disabled (NOP) | Enabled (branch) | Instruction size | Patching mechanism |
|---|---|---|---|---|
| x86-64 | 0x90 (NOP) or 0x0F 0x1F 0x00 (3-byte NOP) |
JMP rel8 or JNE rel8 |
2-5 bytes | arch::current::text_patch: INT3 breakpoint → write → remove INT3 |
| AArch64 | NOP (0xD503201F) |
B <offset> (0x14xxxxxx) |
4 bytes | IPI all CPUs → dc cvau + ic ivau → dsb ish; isb |
| ARMv7 | NOP (0xE320F000) |
B <offset> (0xEAxxxxxx) |
4 bytes | stop_machine() → write → flush I-cache (mcr p15, 0, r0, c7, c5, 0) |
| RISC-V | NOP (0x00000013) |
JAL x0, <offset> (J-type) |
4 bytes | IPI fence.i on all harts (RISC-V has no coherent I-cache guarantee) |
| PPC32 | nop (0x60000000) |
b <offset> (0x48xxxxxx) |
4 bytes | stop_machine() → patch → isync on each CPU |
| PPC64LE | nop (0x60000000) |
b <offset> (0x48xxxxxx) |
4 bytes | Same as PPC32 (stop_machine + isync) |
| s390x | bc 0,0 (NOP, 0x47000000) |
brc <mask>, <offset> (0xA7x4xxxx) |
4-6 bytes | IPI all CPUs via SIGP → patch → bcr 14,0 (serializing) on each CPU |
| LoongArch64 | andi $zero,$zero,0 (NOP) |
b <offset> |
4 bytes | IPI all CPUs → patch → ibar 0 (instruction barrier) on each CPU |
The StaticKey type and its static_key_enable()/static_key_disable() /
static_key_enabled! API are defined in
Section 3.6; the
per-arch patching is handled by arch::current::text_patch. The ML policy
subsystem uses that generic facility — the OBSERVE_ENABLED array is a Tier-0
static [StaticKey; N], and each observe_kernel! expansion is a jump-label
site registered against the corresponding key.
/// Per-subsystem observation gates, indexed by `SubsystemId as usize`.
///
/// Tier-0 static: the gate is tested on every `observe_kernel!` hot path, so the
/// keys live in the kernel image behind no allocation. Each key starts disabled
/// — every call site is a NOP — and is flipped to enabled by the runtime patcher
/// when a Tier 2 policy service first registers for that subsystem, and back to
/// disabled on the last deregistration. `ml_policy=disabled` on the kernel
/// command line leaves every key permanently disabled. `StaticKey` and its API
/// are defined in [Section 3.6](03-concurrency.md#lock-free-data-structures--static-keys-zero-overhead-static-branches);
/// `MAX_SUBSYSTEM_ID` is the same bound used by `POLICY_HANDLERS` below.
static OBSERVE_ENABLED: [StaticKey; MAX_SUBSYSTEM_ID] =
[const { StaticKey::new_disabled() }; MAX_SUBSYSTEM_ID];
/// Emit a kernel observation.
/// Overhead when disabled: 1–3 cycles (static branch miss rate ~0%).
/// Overhead when enabled: ~10–30 cycles (TSC read + ring buffer write).
///
/// # Example
/// ```
/// observe_kernel!(SubsystemId::Scheduler, SchedObs::TaskWoke,
/// cgroup_id, latency_ns as i32, runqueue_len, prev_cpu);
/// ```
macro_rules! observe_kernel {
($subsystem:expr, $obs_type:expr, $cgroup:expr, $($feat:expr),+) => {{
// Static key: single .byte 0x90 (NOP) patched to JMP when no consumer.
// Runtime patcher changes this when a Tier 2 service registers.
if static_key_enabled!(OBSERVE_ENABLED[$subsystem as usize]) {
// Preemption must be disabled while accessing the per-CPU ring.
// The macro acquires a preempt_disable guard to ensure the CPU ID
// read and the ring push are on the same CPU. Without this guard,
// a preemption between cpu_local::cpu_id() and push_current_cpu()
// could push to the wrong CPU's ring (correctness issue: the ring
// has exactly one producer — the CPU that owns it).
let __preempt_guard = preempt_disable();
let __obs = KernelObservation {
timestamp_ns: crate::arch::current::cpu::read_cycle_counter_ns(),
subsystem: $subsystem,
obs_type: $obs_type as u16,
cpu_id: cpu_local::cpu_id() as u16,
_pad: 0,
cgroup_id: $cgroup,
features: collect_features!($($feat),+),
};
ObservationRing::push_current_cpu($subsystem, __obs, &__preempt_guard);
drop(__preempt_guard);
}
}}
}
ObservationRing::push_current_cpu specification:
/// Per-CPU observation ring sets.
///
/// `PerCpu<T>`, deliberately NOT a `CpuLocalBlock` field: `CpuLocalBlock` is
/// reserved for the ~10 hottest scalars and has never carried an observation
/// field, and its whole-block accessors were DELETED by ESC-0431
/// ([Section 3.2](03-concurrency.md#cpulocal-register-based-per-cpu-fast-path)) — no `&CpuLocalBlock` may
/// be formed by runtime code under any precondition. "All other per-CPU data
/// belongs in `PerCpu<T>`" is that section's own rule, and a per-subsystem
/// array of ring pointers is exactly such data. `PerCpu::get()` requires the
/// `&PreemptGuard` the `observe_kernel!` expansion already holds, which is
/// precisely the CPU-pinning proof the ring's single-producer contract needs —
/// so threading the guard costs nothing and makes the contract checkable.
///
/// Populated once during ML-policy init (Phase 5c) with
/// `PerCpu::new(num_possible_cpus(), ObservationRingSet::empty)`, before any
/// `OBSERVE_ENABLED` key can be enabled — so every `observe_kernel!` that
/// reaches the push path finds it initialized.
static OBSERVATION_RINGS: OnceLock<PerCpu<ObservationRingSet>> = OnceLock::new();
impl ObservationRing {
/// Push an observation to the current CPU's ring for `subsystem`.
/// Called from the observe_kernel! macro on the hot path.
///
/// - Reaches this CPU's `ObservationRingSet` through `OBSERVATION_RINGS`,
/// whose accessor consumes the caller's `PreemptGuard` — the same guard
/// the static-key branch already took, so the CPU-ID read and the ring
/// push provably happen on one CPU.
/// - The ring is a circular buffer of `capacity_mask + 1` entries
/// (power-of-two, system-wide runtime value). `head` is the write index;
/// each subscriber holds its own read index in its `ObservationCursor`.
/// - **Overflow policy**: overwrite oldest. `head` advances unconditionally
/// with `Release` ordering. When the producer has lapped the slowest
/// cursor, the oldest entry is silently lost and both
/// `ObservationRingShared::overflow` and `ObservationRingSet::dropped`
/// are incremented (Relaxed). Because the probe reads the single cached
/// `slowest_tail` rather than walking the cursor array, its cost is
/// constant in the number of subscribers.
/// - No lock: one producer (this CPU), many independent consumers, none of
/// which the producer ever waits on.
///
/// **Slot access protocol** (normative). Every access to slot memory —
/// producer and consumer, in-kernel and mapped-service — is an ATOMIC WORD
/// access via `ObservationSlot::store_obs`/`load_obs`. No plain and no
/// merely-`volatile` access to a slot exists anywhere in this design.
///
/// That is what makes overwrite-oldest sound. The producer may overwrite a
/// slot an aggregator is mid-copy; because both sides touch the slot only
/// through relaxed atomic words, the overlap is well-defined rather than a
/// data race, and what the consumer can be left with is a MIXTURE of two
/// observations' words — never undefined behaviour. The head-snapshot
/// validation below then detects exactly that mixture and discards it. The
/// two mechanisms have distinct jobs and neither substitutes for the other:
/// atomics make the concurrent access legal, the head window makes the
/// result trustworthy. A `volatile` access would have provided neither —
/// `volatile` is not atomic, and validating after a racy access does not
/// un-race it.
///
/// No `&`/`&mut` to a `KernelObservation` in the slot array is ever formed;
/// the only reference that exists is `&ObservationSlot`, which grants
/// nothing but atomic operations and is therefore safe to hold while
/// another agent writes the same slot.
pub fn push_current_cpu(
subsystem: SubsystemId,
obs: KernelObservation,
guard: &PreemptGuard,
) {
let Some(rings) = OBSERVATION_RINGS.get() else {
return; // ML policy subsystem not initialized.
};
let ring_set = rings.get(guard);
let ring_ptr = ring_set.rings[subsystem as usize].load(Ordering::Acquire);
if ring_ptr.is_null() {
return; // Category not active — no ring allocated.
}
// SAFETY: a non-null slot holds a ring published (Release) by the
// registration path; it is freed only after its last subscriber is
// dropped AND an RCU grace period has elapsed (deregistration steps
// 1-2), and this section runs with preemption disabled, so the ring
// cannot be freed under us.
let ring = unsafe { &*ring_ptr };
let head = ring.head.load(Relaxed);
let idx = head & ring.capacity_mask;
// SAFETY: `idx <= capacity_mask` by construction and `slots` holds
// `capacity_mask + 1` entries, so the store is in bounds — and the
// bound comes from the kernel-private descriptor, never from the
// service-writable header. This CPU is the ring's only producer.
// `&ObservationSlot` grants only atomic word operations, so forming it
// while a consumer reads the same slot is sound.
let slot = unsafe { &*ring.slots.add(idx as usize) };
slot.store_obs(&obs);
// Publish: authoritative counter first, then the subscriber-visible
// mirror. Consumers see the new entry after the mirror's Release store.
let next = head.wrapping_add(1);
ring.head.store(next, Release);
// SAFETY: `shared` points at the ring's header page, allocated with the
// ring and freed with it.
let shared = unsafe { &*ring.shared };
shared.head.store(next, Release);
// Overflow accounting: ONE load, against the lazily maintained minimum
// cursor. `next - slowest > capacity_mask` means the producer has
// lapped the slowest subscriber.
let slowest = shared.slowest_tail.load(Relaxed);
if next.wrapping_sub(slowest) > ring.capacity_mask {
shared.overflow.fetch_add(1, Relaxed);
ring_set.dropped.fetch_add(1, Relaxed);
}
}
}
/// Consumer read protocol — used identically by a Tier 2 subscriber reading
/// its mapping and by the kernel-internal summary aggregator:
///
/// **Torn-read protection**: a `KernelObservation` is 64 bytes and no
/// architecture writes 64 bytes indivisibly, so the producer's
/// overwrite-oldest policy can leave a consumer holding words from two
/// different observations. The copy itself is race-free — every word goes
/// through `ObservationSlot::load_obs`, i.e. relaxed atomic loads paired with
/// the producer's relaxed atomic stores — so what remains to be decided is
/// only whether the word set is COHERENT. Each consumer reads
/// through its OWN cursor and uses a head-snapshot validation pattern
/// (Linux's `perf_output_put_handle` uses the same shape for the same reason —
/// reference, not authority: UmkaOS diverges in that one published stream
/// serves many cursors rather than one ring per consumer):
///
/// ```pseudo
/// fn read_observations(
/// shared: &ObservationRingShared,
/// slots: *const ObservationSlot, // mapping base + PAGE_SIZE
/// me: usize, // this consumer's cursor slot
/// buf: &mut [KernelObservation],
/// ) -> usize {
/// let mask = shared.capacity - 1;
/// let cursor = &shared.cursors[me];
/// let tail = cursor.tail.load(Relaxed); // OUR read pointer only
/// let head = shared.head.load(Acquire); // kernel write pointer
/// let available = head.wrapping_sub(tail);
/// let to_read = available.min(buf.len() as u64);
/// let mut count = 0;
/// for i in 0..to_read {
/// let idx = tail.wrapping_add(i) & mask;
/// // Copy the slot out word-wise (relaxed atomics — race-free even if
/// // the producer is writing this very slot). Snapshot head around it.
/// let head_before = shared.head.load(Acquire);
/// buf[count] = unsafe { (&*slots.add(idx as usize)).load_obs() };
/// // fence(Acquire) — ensure the data read completes before re-reading head.
/// let head_after = shared.head.load(Acquire);
/// // If head advanced past our slot during the copy, the words we read
/// // came from two different observations — a MIXED, not torn-unsound,
/// // read:
/// // head_after - (tail + i) >= capacity means the producer lapped us
/// // and overwrote the slot we just copied.
/// if head_after.wrapping_sub(tail.wrapping_add(i)) <= mask {
/// count += 1; // valid read
/// }
/// // else: torn observation, skip silently
/// }
/// cursor.tail.store(tail.wrapping_add(to_read), Release);
/// // Opportunistically refine the producer's overflow probe.
/// let slowest = shared.slowest_tail.load(Relaxed);
/// if tail.wrapping_add(to_read).wrapping_sub(slowest) < mask {
/// shared.slowest_tail.store(tail.wrapping_add(to_read), Relaxed);
/// }
/// count
/// }
/// ```
///
/// `head_before` participates only as the pre-copy Acquire that orders the
/// slot copy after the producer's publish; the discard decision uses
/// `head_after`. A consumer that finds `head - tail > capacity` has lost
/// entries wholesale and resynchronizes by storing `head - capacity` into its
/// own cursor — no other consumer is affected.
/// Threshold for triggering an FMA health alert on excessive observation drops.
/// Measured as drops per second per ring (evaluated by the periodic health check
/// timer, not inline on the hot path).
pub const OBSERVATION_DROP_ALERT_THRESHOLD: u64 = 1000;
When dropped exceeds OBSERVATION_DROP_ALERT_THRESHOLD (1000 per second per ring),
an FMA health event is raised with HealthEventClass::Generic. The policy
service can respond by reducing observation frequency via the ObservationThrottleMsg
control message.
Aggregation. Tier 2 services typically consume raw observations and aggregate them into feature vectors over configurable windows (100ms, 1s, 10s, 60s). The ring buffer provides raw data; aggregation policy is entirely in Tier 2 userspace.
23.1.2.2 In-Domain Probe Points: Deferred Emission¶
observe_kernel! touches Core-image data on every enabled emission: the
per-CPU ObservationRingSet (reached through the CPU-local block), the
ring pages themselves, and the calibrated monotonic clock behind
read_cycle_counter_ns(). Probe points that fire while a non-Core module
image is live — the per-call / per-notification observation points inside
a cross-domain consumer loop's in-domain phases
(Section 12.8) — therefore MUST NOT
call observe_kernel! directly on architectures whose isolation is
switched by a per-CPU register: every Core-data load there is either a
fault or, worse, a silent isolation-discipline violation (the
consumer in-domain access discipline,
Section 12.8). Such
probe points capture into a stack-resident deferral buffer and emit at
the next Core-image window.
Capture record. In-domain capture may read the raw cycle counter (a pure register read — always legal) but NOT the ns-calibration data (Core resident). Records therefore carry raw cycles; conversion happens at flush:
/// One deferred in-domain observation sample. Kernel-internal, not KABI —
/// lives only on the capturing loop's stack, never crosses a compilation
/// or domain boundary (the repr(C) + const_assert exist to pin the
/// stack-budget arithmetic below, not for ABI).
///
/// Byte layout: capture_cycles(8) + delta_cycles(8) + arg0(4) + arg1(4) +
/// subsystem(2) + obs_type(2) + _pad(4) = 32 bytes.
#[repr(C)]
pub struct DeferredObservation {
/// Raw cycle-counter value at the probe point
/// (`arch::current::cpu::read_cycle_counter()`). Converted to the
/// observation's `timestamp_ns` at flush — the emitted observation
/// carries its CAPTURE time, never the flush time.
pub capture_cycles: u64,
/// Raw cycle delta for latency-class probes (`current_cycles -
/// entry.timestamp`, both raw); 0 for non-latency probes. Converted
/// to ns at flush and scaled per the probe class's feature-slot table.
pub delta_cycles: u64,
/// First identifier feature of the probe class (e.g. `domain_id`,
/// `vector`) — plain values, no Core access needed to produce them.
pub arg0: i32,
/// Second identifier feature (e.g. `method_index`, `driver_id`,
/// `timer_id`, truncated `cookie`).
pub arg1: i32,
/// Emitting subsystem (`SubsystemId` is `repr(u16)`).
pub subsystem: SubsystemId,
/// Subsystem-defined event type (same namespace as
/// `KernelObservation::obs_type`).
pub obs_type: u16,
/// Explicit tail padding (offset 28..32).
pub _pad: [u8; 4],
}
const_assert!(core::mem::size_of::<DeferredObservation>() == 32);
/// Deferral capacity per in-domain window. Sized to the consumer loops'
/// batch bound (`CONSUMER_MAX_BATCH = 32`,
/// [Section 12.8](12-kabi.md#kabi-domain-runtime)): each ring entry / IRQ notification emits
/// at most ONE in-domain sample today (dispatch latency for kabi
/// entries; exactly one of IRQ/timer/completion latency per
/// notification), so 32 covers a full window with zero drops. Any
/// future probe that can emit twice per entry must double this or
/// accept saturation drops.
pub const OBS_DEFER_CAP: usize = 32;
/// The per-window deferral buffer: a stack local of the consumer loop's
/// frame. 32 × 32 B = 1 KiB of stack — bounded and acceptable for the
/// dedicated, shallow-stack consumer kthreads that host in-domain
/// windows (~6-12% of the smallest 8 KiB kernel stack); NOT for
/// arbitrary deep call paths, which is fine because only consumer-loop
/// probe points are in-domain.
pub type ObsDeferBuf = ArrayVec<DeferredObservation, OBS_DEFER_CAP>;
Capture (obs_defer) — callable from ANY context, including
in-domain Phases 2-4 (touches only the stack buffer and a register
read):
/// Record an in-domain sample into the window's deferral buffer.
/// Overflow policy: saturate (drop-newest) and count into `local_drops`.
/// Rationale for diverging from the ring's overwrite-oldest policy: the
/// flush ALWAYS drains the whole buffer at the window's end, so there is
/// no lagging consumer for overwrite-oldest to protect against — the
/// two policies differ only in WHICH bounded subset survives, and
/// drop-newest avoids ring-index bookkeeping on the stack buffer. The
/// observable loss signal is identical: a count folded into the same
/// `ObservationRingSet::dropped` counter the ring's own overflow uses
/// (and the same `OBSERVATION_DROP_ALERT_THRESHOLD` FMA alert).
#[inline(always)]
pub fn obs_defer(
buf: &mut ObsDeferBuf,
local_drops: &mut u32,
sample: DeferredObservation,
) {
if buf.try_push(sample).is_err() {
*local_drops = local_drops.saturating_add(1);
}
}
Capture-side gating: the static_key_enabled!(OBSERVE_ENABLED[...])
check IS legal in-domain — it is a patched branch instruction in kernel
text (instruction fetch, no data load; kernel text remains executable
under a live module image). In-domain probe points therefore keep the
same zero-cost gate as observe_kernel! and call obs_defer() only when
a consumer is registered — a disabled subsystem defers nothing.
Flush (obs_defer_flush) — Core-image windows ONLY:
/// Drain the deferral buffer into the per-CPU observation rings.
///
/// # Context contract
/// Core-image execution only (the caller's live isolation image is the
/// Core image): converts raw cycles via `arch::current::cpu::cycles_to_ns()`
/// — the same mult/shift calibration `read_cycle_counter_ns()` uses,
/// Core-resident — and calls `ObservationRing::push_current_cpu()`
/// (CPU-local Core data). Preemption disabled by the push path's
/// contract; no allocation, no blocking, no locks.
///
/// # Mandatory flush points (per consumer loop)
/// 1. **Phase 4.5** (the batch boundary, Core transit state) — the
/// window that ends every batch; the kabi consumer loop's Phase-4.5
/// step names this flush explicitly.
/// 2. **Loop exit / disconnect** — before the stack buffer goes out of
/// scope (crash-eject, migration quiesce, module unload).
///
/// A loop MAY additionally flush inside any `consumer_core_section()`
/// (legal — Core image — e.g. when a section is entered anyway for a
/// blocking wait), but is not required to: keeping Core sections minimal
/// outweighs sub-window emission latency.
///
/// # Ordering and timestamp semantics
/// Entries drain FIFO (capture order) and carry capture-time
/// `timestamp_ns` (`cycles_to_ns(capture_cycles)`), never flush time —
/// ring consumers aggregate by `timestamp_ns` windows and must see the
/// probe's true time. Deferral introduces BOUNDED ring-position
/// reordering (an observation emitted directly from a Core section can
/// precede an older deferred entry by ring position, by at most one
/// batch window): consumers order by `timestamp_ns`, not ring position
/// — which the multi-CPU merge already requires of them.
pub fn obs_defer_flush(buf: &mut ObsDeferBuf, local_drops: &mut u32) {
// Direct ring push, NOT observe_kernel!: the macro stamps
// timestamp_ns with the CURRENT clock, but a flushed entry must carry
// its capture time. push_current_cpu() itself handles the
// nobody-listening case (null ring pointer → drop), so no separate
// static-key check is needed here.
let preempt_guard = preempt_disable();
for s in buf.drain(..) {
ObservationRing::push_current_cpu(s.subsystem, KernelObservation {
timestamp_ns: arch::current::cpu::cycles_to_ns(s.capture_cycles),
subsystem: s.subsystem,
obs_type: s.obs_type,
cpu_id: cpu_local::cpu_id() as u16,
_pad: 0,
cgroup_id: 0, // in-domain probes are kernel-context (no cgroup)
// Latency feature: ns-converted delta, pre-scaled to fit i32
// per the truncation rule on KernelObservation::features
// (>> 10 ≈ microsecond granularity, capped well below i32::MAX
// for any bounded in-domain latency).
features: collect_features!(
s.arg0, s.arg1,
(arch::current::cpu::cycles_to_ns(s.delta_cycles) >> 10) as i32),
}, &preempt_guard);
}
if *local_drops > 0 {
if let Some(rings) = OBSERVATION_RINGS.get() {
rings.get(&preempt_guard)
.dropped.fetch_add(*local_drops as u64, Relaxed);
}
*local_drops = 0;
}
drop(preempt_guard);
}
The arch::current::cpu conversion primitives cycles_to_ns() and
read_cycle_counter_ns() are declared at the canonical clock home,
Section 7.8. In-domain capture takes the raw reading
and postpones calibrated conversion to a Core window.
Tier and architecture neutrality. Probe sites compile identically at
every tier — the buffer-and-flush shape is unconditional. When the
capturing loop runs under the Core image the whole time (a Tier 0 /
same-domain module, or an architecture whose Tier 1 fallback runs
consumer loops fully in Core — see the per-architecture isolation table,
Section 11.3), the deferral degenerates to a harmless
≤-one-window batching of emissions; implementations MAY specialize such
loops to emit directly via observe_kernel! (the discipline only forbids
direct emission while a NON-Core image is live). Generic code names no
isolation hardware: the trigger condition is "non-Core image live", as
defined by the in-domain access discipline
(Section 12.8).
23.1.3 Tunable Parameter Store¶
Every kernel subsystem that accepts ML-driven tuning registers its parameters in a
global KernelParamStore. Parameters are read by the kernel with a single atomic load
(~1–3 cycles); writes require a CAS plus a version increment.
// umka-nucleus/src/ml/params.rs
/// A single tunable kernel parameter.
/// Layout: 128 bytes (two cache lines; align(64) rounds up the fields to prevent torn reads).
///
/// Byte layout:
/// param_id(4) + subsystem(2) + param_name(24) + _pad0(2) +
/// current(8) + default_value(8) + min_value(8) + max_value(8) +
/// decay_period_ms(4) + _pad1(4) + version(8) + last_updated_ns(8) +
/// last_model_seq(8) + _pad2(32) = 128 bytes.
#[repr(C, align(64))]
pub struct KernelTunableParam {
/// Unique monotonic ID assigned at registration time.
/// u32 matches wire protocol format; callers use ParamId enum for type safety.
pub param_id: u32,
pub subsystem: SubsystemId,
pub param_name: [u8; 24], // Null-terminated ASCII name
/// Explicit padding: [u8; 24] ends at offset 30; AtomicI64 needs
/// 8-byte alignment at offset 32. Must be zeroed.
pub _pad0: [u8; 2],
/// Current value (ML-adjusted). Read with AtomicI64::load(Relaxed).
pub current: AtomicI64,
pub default_value: i64,
pub min_value: i64,
pub max_value: i64,
/// After this many ms without a refresh from the ML layer, the parameter
/// automatically decays to `default_value`. 0 = no decay (permanent until reset).
pub decay_period_ms: u32,
/// Explicit padding: decay_period_ms ends at offset 68; AtomicU64 needs
/// 8-byte alignment at offset 72. Must be zeroed.
pub _pad1: [u8; 4],
/// Monotonic version AND the parameter's write-side sequence lock.
///
/// **Odd = a write is in progress, even = stable** (the seqlock convention
/// used across the kernel; see the shadow store's `epoch` below). The
/// parameter has TWO writers — the policy-update path (`set_with_expiry`)
/// and the decay sweeper (`enforce_param_decay`) — so unlike a
/// single-writer seqlock the odd slot is CLAIMED BY CAS rather than by a
/// plain store; `begin_write`/`end_write` below are the only mutators.
/// Version still increases monotonically and still changes on every
/// successful update (by 2), so version-watching consumers are unaffected.
/// READERS of `current` never touch this field: the hot-path read stays a
/// single `AtomicI64::load(Relaxed)`.
pub version: AtomicU64,
/// Timestamp of last ML-driven update (ns). 0 = never updated.
/// Written only inside a `begin_write`/`end_write` section.
pub last_updated_ns: AtomicU64,
/// Highest `PolicyUpdateMsg::model_seq` applied to this parameter at GLOBAL
/// scope (`cgroup_id == 0`). Updates with a lower or equal sequence are
/// stale or duplicated and are dropped — the per-`(param_id, cgroup_id)`
/// monotonicity rule in the update-validation list. The per-cgroup half of
/// that rule is carried by `ParamOverride::last_model_seq`.
/// u64: kernel-internal counter, never wraps in an operational lifetime.
pub last_model_seq: AtomicU64,
/// Trailing padding to fill the 128-byte align(64) boundary.
/// Fields end at offset 96; align(64) rounds to 128. Must be zeroed.
pub _pad2: [u8; 32],
}
const_assert!(core::mem::size_of::<KernelTunableParam>() == 128);
/// Maximum number of subsystem groups (ParamId bits [11:8]).
/// Group indices map to `SubsystemId` discriminants via `(group + 1)`:
/// 0=Scheduler, 1=MemoryManager, 2=TcpStack, 3=BlockIo, 4=PowerManager,
/// 5=FmaHealth, 6=NvmeDriver, 7=NetworkDriver, 8=IoScheduler, 9=Gpu,
/// 10=Storage, 11=Accel, 12=VfsLayer, 13=ContainerMgr, 14-15=Reserved.
/// These names match the `SubsystemId` enum variants exactly.
pub const PARAM_GROUP_COUNT: usize = 16;
/// Maximum number of parameters per group (ParamId bits [7:0]).
pub const PARAMS_PER_GROUP: usize = 256;
/// Global parameter registry. Two-level indexed by ParamId:
/// group = (param_id >> 8) & 0x0F (4 bits, max 16 groups)
/// index = param_id & 0xFF (max 256 per group)
/// Total capacity: 16 * 256 = 4096 slots (512 KiB params + 4 KiB registered bitmap).
/// O(1) lookup, no heap allocation, cache-friendly per-group access.
/// Initialized at boot from per-subsystem `register_param!` calls.
/// Note: BSS (zero-initialized, no physical pages consumed until written).
/// On memory-constrained targets, `PARAMS_PER_GROUP` can be reduced to 64
/// (128 KiB total) via compile-time configuration. The ML policy subsystem
/// is Phase 4+ and targets server workloads.
///
/// **`register_param!` macro** — registers a kernel tunable parameter at boot time.
/// Takes an explicit `ParamId` variant (the enum has fixed discriminants assigned
/// per-subsystem, so dynamic ID allocation is neither needed nor desired):
///
/// ```rust
/// /// Register a kernel tunable parameter. Expands to a KernelParamStore::register()
/// /// call at boot time. The caller supplies an explicit ParamId variant — IDs are
/// /// statically assigned in the `ParamId` enum defined below.
/// macro_rules! register_param {
/// ($id:expr, $name:literal, $default:expr, $min:expr, $max:expr, $decay_ms:expr) => {
/// KERNEL_PARAM_STORE.register(KernelTunableParam {
/// param_id: $id as u32,
/// subsystem: SubsystemId::from_param_id($id)
/// .expect("ParamId has no valid SubsystemId mapping"),
/// param_name: ascii_pad!($name),
/// _pad0: [0; 2],
/// current: AtomicI64::new($default),
/// default_value: $default,
/// min_value: $min,
/// max_value: $max,
/// decay_period_ms: $decay_ms,
/// _pad1: [0; 4],
/// version: AtomicU64::new(0),
/// // Initialize to current boot timestamp to prevent immediate
/// // decay expiry. With 0, `now_ns - 0 > decay_period_ms * 1e6`
/// // would be true on the first decay check.
/// last_updated_ns: AtomicU64::new(ktime_get_ns()),
/// last_model_seq: AtomicU64::new(0),
/// _pad2: [0; 32],
/// })
/// };
/// }
///
/// // Usage example:
/// // register_param!(ParamId::SchedEevdfWeightScale, "eevdf_weight_scale", 100, 10, 1000, 5000);
/// ```
///
/// `SubsystemId::from_param_id()` derives the subsystem from the ParamId discriminant
/// (upper byte: 0x00xx = Scheduler, 0x01xx = Memory, 0x02xx = Network, etc.).
pub struct KernelParamStore {
/// MaybeUninit preserves the 128-byte alignment of KernelTunableParam
/// without the Option discriminant overhead (which would break repr(C)
/// layout assumptions for the align(64) struct).
pub params: [[MaybeUninit<KernelTunableParam>; PARAMS_PER_GROUP]; PARAM_GROUP_COUNT],
/// Bitmap tracking which slots contain initialized parameters.
/// Uses `AtomicBool` with Acquire/Release ordering because `get()`
/// reads this array lock-free (without `store_lock`) while
/// `register()` writes it under `store_lock`. Without atomics,
/// a concurrent `get()` racing with `register()` could see a
/// torn `true` value before the corresponding `params` slot is
/// fully initialized — reading uninitialized memory.
///
/// Protocol:
/// - Writer (register): initialize `params[g][i]` first, then
/// `registered[g][i].store(true, Release)`. The Release ensures
/// all writes to `params` are visible before the flag is set.
/// - Reader (get): `registered[g][i].load(Acquire)`. The Acquire
/// ensures all writes by the writer are visible after the flag
/// reads `true`.
pub registered: [[AtomicBool; PARAMS_PER_GROUP]; PARAM_GROUP_COUNT],
pub count: AtomicU32,
pub store_lock: SpinLock<()>, // Protects registration; not held during reads
}
impl KernelParamStore {
/// Look up a parameter by its `ParamId`. O(1), lock-free.
#[inline]
pub fn get(&self, id: ParamId) -> Option<&KernelTunableParam> {
let raw = id as u32;
let group = ((raw >> 8) & 0x0F) as usize;
let index = (raw & 0xFF) as usize;
if self.registered[group][index].load(Acquire) {
// SAFETY: registered[group][index] is only set to true (via
// Release store) after the corresponding params[group][index]
// has been fully initialized. The Acquire load above pairs
// with the Release store in register(), ensuring all writes
// to params[group][index] are visible.
Some(unsafe { self.params[group][index].assume_init_ref() })
} else {
None
}
}
/// Iterate all registered (initialized) parameters across all groups.
pub fn active_params(&self) -> impl Iterator<Item = &KernelTunableParam> {
self.params.iter().zip(self.registered.iter())
.flat_map(|(pg, rg)| pg.iter().zip(rg.iter()))
.filter_map(|(slot, reg)| {
if reg.load(Acquire) {
// SAFETY: registered flag (Acquire) guarantees the slot is initialized.
Some(unsafe { slot.assume_init_ref() })
} else {
None
}
})
}
/// Registered `[min, max]` clamp bounds for a parameter. Used by the policy
/// dispatch path to clamp incoming values (out-of-range values are ALWAYS
/// clamped, never rejected). If the parameter is not registered, returns the
/// full `i64` range so the clamp is an identity — the target subsystem's own
/// store still enforces its private bounds.
#[inline]
pub fn bounds(&self, id: ParamId) -> ParamBounds {
match self.get(id) {
Some(p) => ParamBounds { min: p.min_value, max: p.max_value },
None => ParamBounds { min: i64::MIN, max: i64::MAX },
}
}
/// Empty store: every slot uninitialized, every `registered` flag false.
/// `const` so the `KERNEL_PARAM_STORE` global can be a `static` initializer.
pub const fn new() -> Self {
KernelParamStore {
params: [const {
[const { MaybeUninit::uninit() }; PARAMS_PER_GROUP]
}; PARAM_GROUP_COUNT],
registered: [const {
[const { AtomicBool::new(false) }; PARAMS_PER_GROUP]
}; PARAM_GROUP_COUNT],
count: AtomicU32::new(0),
store_lock: SpinLock::new(()),
}
}
}
/// Clamp bounds returned by [`KernelParamStore::bounds`].
pub struct ParamBounds {
/// Inclusive lower bound (a parameter's `min_value`).
pub min: i64,
/// Inclusive upper bound (a parameter's `max_value`).
pub max: i64,
}
/// Global kernel tunable-parameter store. Populated at boot via `register_param!`
/// and read lock-free on hot paths via `get()`.
static KERNEL_PARAM_STORE: KernelParamStore = KernelParamStore::new();
Reading a parameter (zero overhead when compared to hardcoded values):
// Hot-path read pattern — no lock, single atomic load:
let weight_scale = KERNEL_PARAM_STORE.get(ParamId::SchedEevdfWeightScale)
.map_or(100, |p| p.current.load(Relaxed));
Parameter consumption points:
| Parameter | Consumer | Read Path | Frequency |
|---|---|---|---|
sched_slice_ns |
Scheduler (pick_eevdf) |
CpuLocal cached copy |
Every scheduling tick |
mem_reclaim_aggression |
reclaimd reclaim scan | Direct read from KernelParamStore |
Every reclaim cycle |
io_congestion_threshold |
Block layer congestion check | Cached in BlockDevice.cached_params |
Every bio submission |
Parameters are published via KernelParamStore::update() which uses atomic
stores. Consumers that cache values refresh on a configurable interval
(default: 100ms) or on explicit policy push notification.
Scope limitation: The CpuLocal cached copies are system-wide parameter values,
NOT per-cgroup overrides. When a task runs in a cgroup with ML policy overrides
(MlPolicyCss::overrides), the hot-path consumer reads the global CpuLocal cache
first (fast path: ~1 cycle), then checks for a per-cgroup override via
the current task's cgroup MlPolicyCss::overrides for param_id (warm path: XArray lookup,
~20-50 cycles). The per-cgroup override takes precedence when present. This two-level
lookup ensures the common case (no override) pays only the CpuLocal read cost.
Decay enforcement runs once per second (not per tick). The scheduler tick checks a
decay_due flag that is set by a 1-second periodic timer. Per-cgroup decay runs on
a designated decay CPU to avoid multi-CPU races on the cgroup tree walk.
/// The CPU designated to run per-cgroup parameter decay.
/// Initialized to 0 at boot. Reassigned via CPU hotplug notifier when the
/// current decay CPU goes offline:
/// dynamic CPU-online state "ml-policy/decay" with callbacks:
/// online = `decay_cpu_online`, offline = `decay_cpu_offline`
/// On offline: `DECAY_CPU.store(cpumask_any_online(), Relaxed)`.
/// On online: no action (keep current decay CPU).
static DECAY_CPU: AtomicU32 = AtomicU32::new(0);
// Called from schedule_tick() — guarded by decay_due flag (1 Hz periodic timer).
// Only the designated decay CPU runs this function.
// O(active_params) with early-exit on no-expiry.
fn enforce_param_decay(now_ns: u64) {
// Phase 1: Global parameter store decay
for param in KERNEL_PARAM_STORE.active_params() {
if param.decay_period_ms > 0 {
let window_ns = param.decay_period_ms as u64 * 1_000_000;
// Unlocked PROBE only — cheap rejection for the overwhelming
// majority of parameters that are nowhere near expiry, so the 1 Hz
// sweep does not claim a write section per registered parameter.
// saturating_sub prevents wrap to very large u64 if last > now_ns
// (possible on multi-socket systems with TSC skew across packages).
if now_ns.saturating_sub(param.last_updated_ns.load(Acquire)) <= window_ns {
continue;
}
// Claim the write section, then RE-READ the marker under it. The
// re-read is load-bearing: without it the sweeper would decide from
// a marker sampled before a concurrent `set_with_expiry`, and then
// store the default OVER that writer's freshly published value —
// a silent premature revert. Release ordering on the writer's
// stores cannot close that window, because it does not constrain a
// decision the sweeper has ALREADY taken; only mutual exclusion
// between the decision and the store does.
let (seq, guard) = param.begin_write();
if now_ns.saturating_sub(param.last_updated_ns.load(Relaxed)) > window_ns {
param.current.store(param.default_value, Release);
// The parameter is back at its default and unowned: reset the
// model-sequence watermark so a service that restarts with a
// fresh sequence is not rejected as stale forever.
param.last_model_seq.store(0, Relaxed);
}
param.end_write(seq, guard);
}
}
// Phase 2: Per-cgroup override decay (MlPolicyCss::overrides)
// Runs only on CPU 0 (single writer avoids contention on cgroup tree walk).
// **Cost**: O(N) where N = total cgroups with ML overrides. At 1 Hz, this is
// acceptable for up to ~10,000 cgroups (walk takes <1ms at ~100ns/cgroup).
// For larger deployments, a future optimization could use a "dirty list" of
// cgroups with pending decay (linked at override write time), reducing the
// walk to O(dirty) instead of O(all).
if cpu_local::cpu_id() == DECAY_CPU.load(Relaxed) {
enforce_cgroup_override_decay(now_ns);
}
}
impl KernelTunableParam {
/// Atomically publish `value` into `current` and schedule its auto-revert
/// to `default_value` at absolute monotonic time `expiry_ns`. This is the
/// sole writer of the (value, expiry) pair from the policy-update path
/// ([Section 7.10](07-scheduling.md#intent-based-resource-management) `apply_policy_update` →
/// `param.set_with_expiry(stored, expiry_ns)`).
///
/// **Revert is enforced by `enforce_param_decay` above**, which reverts when
/// `now - last_updated_ns > decay_period_ms * 1e6`. To make that fire
/// exactly at `expiry_ns`, this backs the anti-decay marker off by one decay
/// window: `last_updated_ns = expiry_ns - decay_period_ms*1e6`
/// (saturating). A param with `decay_period_ms == 0` never auto-reverts
/// (permanent) — pass `expiry_ns = u64::MAX` for that case; a finite
/// `expiry_ns` is then advisory.
///
/// **Ordering — no premature revert, no torn pair**: the (marker, value)
/// pair is published inside a `begin_write`/`end_write` section, which is
/// mutually exclusive with the decay sweeper's revert section. Store
/// ordering alone is NOT sufficient here and the earlier marker-first
/// `Release` argument was unsound: a sweeper that had already loaded the
/// stale marker and decided to revert could still execute its
/// `store(default)` after this function published marker and value, so the
/// asserted "either observes the fresh marker or the revert is the intended
/// one" case split had a third, silent case. Under the write section the
/// sweeper re-reads the marker after claiming it, so the two writers
/// linearize: either the refresh happens first and the sweeper then sees a
/// fresh marker and skips, or the revert happens first and this refresh
/// immediately supersedes it. Version-watching consumers see an even
/// `version` only outside a write section, hence never a half-updated pair.
pub fn set_with_expiry(&self, value: i64, expiry_ns: u64) {
let window_ns = self.decay_period_ms as u64 * 1_000_000;
// Anti-decay marker chosen so `enforce_param_decay` reverts at expiry_ns.
let marker = expiry_ns.saturating_sub(window_ns);
let (seq, guard) = self.begin_write();
self.last_updated_ns.store(marker, Relaxed);
self.current.store(value, Release);
self.end_write(seq, guard);
}
/// Claim this parameter's write section, returning the odd sequence value
/// that `end_write` must close. Spins while another writer holds the
/// section.
///
/// The parameter has exactly two writers — the policy-update path and the
/// 1 Hz decay sweeper on `DECAY_CPU` — and each section is three atomic
/// stores with no call, no allocation and no nested lock, so this is a leaf
/// write-side seqlock: it acquires nothing and can therefore take no part
/// in a lock cycle. Readers of `current` do NOT participate — the hot-path
/// parameter read remains one relaxed atomic load, so this costs the read
/// path zero cycles.
///
/// **IRQs, not merely preemption, are disabled across the section.** The
/// decay sweeper runs from `schedule_tick()`, i.e. in interrupt context. A
/// tick landing on a CPU whose task-context writer holds the odd sequence
/// would spin waiting for a section only that interrupted task can release
/// — a single-CPU self-deadlock. Preemption-disable does not mask
/// interrupts and therefore does not prevent it; IRQ-disable does. The
/// window is three atomic stores, so the added interrupt latency is
/// negligible. `local_irq_save()` is a safe nestable factory of independent
/// guards ([Section 3.8](03-concurrency.md#interrupt-handling)), so nesting inside an already-IRQ-off
/// tick is well-defined.
///
/// A CAS claim (not the single-writer `load`+`store` form) is required
/// precisely BECAUSE there are two writers; using the plain form here would
/// be the missing-mutual-exclusion bug the single-writer convention exists
/// to expose.
///
/// Returns the claimed odd sequence together with the `IrqDisabledGuard`
/// that pins the section — the guard is returned rather than left to the
/// caller so the discipline is structural, not advisory.
#[inline]
fn begin_write(&self) -> (u64, IrqDisabledGuard) {
let guard = local_irq_save();
loop {
let v = self.version.load(Acquire);
if v & 1 != 0 {
core::hint::spin_loop();
continue; // another writer holds the section
}
if self.version
.compare_exchange_weak(v, v + 1, AcqRel, Relaxed)
.is_ok()
{
return (v + 1, guard);
}
}
}
/// Close the write section claimed by `begin_write`, returning `version` to
/// an even value and publishing every store made inside the section.
#[inline]
fn end_write(&self, claimed: u64, guard: IrqDisabledGuard) {
self.version.store(claimed + 1, Release);
drop(guard);
}
/// Admit a policy update's `model_seq` at GLOBAL scope. Returns `true` iff
/// `seq` is strictly newer than the last applied sequence, recording it
/// atomically so two consumer threads cannot both admit the same value.
/// Stale and duplicate updates are dropped silently, as specified.
#[inline]
pub fn admit_model_seq(&self, seq: u64) -> bool {
let mut last = self.last_model_seq.load(Acquire);
loop {
if seq <= last {
return false;
}
match self.last_model_seq
.compare_exchange_weak(last, seq, AcqRel, Acquire)
{
Ok(_) => return true,
Err(observed) => last = observed,
}
}
}
}
/// Maximum number of cgroups processed by one override-decay pass. This is a
/// batch bound, not a cgroup-count limit: a matching ID that would overflow a
/// full batch records truncation and makes the caller repeat the collect/process
/// procedure until no matching cgroup was truncated.
pub const CGROUP_OVERRIDE_DECAY_BATCH: usize = 64;
/// Walk the cgroup tree and expire stale per-cgroup parameter overrides.
/// Each MlPolicyCss holds an ArrayVec of ParamOverride entries (param_id, value, expiry_ns).
/// Expired entries are removed by swapping with the last element (O(1) remove).
///
/// This is deliberately two-phase. Non-preemptible RCU readers must not sleep:
/// holding the walk's guard across `config_lock.lock()` is forbidden, as is
/// carrying any `&'g`-bound iterator result past that guard's drop.
fn enforce_cgroup_override_decay(now_ns: u64) {
loop {
let mut batch: ArrayVec<CgroupId, CGROUP_OVERRIDE_DECAY_BATCH> =
ArrayVec::new();
// Phase 1 — collect IDs under one bounded, sleep-free RCU walk. The
// block boundary drops the guard, iterator, and every yielded `&S`
// before Phase 2 can take a sleeping lock.
let truncated = {
// `descendants::<S>(root_cgroup, &guard)` is the typed
// registered-subsystem iterator ([Section 17.2](17-containers.md#control-groups)): it yields
// `&S` (here `&MlPolicyCss`) for each descendant that has the
// subsystem attached, skipping those that do not — no separate
// `subsys_state()` downcast, and no bare
// `for_each_descendant()` call.
let guard = rcu_read_lock();
let root_cgroup = ML_POLICY_ROOT_CSS
.cgroup(&guard)
.expect("ml_policy root cgroup live");
let mut truncated = false;
for ml_css in descendants::<MlPolicyCss>(root_cgroup, &guard) {
// overrides is RcuCell<ArrayVec<ParamOverride,
// CAP_CGROUP_ML_PARAMS>>. RcuCell provides .read(&guard) and
// .update(new, &proof), NOT .lock().
//
// Unlocked PROBE: skip the overwhelming majority of cgroups without
// touching config_lock. Its result is advisory — it decides only
// whether to take the lock, never what to publish.
let probe = ml_css.overrides.read(&guard);
if !probe.iter().any(|e| e.expiry_ns <= now_ns) {
continue;
}
if batch.try_push(ml_css.header.cgroup_id).is_err() {
truncated = true;
break;
}
}
truncated
};
// Phase 2 — process each scalar ID with no walk guard alive. The
// registry accessor clones an owned Arc inside its own short RCU
// section, pinning the cgroup across the sleeping config_lock.
for cgroup_id in batch {
let Some(cg) = find_cgroup_by_id(cgroup_id) else {
// rmdir removed the cgroup; its overrides die with it.
continue;
};
let proof = cg.config_lock.lock();
{
// The subsystem may have detached while the ID was batched.
// Re-resolve it under a fresh RCU guard while the writer proof
// prevents a concurrent override publication.
let guard = rcu_read_lock();
let Some(ml_css) = cg.subsys_state::<MlPolicyCss>() else {
continue;
};
// AUTHORITATIVE snapshot, re-read UNDER the writer proof. Cloning the
// pre-lock snapshot instead would publish a stale copy-on-write image
// and silently discard any override a policy writer installed between
// the probe and the lock — the write path (steps 3-5 below) publishes
// under this same proof, so only a snapshot taken under it is a valid
// base for the copy.
let current = ml_css.overrides.read(&guard);
if current.iter().any(|e| e.expiry_ns <= now_ns) {
// Clone, filter expired, publish via RCU copy-on-write.
let mut new = current.clone();
new.retain(|entry| entry.expiry_ns > now_ns);
ml_css.overrides.update(new, &proof).ok();
}
}
}
if !truncated {
break;
}
// Every processed match had its expired entries removed (or vanished
// through rmdir/detach), so the next probe skips it. A stable set of M
// matches therefore completes in ceil(M / CGROUP_OVERRIDE_DECAY_BATCH)
// passes while each RCU section remains bounded to one batch walk.
}
}
23.1.4 Synchronous Policy Query¶
The KernelParamStore above serves the asynchronous model: a Tier 2 service
writes a scalar tuned value, the kernel reads it with a single atomic load.
Some decisions instead need a per-subject answer — scoring one OOM candidate
(Section 4.2),
judging one page's NUMA migration candidacy — where the input is a feature
vector, not a global scalar. ml_policy_query provides that: it evaluates a
registered in-kernel model (Section 22.6) against a
per-decision feature vector and returns its output, or None when no model is
registered (caller then uses its heuristic default). Because the model's
load-time structural validation bounds its operation count, inference is safe
on the warm/cold decision paths that use it.
/// Integer feature dimensions in a synchronous policy-query input vector.
/// Matches `KernelObservation.features` so the same `define_feature_extractor!`
/// layouts describe both training observations and query inputs.
pub const POLICY_QUERY_FEATURE_DIMS: usize = 10;
/// Per-decision inference budget (ns). A registered model exceeding this is
/// disabled by `infer_safe`'s defense-in-depth cycle check (Class B ceiling).
pub const POLICY_QUERY_MAX_NS: u64 = 50_000;
/// Per-decision feature struct passed to [`ml_policy_query`]. Packs its named
/// fields into the fixed `[i32; POLICY_QUERY_FEATURE_DIMS]` inference input.
/// Implemented by e.g. `OomCandidateFeatures`
/// ([Section 4.2](04-memory.md#physical-memory-allocator--ml-policy-integration-intelligent-oom-victim-selection)).
pub trait PolicyFeatures {
fn write_features(&self, out: &mut [i32; POLICY_QUERY_FEATURE_DIMS]);
}
/// Per-`ParamId` synchronous-inference model registry. Two-level indexed
/// exactly like `KernelParamStore` (group = (id>>8)&0xF, index = id&0xFF); a
/// null slot means "no model registered for this parameter". Registered on ML
/// policy service activation and cleared on teardown — the same
/// policy-service-activated lifecycle as the observation rings. BSS (no
/// physical pages until a model is registered).
///
/// **Model lifetime**: the registry OWNS every model it holds. `register`
/// takes a `Box<KernelModel>` and leaks it into the slot; `unregister` (and a
/// replacing `register`) swaps the slot and releases the displaced model
/// through `rcu_call_box_drop`, i.e. only after every reader that could still
/// hold it has left its RCU read-side critical section. Readers borrow under an
/// `RcuReadGuard` and the returned reference is tied to THAT guard.
///
/// The earlier shape — bare `AtomicPtr` slots holding `&'static KernelModel`
/// borrowed from the service's loaded-model table, cleared by a plain null
/// store with no grace period — was unsound in both directions: the `'static`
/// claim was false (its own safety note conceded the model lived only "until
/// `unregister`"), and teardown could free a model while a query was inside
/// `infer_safe`. Both halves are one defect and are fixed together, because
/// fixing either alone leaves the window open: an RCU-deferred free does
/// nothing if readers are not RCU-delimited, and delimiting readers does
/// nothing if the free is immediate. `AtomicModelRef` below already holds this
/// exact discipline; the registry now matches it.
pub struct PolicyQueryRegistry {
models: [[AtomicPtr<KernelModel>; PARAMS_PER_GROUP]; PARAM_GROUP_COUNT],
}
impl PolicyQueryRegistry {
/// Resolve the model registered for `id`, or `None`. Lock-free `Acquire`
/// load, paired with the `Release` swap in `register`.
///
/// The borrow is tied to the caller's `RcuReadGuard`: the model cannot be
/// freed before that guard drops, which is what makes holding it across
/// inference safe. The guard is the caller's, not this function's, so a
/// caller cannot accidentally let the reference outlive the protection.
pub fn get<'g>(&self, id: ParamId, _guard: &'g RcuReadGuard)
-> Option<&'g KernelModel>
{
let raw = id as u32;
let group = ((raw >> 8) & 0x0F) as usize;
let index = (raw & 0xFF) as usize;
let p = self.models[group][index].load(Ordering::Acquire);
// SAFETY: a non-null slot holds a model the registry owns, published by
// `register` with Release. It is released only via `rcu_call_box_drop`,
// so it stays live for the whole of `_guard`'s read-side critical
// section — exactly the lifetime the returned reference carries.
if p.is_null() { None } else { Some(unsafe { &*p }) }
}
/// Register (or replace) the model for `id`. Called on service activation.
/// Ownership of `model` transfers to the registry.
///
/// **Serialization**: callers must hold `MODEL_UPDATE_LOCK`, for the same
/// reason `AtomicModelRef::update` does — two concurrent swaps on one slot
/// would make the loser's "old" pointer be the winner's live model, which
/// would then be freed after a grace period while still registered.
pub fn register(&self, id: ParamId, model: Box<KernelModel>) {
let raw = id as u32;
let new_ptr = Box::into_raw(model);
let old = self.models[((raw >> 8) & 0x0F) as usize][(raw & 0xFF) as usize]
.swap(new_ptr, Ordering::AcqRel);
if !old.is_null() {
// SAFETY: `old` was produced by `Box::into_raw` in a previous
// `register`; deferring the drop lets in-flight queries finish.
unsafe { rcu_call_box_drop(old) };
}
}
/// Clear the model for `id` (service teardown). Later queries fall back to
/// the caller's heuristic. Requires `MODEL_UPDATE_LOCK`, as `register` does.
///
/// The displaced model is released through `rcu_call_box_drop`, so a query
/// that loaded the pointer immediately before this swap completes its
/// inference on live memory.
pub fn unregister(&self, id: ParamId) {
let raw = id as u32;
let old = self.models[((raw >> 8) & 0x0F) as usize][(raw & 0xFF) as usize]
.swap(core::ptr::null_mut(), Ordering::AcqRel);
if !old.is_null() {
// SAFETY: as in `register`.
unsafe { rcu_call_box_drop(old) };
}
}
}
pub static POLICY_QUERY_MODELS: PolicyQueryRegistry = PolicyQueryRegistry {
models: [const { [const { AtomicPtr::new(core::ptr::null_mut()) }; PARAMS_PER_GROUP] };
PARAM_GROUP_COUNT],
};
/// Synchronous per-decision ML policy query. Resolves `id` to its registered
/// model, packs `features`, runs bounded in-kernel inference, and returns the
/// single clamped output — or `None` if no model is registered (or it is
/// inactive / returned an invalid output), in which case the caller uses its
/// heuristic default.
pub fn ml_policy_query<F: PolicyFeatures>(id: ParamId, features: &F) -> Option<i32> {
// RCU read section covers the model lookup AND the whole inference. The
// model is owned by the registry and released only via `rcu_call_box_drop`,
// so a service teardown concurrent with this query (a Tier 2 service can be
// SIGKILLed or OOM-killed at any instant — and OOM victim scoring is itself
// one of this function's callers) cannot free the model mid-inference.
let guard = rcu_read_lock();
let model = POLICY_QUERY_MODELS.get(id, &guard)?;
let mut input = [0i32; POLICY_QUERY_FEATURE_DIMS];
features.write_features(&mut input);
let mut output = [0i32; 1];
let mut fell_back = false;
// Bounded in-kernel inference (Class B). `infer_safe` runs the fallback
// closure iff the model is inactive/invalid; treat that as "unavailable".
infer_safe::<POLICY_QUERY_MAX_NS>(model, &input, &mut output,
|_in, _out| { fell_back = true; });
if fell_back {
return None;
}
// Clamp to the parameter's registered bounds before the value escapes.
// The engine's own `validate_output` rejects only `i32::MIN`/`i32::MAX`
// ([Section 22.6](22-accelerators.md#in-kernel-inference-engine)), so every other magnitude — including
// anything an adversarially-trained or adversarially-fed model emits —
// would otherwise reach the caller unbounded. Bounded parameters are the
// framework's safety mechanism; the synchronous path owes the same
// guarantee as the asynchronous one, where `dispatch_to_subsystem` clamps.
// An unregistered `id` yields the full `i64` range, i.e. an identity clamp.
let bounds = KERNEL_PARAM_STORE.bounds(id);
let lo = bounds.min.clamp(i32::MIN as i64, i32::MAX as i64) as i32;
let hi = bounds.max.clamp(i32::MIN as i64, i32::MAX as i64) as i32;
Some(output[0].clamp(lo, hi))
}
23.1.5 Policy Consumer KABI (Tier 2 → Kernel)¶
Before the vtable is defined, the two supporting types passed during registration are specified:
// umka-nucleus/src/ml/observation.rs (continued)
/// Per-CPU set of observation ring descriptors, one per KernelObservation
/// category. Populated by the `observe_kernel!` macro; consumed by policy
/// services. This struct is KERNEL-PRIVATE and is never mapped: what a service
/// maps is the per-ring shared region (`ObservationRingShared` header page,
/// read-write, plus the read-only slot array behind it), one mapping per
/// subscribed category per CPU.
pub struct ObservationRingSet {
/// Per-category ring pointers. Null = unallocated (no subscriber for this
/// category). Uses AtomicPtr for lock-free lazy initialization (CAS
/// null -> allocated). Indexed by `SubsystemId as usize`. A ring pointer is
/// installed by the FIRST registration to subscribe to that category and
/// cleared only after the LAST one leaves — later registrations claim a
/// cursor on the existing ring rather than replacing the pointer.
pub rings: [AtomicPtr<ObservationRing>; OBSERVATION_CATEGORY_COUNT],
/// CPU this ring set belongs to (for NUMA-aware allocation by the consumer).
pub cpu_id: u32,
/// Total observations dropped due to ring overflow since last reset,
/// summed across this CPU's rings. Updated atomically by the kernel
/// producer; the per-ring figure lives in `ObservationRingShared::overflow`.
pub dropped: AtomicU64,
}
/// **Memory footprint**: `OBSERVATION_CATEGORY_COUNT` (16) categories ×
/// `capacity` entries × 64 bytes per `KernelObservation`, plus one header page
/// per ring. At the default capacity (4096) that is **4 MiB per CPU** with all
/// categories active — ~512 MiB on a 128-core system. Capacity is a runtime
/// value precisely so this scales DOWN as well as up: at the minimum (256) the
/// same all-categories worst case is 256 KiB per CPU, which is what makes the
/// framework usable on laptop, edge and embedded targets rather than servers
/// only.
///
/// **Policy-service-activated allocation**: Rings are NOT allocated at boot.
/// Ring for category C on CPU K is allocated when the first Tier 2 policy
/// service registers with `subsystem_mask` bit C set (via `ML_POLICY_REGISTER`
/// ioctl). Allocation happens on the registration code path (cold, may sleep),
/// not on the observe path (warm). This means a system with ML policy disabled
/// uses zero ring memory. A system with only scheduler observations uses
/// 256 KiB/CPU at the default capacity (1 category x 4096 x 64B), not 4 MiB.
///
/// **Lifecycle**: allocate → store pointer (Release) → enable static key →
/// observations flow → last subscriber leaves → disable static key → RCU GP →
/// store null → free rings. The static key disable happens-before the
/// AtomicPtr null store, and the null store happens-before the free.
///
/// **Memory budget**: categories x CPUs x (capacity x 64 B + header page +
/// summary ring). Registration failure (ENOMEM) rolls back all per-CPU
/// allocations. The per-ring capacity is system-wide, set at boot via the
/// `umka.ml.ring_capacity` command-line parameter and re-settable at runtime
/// via `/sys/kernel/umka/ml/ring_capacity` (a runtime change takes effect at
/// the next ring allocation, i.e. the next first-subscriber registration for a
/// category — live rings are not resized underneath their subscribers).
/// Number of observation categories (= max SubsystemId discriminant + 1,
/// rounded up to the next power of two for index-by-enum access).
pub const OBSERVATION_CATEGORY_COUNT: usize = 16;
23.1.5.1.1 Observation Summary Ring (Stress Resilience)¶
Under sustained system stress (IRQ storms, memory pressure, CPU saturation), the raw
observation ring may overflow faster than the Tier 2 policy service can drain it. When
the per-CPU dropped counter exceeds a threshold (default: 64 dropped observations),
the kernel activates a secondary aggregation path.
/// Pre-aggregated observation summary. Produced by a 100ms softirq timer
/// that drains the raw observation ring into fixed-size summaries.
/// The Tier 2 service reads summaries when raw ring overflow is detected,
/// maintaining statistical awareness even when individual observations are lost.
#[repr(C)]
pub struct ObservationSummary {
/// Observation type discriminant — identifies which metric type within
/// this subsystem's summary ring this entry aggregates (e.g.,
/// SchedObs::TaskWoke, MemObs::PageFault). The subsystem identity is
/// implicit from the ring index (ObservationRingSet.rings[subsystem_id]).
/// Without this field, the Tier 2 consumer cannot distinguish summaries
/// from different metric types within the same subsystem ring.
pub obs_type: u16,
/// Explicit padding for u64 alignment of timestamp_ns.
/// (obs_type is u16 to match KernelObservation.obs_type and all *Obs enum
/// discriminants, which are #[repr(u16)].)
pub _pad: [u8; 2],
pub _pad2: [u8; 4],
/// Aggregation window start (monotonic nanoseconds).
pub timestamp_ns: u64,
/// Number of raw observations aggregated into this summary.
/// **Overflow check**: Capped at u32::MAX; if more than ~4 billion
/// observations occur in a single 100ms window (physically impossible),
/// count saturates rather than wrapping.
pub count: u32,
/// Explicit padding for u64 alignment of sum.
pub _pad3: [u8; 4],
/// Sum of the observed metric values (for computing mean).
/// `i64` because `KernelObservation.features` are `[i32; 10]` (signed).
/// Features like `oom_score_adjustment` range [-500, +500] and delta values
/// can be negative. u64 would wrap to a very large value when summing
/// predominantly negative features. i64 safely holds u32::MAX i32 values
/// (max magnitude 9.22e18, within i64::MAX).
pub sum: i64,
/// Minimum observed value in this window.
/// `i32` to match `KernelObservation.features` signedness.
pub min: i32,
/// Maximum observed value in this window.
/// `i32` to match `KernelObservation.features` signedness.
pub max: i32,
/// Approximate 99th percentile (DDSketch or t-digest compatible).
/// `i32` to match `KernelObservation.features` signedness.
pub p99_approx: i32,
/// Trailing padding for struct alignment (8 bytes).
pub _pad4: [u8; 4],
}
const_assert!(size_of::<ObservationSummary>() == 48);
/// Per-CPU summary ring. 64 entries = 6.4 seconds of summaries at 100ms intervals.
/// Sized to bridge a worst-case stress burst without losing statistical coverage.
pub struct ObservationSummaryRing {
entries: [ObservationSummary; 64],
/// Producer index (mod 64). **Longevity**: u32 wraps after ~4.3 billion
/// entries. At 10 entries/sec (100ms intervals), wraps in ~13.6 years.
/// Acceptable: only the low 6 bits are used for indexing (mod 64), and
/// `head - tail` distance is always < 64, so wrap is transparent.
/// u64 is unnecessary because the distance (not the absolute index)
/// determines correctness.
head: AtomicU32,
tail: AtomicU32,
}
Aggregation timer: A per-CPU softirq timer fires every 100ms. On each tick:
1. Read every entry published since the last tick through the aggregator's own
cursor (ObservationRingShared::cursors[0], owner CURSOR_SLOT_AGGREGATOR),
using the consumer read protocol above.
2. Compute count/sum/min/max/p99 for each metric type.
3. Write one ObservationSummary per metric type to the summary ring.
4. Recompute slowest_tail exactly, by taking the minimum over the claimed
cursors — this is the refresh that keeps the producer's single-load overflow
probe honest, and the softirq is the natural place for it because it is
already walking the cursor array.
The aggregator is a first-class subscriber, not a second reader of somebody else's cursor. That matters: a single shared read pointer would make the softirq and a Tier 2 service race to advance one tail, each consuming entries the other would then never see, with no handoff state to arbitrate it. With per-subscriber cursors the aggregator and every service observe the identical published stream and advance independently, and the aggregation tick reserves no ring state that a service could be mid-use of.
Tier 2 fallback path: A Tier 2 policy service detects loss on ITS OWN cursor
— head - my_tail > capacity means the producer lapped it — and switches to
reading from the summary ring instead of the raw ring for as long as that holds.
This provides degraded-but-usable statistical data (mean, min, max, p99) even
during stress events when per-observation granularity is lost. Per-cursor
detection is what makes the fallback correct with several subscribers: the shared
overflow/dropped counters are aggregate diagnostics (they drive the
OBSERVATION_DROP_ALERT_THRESHOLD FMA alert), and a slow service must not be
able to push a service that is keeping up onto the degraded path.
Phase assignment: Phase 5c (the ML policy framework is Phase 5c per Section 24.2). The raw observation ring is the primary data source; the summary ring is a hardening improvement for production resilience.
// umka-nucleus/src/ml/params.rs (continued)
/// Read-only shadow view of the KernelTunableParam store for use by policy consumers.
/// Updated atomically on each policy epoch (a complete pass of decay enforcement);
/// consumers read from this without needing to acquire the param store write lock.
///
/// The kernel mmaps this structure into Tier 2 service address space as read-only.
/// Consumers detect staleness by watching `epoch`: if `epoch` changes between two reads
/// of `params[i]`, re-read the slot.
pub struct KernelParamStoreShadow {
/// Epoch counter; incremented on every param store update.
/// Odd epoch = update in progress; even epoch = stable snapshot.
/// Consumers must spin-wait for an even epoch before reading `params`.
pub epoch: AtomicU64,
/// Snapshot of all tunable parameters as of `epoch`.
/// Two-level indexed identically to `KernelParamStore`:
/// group = (param_id >> 8) & 0x0F, index = param_id & 0xFF.
/// Entries whose corresponding param is unregistered hold `i64::MIN` as a sentinel.
pub params: [[AtomicI64; PARAMS_PER_GROUP]; PARAM_GROUP_COUNT],
/// Timestamp of the last shadow update (monotonic clock, nanoseconds since boot).
pub last_update_ns: AtomicU64,
/// Count of seqlock retry exhaustions (reader hit MAX_SEQLOCK_RETRIES=16
/// without obtaining a consistent snapshot). Exposed via
/// `/sys/kernel/umka/ml/shadow_stale_reads`. Non-zero indicates the writer
/// is holding the odd epoch for too long (e.g., large batch of parameter
/// updates). The Tier 2 service receives stale data when this fires —
/// policy parameters are advisory, so stale values cause suboptimal tuning
/// for one cycle, not correctness failure. Incrementing this counter is the
/// diagnostic signal for operators to investigate writer batch sizes.
pub stale_read_count: AtomicU64,
}
/// Total capacity of the shadow store: PARAM_GROUP_COUNT * PARAMS_PER_GROUP = 4096.
/// Both `KernelParamStore` and `KernelParamStoreShadow` use identical two-level
/// indexing so param_id lookup is a single array dereference in both.
pub const MAX_TUNABLE_PARAMS: usize = PARAM_GROUP_COUNT * PARAMS_PER_GROUP;
/// Shadow update protocol:
///
/// 1. The policy consumer thread (kernel-side, one per registered service)
/// processes a batch of PolicyUpdateMsg entries.
/// 2. Before writing to the shadow store:
/// `shadow.epoch.fetch_add(1, Release);` // even → odd (in-progress)
/// 3. Write updated param values to `shadow.params[group][index]` with
/// Relaxed ordering (the epoch fence handles visibility).
/// 4. After all writes:
/// `shadow.last_update_ns.store(now_ns, Relaxed);`
/// `shadow.epoch.fetch_add(1, Release);` // odd → even (stable)
///
/// Consumer (Tier 2 service) read protocol:
/// loop {
/// let e1 = shadow.epoch.load(Acquire);
/// if e1 & 1 != 0 { core::hint::spin_loop(); continue; } // odd = in-progress
/// let val = shadow.params[group][index].load(Relaxed);
/// let e2 = shadow.epoch.load(Acquire);
/// if e1 == e2 { break val; } // consistent read
/// // epoch changed — retry. After MAX_SEQLOCK_RETRIES (16) iterations
/// // (~1µs at ~60ns/iter on modern CPUs), return the last read value
/// // as a stale-but-safe fallback. Policy parameters are advisory;
/// // a stale value causes suboptimal tuning for one cycle, not
/// // correctness failure.
/// retry_count += 1;
/// if retry_count >= MAX_SEQLOCK_RETRIES { break val; }
/// }
///
/// const MAX_SEQLOCK_RETRIES: u32 = 16;
///
/// The decay function (enforce_param_decay) also uses this epoch protocol
/// when resetting expired params in the shadow.
23.1.5.2 Per-Cgroup Parameter Overrides¶
Global parameters (cgroup_id = 0 in PolicyUpdateMsg) are stored in the
KernelParamStore. Per-cgroup overrides allow a Tier 2 policy service to tune
parameters for a specific cgroup (e.g., raise tcp_congestion_min_rtt_ms for a
latency-sensitive container) without affecting other workloads.
Per-cgroup overrides are stored in a MlPolicyCss — a cgroup subsystem state struct
attached to each cgroup via the ml_policy cgroup subsystem:
/// Per-cgroup state for the ml_policy cgroup subsystem.
/// Attached to each cgroup via the `dyn_subsys[CgroupSubsysId::MlPolicy]` slot
/// ([Section 17.2](17-containers.md#control-groups)); access is RCU-protected for reads, the cgroup's
/// `config_lock` for writes. `#[repr(C)]` with the header FIRST so
/// `subsys_state::<MlPolicyCss>()` can up-cast a `dyn_subsys` slot pointer.
#[repr(C)]
pub struct MlPolicyCss {
/// Registered-subsystem header (owning `CgroupId`, subsys id, refcount).
/// MUST be the first field. Replaces the earlier `css` subsystem-state field
/// — a bare trait used as a field type (uncompilable) that also demanded an
/// owned `&Arc<Cgroup>` back-edge; the header stores a `CgroupId` integer
/// instead (resolved via `CGROUP_REGISTRY`), avoiding the reference cycle.
pub header: CgroupSubsysStateHeader,
/// Sparse set of parameter overrides for this cgroup.
/// Bounded at CAP_CGROUP_ML_PARAMS entries.
pub overrides: RcuCell<ArrayVec<ParamOverride, CAP_CGROUP_ML_PARAMS>>,
}
// kernel-internal (never KABI/wire), `#[repr(C)]` only for the offset-0 header
// up-cast. Layout: 16 (header) + 16 (RcuCell = AtomicPtr + static_ptr) = 32.
const_assert!(size_of::<MlPolicyCss>() == 32);
// SAFETY: `#[repr(C)]` with `CgroupSubsysStateHeader` as the first field.
unsafe impl CgroupSubsys for MlPolicyCss {
const SUBSYS_ID: CgroupSubsysId = CgroupSubsysId::MlPolicy;
fn header(&self) -> &CgroupSubsysStateHeader { &self.header }
}
// Lifecycle: MlPolicyCss is allocated lazily on first ML parameter write to a
// cgroup. Freed in the free_state callback when the cgroup is destroyed — pending
// overrides are drained. Updates targeting a destroyed cgroup's ID are silently
// dropped (cgroup_id lookup returns None).
//
// **Cgroup controller registration**: `ml_policy` is registered as a cgroup v2
// subsystem ([Section 17.2](17-containers.md#control-groups)) at boot (Phase 5c, after cgroup
// framework is available — cgroups ship in Phase 3). Registration provides these callbacks:
// alloc_state: allocate MlPolicyCss (slab: ml_policy_css_cache).
// free_state: drain overrides, free MlPolicyCss.
// activate: no-op (overrides are lazily populated).
// deactivate: drain overrides (same as free_state path).
// The subsystem exposes these cgroup files:
// `ml_policy.overrides` — read: JSON array of active overrides;
// write: `{"param_id":N, "value":V, "ttl_ms":T}` to set an override.
// `ml_policy.stats` — read-only: per-cgroup ML policy hit/miss counters.
// The controller has no resource charge/uncharge cycle — it is purely
// a parameter-override namespace, not a resource controller.
/// Maximum per-cgroup ML parameter overrides.
/// 8 entries is sufficient for the typical use case (scheduling + memory +
/// network tuning). 8 × 32 bytes = 256 bytes = 4 cache lines.
///
/// **Tradeoff**: With 14+ tunable subsystems, 8 overrides per cgroup is
/// tight for workloads needing per-cgroup tuning across many subsystems
/// simultaneously. However, expanding this increases the per-cgroup
/// memory footprint and worsens linear-scan latency on the lookup hot path
/// (~5 ns for 8 entries). A practical mitigation: subsystems that share
/// the same optimal tuning can be covered by a single override using
/// a composite `param_id` range. If future workloads demonstrate need
/// for >8 overrides, this constant can be increased to 16 (6 cache lines)
/// with proportional memory and latency cost. The lookup remains a linear scan
/// over at most 8 entries; 4 cache lines instead of 3 does not change the
/// ~5 ns figure quoted for the warm path.
pub const CAP_CGROUP_ML_PARAMS: usize = 8;
/// A single parameter override for one cgroup.
/// Tuple: (param_id, value, expiry_ns, last_model_seq). The decay function
/// (`enforce_cgroup_override_decay`) removes entries where
/// `expiry_ns <= now_ns`. Set `expiry_ns = u64::MAX` for permanent overrides.
#[repr(C, align(8))]
pub struct ParamOverride {
pub param_id: u32, // matches KernelTunableParam::param_id
pub _pad: u32,
pub value: i64, // overrides KernelTunableParam::current for this cgroup
pub expiry_ns: u64, // monotonic clock expiration; u64::MAX = permanent
/// Highest `PolicyUpdateMsg::model_seq` applied to this `(param_id,
/// cgroup_id)` pair. This is where the per-cgroup half of the
/// model-sequence monotonicity rule lives: the entry and its watermark are
/// updated in the same `config_lock` section, which is the only place the
/// admission decision and the write can be atomic with respect to each
/// other. The global half lives in `KernelTunableParam::last_model_seq`.
pub last_model_seq: u64,
}
// Size is load-bearing: ArrayVec<ParamOverride, 8> = 256 bytes = 4 cache lines.
const_assert!(core::mem::size_of::<ParamOverride>() == 32);
Lookup semantics (used at subsystem integration points, §23.1.5):
- Read the current task's cgroup:
CpuLocal::current_task().cgroup. - Load the cgroup's
MlPolicyCss::overridesunder an RCU read guard. - Linear-scan the
ArrayVecfor a matchingparam_id(≤8 entries; fits in 2 cache lines; faster than a hash lookup for this size). - If found, return the override value; otherwise fall through to the global
KernelParamStore::params[param_id].current.load(Acquire).
Two-level lookup adds ~5 ns in the common case (no override: RCU dereference + small empty-array scan + global load).
Write path (PolicyUpdateMsg with non-zero cgroup_id):
- Verify caller holds
Capability::KernelMlTunescoped to the target cgroup's user namespace. - Locate the target cgroup by
cgroup_idvia RCU-protected idr lookup. - Acquire the cgroup's
config_lock. Read the currentoverridesUNDER that lock (never a snapshot taken before it — seeenforce_cgroup_override_decay) and clone it. - Admit
model_seq: find the existingParamOverrideforparam_id; if one exists andmsg.model_seq <= entry.last_model_seq, the update is stale or duplicated — drop it silently, release the lock and publish nothing. This is the per-(param_id, cgroup_id)half of the model-sequence rule, performed here because only this critical section makes the decision and the write atomic with respect to each other. - Insert or update the
ParamOverrideentry in the clone, settingvalue,expiry_ns(from the dispatch context, i.e.valid_for_msresolved against the parameter'sdecay_period_ms) andlast_model_seq = msg.model_seq. - Publish via
rcu_assign()onoverrides. Releaseconfig_lock. - The old
ArrayVecis freed after the current RCU grace period (Section 3.1).
The write path resolves the target cgroup by cgroup_id and re-validates capability
scope against the resolved cgroup under the cgroup spinlock. Cgroup IDs are monotonic
u64 counters (never recycled) — a stale cgroup_id produces ENOENT, not a
use-after-free.
A Tier 2 policy service communicates with the kernel entirely through shared-memory ring buffers and ioctls — no function-pointer callbacks. This is a hard requirement because Tier 2 services run in Ring 3 (userspace processes) and the kernel cannot call function pointers into a Ring 3 address space.
/// Registration request passed via ioctl. No function pointers —
/// all communication uses shared-memory rings.
#[repr(C)]
pub struct PolicyRegisterRequest {
/// Struct size for forward/backward compatibility.
pub struct_size: u64,
/// Protocol version (currently 1).
pub version: u32,
/// Bitmask of subsystem observation streams to subscribe to.
/// Bit positions correspond to `SubsystemId` enum values.
/// 32-bit mask supports up to 32 subsystems. Current capacity:
/// 14 used, 18 available.
pub subsystem_mask: u32,
/// Requested capacity of THIS registration's policy UPDATE ring, in
/// `PolicyUpdateMsg` entries. Power of two, clamped by the kernel to
/// `[POLICY_UPDATE_RING_MIN_CAPACITY, POLICY_UPDATE_RING_MAX_CAPACITY]`.
/// Zero means "use the system default" (`POLICY_UPDATE_RING_CAPACITY`, or
/// whatever the `umka.ml_policy_ring_capacity` boot parameter set).
///
/// It does NOT control observation ring capacity, and there is no
/// per-registration observation capacity to request: one observation ring
/// per (CPU, subsystem) is SHARED by every subscriber to that subsystem, so
/// its size cannot be a property of one registration. Observation ring
/// capacity is the system-wide `umka.ml.ring_capacity` /
/// `/sys/kernel/umka/ml/ring_capacity` value described with
/// `ObservationRing` above. The update ring, by contrast, is private to one
/// registration, which is exactly why per-registration sizing is meaningful
/// there and meaningless here.
pub ring_capacity: u32,
/// Reserved for future extension. Must be zeroed by the caller;
/// the kernel rejects requests where any reserved byte is non-zero.
pub _reserved: [u8; 44],
}
const_assert!(size_of::<PolicyRegisterRequest>() == 64);
/// Registration response returned by ioctl.
#[repr(C)]
pub struct PolicyRegisterResponse {
/// File descriptor for the observation ring (mmap to access).
/// The ring contains `KernelObservation` entries produced by the kernel.
/// The service polls or uses epoll on this fd for readability.
pub obs_ring_fd: i32,
/// File descriptor for the parameter update ring (mmap to access).
/// The service writes `PolicyUpdateMsg` entries; kernel consumes them.
pub update_ring_fd: i32,
/// File descriptor for the read-only parameter store snapshot (mmap to access).
/// Maps a `KernelParamStoreShadow` page. Service reads current parameter values.
pub param_store_fd: i32,
pub _pad: i32,
/// Per-registration HMAC session key (32 bytes, HMAC-SHA3-256).
/// Filled with 32 bytes from the kernel CSPRNG during ioctl
/// processing. Unique per registration; destroyed when the service
/// deregisters (fd close). The service includes this key in every
/// `PolicyUpdateMsg` HMAC computation (Section 23.1.4.1).
///
/// **Security**: The key is returned only once, in the ioctl response
/// buffer. It is never stored in procfs, sysfs, or any other
/// user-visible location. The kernel stores a copy in the
/// `PolicyServiceRegistration` struct for HMAC verification.
///
/// **Threat model note (ptrace)**: A ptrace attacker with `PTRACE_ATTACH`
/// capability already has full control over the target process (can read
/// memory, inject syscalls, modify registers). Exposure of the session
/// key via ptrace adds no attack surface beyond existing ptrace
/// capabilities. Processes requiring stronger isolation should use
/// `PR_SET_DUMPABLE(0)` or run in a non-dumpable cgroup.
pub session_key: [u8; 32],
}
// PolicyRegisterResponse: 4 + 4 + 4 + 4 + 32 = 48 bytes.
// Returned via ioctl to userspace — ABI struct requires size verification.
const_assert!(size_of::<PolicyRegisterResponse>() == 48);
Registration protocol: A Tier 2 policy service registers with the kernel via
/dev/umka_ml_policy (a character device created by umka-nucleus at boot):
- Open: The service opens
/dev/umka_ml_policy. The kernel allocates aPolicyServiceRegistrationstruct. - Register:
ioctl(fd, ML_POLICY_REGISTER, &PolicyRegisterRequest). The kernel validates: - Caller holds
Capability::KernelMlTune(per-user-namespace). struct_sizeandversionare compatible.subsystem_maskcontains only valid subsystem bits. On success, returns aPolicyRegisterResponsewith three file descriptors: the observation ring fd (readable, pollable), the update ring fd (writable), and the parameter store fd (read-only mmap).- Run: The service polls
obs_ring_fdfor new observations, readsKernelObservationentries from the mmap'd ring, runs its ML model, and writesPolicyUpdateMsgentries to the update ring. The kernel's policy consumer thread reads update messages and applies them to the parameter store. - Close / Deregistration: Closing the fd deregisters the service. The kernel performs the following steps in order:
- Release observation cursors: For every subsystem in this
registration's
subsystem_mask, on every CPU, release itsObservationCursor(storeCURSOR_SLOT_FREEintoowner) and decrement the ring'ssubscriberscount. After this step no observation is delivered to THIS service, because delivery to a service is precisely its cursor. Only for the subsystems whosesubscriberscount reached zero does the kernel additionally disableOBSERVE_ENABLED[subsystem]— the static key is per-SUBSYSTEM and refcounted (enabled on the first registration for that subsystem, disabled on the last), NOT per-registration. There has never been a per-registration key, and toggling the subsystem key on any single deregistration would starve every peer still subscribed to that subsystem. - RCU grace period: Wait for a full RCU grace period. For a subsystem
that went quiet in step 1 this ensures every in-flight
observe_kernel!call that entered the ring write path before the key toggle has completed, so the ring can be freed in step 6. For a subsystem still serving peers the producer keeps running and only THIS registration's cursor and mapping are being retired — which is why the drain premise below is scoped to this service's rings, not to the observation bus as a whole. - Drain update ring: Read and apply the remaining
PolicyUpdateMsgentries from the update ring, under the bounded-drain rule above (at mostupdate_ring_capacityentries, however large awrite_idxthe service left behind). Each entry is validated via HMAC; invalid entries are silently discarded. - Remove registration: Remove the
PolicyServiceRegistrationfrom the global registry. After this step, no kernel consumer thread will attempt to read from this service's rings. - Unmap rings: Unmap the observation ring, update ring, and parameter store mmap regions from the service's address space.
- Free resources: Free the per-CPU observation rings whose
subscriberscount reached zero in step 1 (rings still serving a peer registration are left running), the update ring, the session key, and the per-registration rate limiter. - Parameter decay: Parameters managed by this service are NOT immediately
reverted. They retain their current values until
valid_for_msexpires, at which point they decay to defaults. Crash recovery: If the Tier 2 service is killed (SIGKILL, OOM), the fd is closed by the kernel's process exit cleanup path (close_task_files()), triggering the same deregistration sequence (steps 1-7 above). Partially-written ring entries may contain corrupt data — step 3 validates each entry's HMAC and discards entries with invalid or incomplete HMAC fields.
ML_POLICY_REGISTER ioctl number: _IOW('M', 0x01, PolicyRegisterRequest).
Error codes returned by ML_POLICY_REGISTER:
| Errno | Condition |
|---|---|
EPERM |
Caller does not hold Capability::KernelMlTune in the current user namespace |
EINVAL |
struct_size does not match any known PolicyRegisterRequest layout version |
EINVAL |
version is not compatible with the kernel's supported range |
EINVAL |
subsystem_mask contains bits for undefined subsystems |
EBUSY |
Maximum number of concurrent policy services reached (MAX_POLICY_SERVICES = 4; a hard bound, since it sizes each ring's cursor array). Overlapping subsystem_mask bits are NOT a reason for EBUSY — subscribers to one subsystem share the ring through independent cursors |
ENOMEM |
Failed to allocate observation ring, update ring, or parameter store mapping |
EACCES |
Policy service binary does not have a valid ML-DSA-65 signature (when CONFIG_ML_POLICY_SIGNED is enabled) |
/// Message type discriminants for the policy update ring.
/// The first byte of every message on the update ring is a `msg_type` field,
/// allowing the kernel to demultiplex `PolicyUpdateMsg` from control messages
/// (e.g., `ObservationThrottleMsg`) without ambiguity.
pub const MSG_TYPE_POLICY_UPDATE: u8 = 2;
pub const MSG_TYPE_THROTTLE: u8 = 1;
/// Message sent from Tier 2 policy service to the kernel to update a parameter.
/// Submitted via a dedicated policy update ring buffer (separate from observations).
///
/// Layout is fixed at 128 bytes for KABI stability. All implicit compiler padding
/// is made explicit via named `_pad*` fields so that the layout is identical
/// across compiler versions, optimization levels, and target architectures.
/// Future fields may be added within `_reserved`; increment `PolicyRegisterRequest::version`
/// when doing so.
///
/// **Version scope**: Individual `PolicyUpdateMsg` entries do not carry a
/// version field. The version is established per-registration (per-session)
/// via `PolicyRegisterRequest::version`. The service and kernel agree on a
/// single wire format version for the entire session. This avoids 4 bytes
/// of per-message overhead on a 128-byte message.
///
/// The `msg_type` field (first byte) is `MSG_TYPE_POLICY_UPDATE = 2`, which
/// the kernel uses to distinguish this message from control messages such as
/// `ObservationThrottleMsg` (`MSG_TYPE_THROTTLE = 1`). Without this field,
/// the low byte of `param_id` on little-endian would collide with throttle
/// messages when `param_id == 1`.
#[repr(C)]
pub struct PolicyUpdateMsg {
/// Message type discriminant: `MSG_TYPE_POLICY_UPDATE = 2`.
/// Must be the first byte so the kernel can demultiplex all
/// message types on the update ring by inspecting `ring[offset + 0]`.
pub msg_type: u8,
/// Explicit padding after msg_type to align param_id to 4 bytes.
pub _pad_mt: [u8; 3],
/// Which parameter to update.
pub param_id: u32,
/// New value — kernel clamps to [min_value, max_value] before applying.
pub new_value: i64,
/// Parameter is valid for this many ms (0 = use param's default decay_period).
/// If the service crashes, the parameter decays after this interval.
pub valid_for_ms: u32,
/// Explicit padding to align `model_seq` to 8 bytes.
pub _pad1: [u8; 4],
/// Monotonic sequence number from the ML model that produced this update.
/// Out-of-order or duplicate updates (lower seq than current) are silently dropped.
pub model_seq: u64,
/// Optional: restrict update to a specific cgroup (0 = global / kernel-wide).
pub cgroup_id: u64,
/// Reserved for future fields (increment `version` when adding).
/// Placed before HMAC so future fields are automatically authenticated.
pub _reserved: [u8; 56],
/// HMAC-SHA3-256 over all preceding bytes (msg_type through _reserved).
/// HMAC-last ensures any future field addition in _reserved is
/// automatically covered by the authentication. Computed with the
/// per-registration session key (derived during ioctl handshake).
pub hmac: [u8; 32],
}
const _: () = assert!(core::mem::size_of::<PolicyUpdateMsg>() == 128);
/// Control message sent by a Tier 2 policy service to throttle observation
/// delivery. Written to the update ring with `msg_type = MSG_TYPE_THROTTLE`
/// (discriminated by the first byte, same as `PolicyUpdateMsg::msg_type`).
///
/// Use case: when the service's model cannot keep up with observation rate
/// (detected via its own cursor lag, or the aggregate
/// `ObservationRingShared::overflow` counter), it sends a
/// throttle message to reduce kernel-side emission.
#[repr(C)]
pub struct ObservationThrottleMsg {
/// Message type discriminant: `MSG_TYPE_THROTTLE` (= 1).
pub msg_type: u8,
pub _pad0: [u8; 3],
/// Target subsystem to throttle (SubsystemId discriminant value).
/// 0xFF = all subsystems for this registration.
///
/// **Scope**: Throttle affects only this service's observation ring
/// delivery, not kernel-side emission or other registered services.
/// The kernel continues to emit observations internally (for other
/// consumers or for internal health monitoring) — only the delivery
/// to this service's per-CPU observation ring is sampled down.
pub subsystem: u8,
pub _pad1: [u8; 3],
/// Throttle factor: emit 1-in-N observations. 1 = no throttle (full rate).
/// 2 = emit every other observation. 0 = pause entirely (no observations).
/// The kernel applies this as a per-subsystem sampling divisor.
pub throttle_factor: u32,
/// Duration in milliseconds. After this period, the kernel reverts to
/// full-rate emission (throttle_factor = 1). 0 = indefinite (until next
/// throttle message or service deregistration).
pub duration_ms: u32,
/// Reserved for future fields. Placed before HMAC so that future fields
/// are automatically covered by HMAC verification (same pattern as
/// PolicyUpdateMsg).
pub _reserved: [u8; 80],
/// HMAC-SHA3-256 over all preceding bytes (msg_type through _reserved)
/// using the per-registration session key.
/// Replay protection unnecessary: throttle messages are written to the
/// service's own ring. Replay is a self-attack.
pub hmac: [u8; 32],
}
const _: () = assert!(core::mem::size_of::<ObservationThrottleMsg>() == 128);
The kernel validates the HMAC on every message consumption. Invalid HMACs increment a
per-service hmac_reject_count counter and raise an FMA alert.
Policy update ring buffer — shared memory between Tier 2 service and kernel:
/// Ring buffer for PolicyUpdateMsg entries. Allocated by the kernel,
/// mmap'd into the Tier 2 service's address space (writable for the service,
/// readable for the kernel).
///
/// Layout: [PolicyUpdateRingHeader][PolicyUpdateMsg; capacity]
/// The service writes messages at `write_idx`, the kernel reads from `read_idx`.
#[repr(C)]
pub struct PolicyUpdateRingHeader {
/// Ring capacity in entries (power of two). Set by kernel at allocation,
/// published here for the service's index arithmetic. INFORMATIONAL ONLY —
/// this page is service-writable, so the kernel bounds itself by
/// `PolicyServiceRegistration::update_ring_capacity` instead.
pub capacity: u32,
pub _pad0: u32,
/// Write index (next slot to write). Updated by the Tier 2 service
/// with Release ordering after writing the message. UNTRUSTED: the drain
/// clamps `write_idx - read_idx` to the kernel's capacity before doing any
/// work proportional to it.
pub write_idx: AtomicU64,
/// Read index (next slot to read), published by the kernel's policy
/// consumer thread with Release ordering after processing. A mirror of
/// `PolicyServiceRegistration::update_ring_read_idx`, which is the copy the
/// kernel actually reads — a service cannot rewind the kernel by writing
/// here, only confuse its own backpressure calculation.
pub read_idx: AtomicU64,
/// Overflow counter (incremented by service if ring is full).
pub overflow: AtomicU64,
}
const_assert!(core::mem::size_of::<PolicyUpdateRingHeader>() == 32);
/// Default capacity: 256 entries (256 × 128 = 32 KiB + 32-byte header).
/// Fits in 9 4 KiB pages. The kernel allocates this from its ring buffer
/// slab and maps it into the service's address space via the `update_ring_fd`
/// returned by ML_POLICY_REGISTER.
///
/// **Override**: The service may request a different capacity via the
/// `PolicyRegisterRequest::ring_capacity` field (power-of-two, clamped to
/// `[POLICY_UPDATE_RING_MIN_CAPACITY, POLICY_UPDATE_RING_MAX_CAPACITY]`). If 0
/// or omitted, the default is used. The kernel parameter
/// `umka.ml_policy_ring_capacity` (boot parameter) sets the system-wide
/// default, overriding this constant. Larger rings reduce backpressure
/// for bursty services at the cost of per-registration memory.
pub const POLICY_UPDATE_RING_CAPACITY: u32 = 256;
/// Bounds on a requested update-ring capacity. The accepted capacity is
/// recorded in `PolicyServiceRegistration::update_ring_capacity` — the KERNEL's
/// copy, which is what the drain loop bounds itself by. The `capacity` field in
/// the mmap'd `PolicyUpdateRingHeader` is informational for the service and is
/// never trusted by the kernel: that page is service-writable.
pub const POLICY_UPDATE_RING_MIN_CAPACITY: u32 = 64;
pub const POLICY_UPDATE_RING_MAX_CAPACITY: u32 = 4096;
The service submits updates by writing a PolicyUpdateMsg at
buffer[write_idx & (capacity - 1)], then incrementing write_idx with
Release ordering. The kernel's policy consumer thread (one per registered
service) polls write_idx > read_idx and processes entries sequentially.
If write_idx - read_idx >= capacity, the ring is full and the service
must wait (spin or poll on read_idx advancing).
Bounded drain (normative). write_idx lives in the service-writable
mapping, so the kernel treats it as an untrusted input and never derives an
unbounded amount of work from it. Every drain — the consumer thread's and
deregistration step 3's — computes
// `capacity` is svc.update_ring_capacity, the KERNEL's copy; the header's
// `capacity` field is service-writable and is never read here.
let capacity = svc.update_ring_capacity as u64;
let mut read = svc.update_ring_read_idx.load(Relaxed); // kernel-owned
let write = header.write_idx.load(Acquire); // untrusted
let mut available = write.wrapping_sub(read);
if available > capacity {
// Impossible distance: the service's ring state is corrupt (crashed
// mid-write, or hostile). At most `capacity` slots can hold unread
// messages, so RESYNCHRONIZE the kernel's own read index to the oldest
// slot that can still hold one — `write - capacity` — count the loss, and
// raise an FMA alert (`category: PolicyConsumerMisbehaving, service_id`).
// Without this a service could publish an arbitrary write_idx and make
// the kernel process an attacker-chosen number of aliased slots.
//
// Moving `read` is the load-bearing half, and clamping `available` alone
// is NOT a substitute: the clamp bounds one pass while leaving `read`
// where it was, so the impossible distance SURVIVES the pass and every
// later pass re-enters this branch with the same enormous
// `write - read`. The drain would then be permanently in resync mode,
// never converging, and the "drain what remains" caller
// (deregistration step 3) would keep finding work for as long as the
// service's chosen `write_idx` said it should — which is precisely the
// unbounded, service-controlled work this rule exists to deny.
read = write.wrapping_sub(capacity);
svc.update_ring_read_idx.store(read, Relaxed);
available = capacity;
svc.ring_desync_count.fetch_add(1, Relaxed);
}
let to_process = available.min(POLICY_DRAIN_BATCH as u64);
and processes exactly to_process entries, advancing its own read_idx by that
amount. POLICY_DRAIN_BATCH additionally bounds one pass so a full ring cannot
monopolize the consumer thread; the rate limiter still governs how much of it is
accepted.
Post-condition (what makes teardown terminate). After the block above,
write - read <= capacity holds unconditionally, whatever the service wrote.
A caller that drains "what remains" therefore completes in at most
ceil(capacity / POLICY_DRAIN_BATCH) passes. Deregistration step 3 additionally
takes its write_idx snapshot ONCE, before its first pass, and drains only up
to that snapshot: by then the registration is being retired, so a write_idx
that keeps advancing is a misbehaving service, and a misbehaving service must
not be able to extend its own teardown.
/// Maximum `PolicyUpdateMsg` entries processed in one drain pass.
pub const POLICY_DRAIN_BATCH: u32 = 64;
The kernel validates all PolicyUpdateMsg entries before applying:
1. param_id must exist in KernelParamStore
2. new_value is clamped to [min_value, max_value] (never rejected; clamped silently)
3. Caller must hold Capability::KernelMlTune (Tier 2 services receive this at registration if granted by the operator; see Section 23.1.8)
4. model_seq must be ≥ the last applied seq for this (param_id, cgroup_id) pair
Parameter ID enumeration — Every tunable parameter has an explicit param_id value.
IDs are partitioned by subsystem prefix to allow independent subsystem evolution.
The KernelTunableParam::param_id field stores these values.
ParamId ranges are allocated per subsystem in 256-entry blocks, mapped to
SubsystemId via (param_id >> 8) & 0x0F + 1:
| Range | Group | SubsystemId | Name |
|---|---|---|---|
| 0x0000-0x00FF | 0 | 1 | Scheduler |
| 0x0100-0x01FF | 1 | 2 | MemoryManager |
| 0x0200-0x02FF | 2 | 3 | TcpStack |
| 0x0300-0x03FF | 3 | 4 | BlockIo |
| 0x0400-0x04FF | 4 | 5 | PowerManager |
| 0x0500-0x05FF | 5 | 6 | FmaHealth |
| 0x0600-0x06FF | 6 | 7 | NvmeDriver |
| 0x0700-0x07FF | 7 | 8 | NetworkDriver |
| 0x0800-0x08FF | 8 | 9 | IoScheduler |
| 0x0900-0x09FF | 9 | 10 | Gpu |
| 0x0A00-0x0AFF | 10 | 11 | Storage |
| 0x0B00-0x0BFF | 11 | 12 | Accel |
| 0x0C00-0x0CFF | 12 | 13 | VfsLayer |
| 0x0D00-0x0DFF | 13 | 14 | ContainerMgr |
| 0x0E00-0x0EFF | 14 | 15 | Kvm |
IDs within a range are assigned contiguously from the base.
/// Well-known parameter IDs. Subsystem ranges are 256 entries wide;
/// new parameters are appended within their range (never reuse deleted IDs).
#[repr(u32)]
pub enum ParamId {
// — Scheduler (0x0000–0x00FF) —
SchedEevdfWeightScale = 0x0000,
SchedMigrationBenefitThreshold = 0x0001,
SchedPreemptionLatencyBudget = 0x0002,
SchedEasEnergyBias = 0x0003,
SchedCfsBurstQuotaUs = 0x0004,
/// Intent scheduler PID controller: proportional gain for latency.
/// This is an ML policy tuning knob that controls HOW the intent
/// scheduler adjusts parameters — not part of the ResourceIntent struct
/// itself (defined in Section 7.7.1). ResourceIntent declares WHAT the
/// workload needs (target_latency_ns, target_ops_per_sec, etc.); these
/// ParamId entries control the PID controller that tunes kernel internals
/// to satisfy those intent targets.
SchedIntentKpLatency = 0x0005,
/// Intent scheduler PID controller: derivative gain for latency.
SchedIntentKdLatency = 0x0006,
/// Maximum single-step adjustment magnitude for intent tuning.
SchedIntentMaxAdjustment = 0x0007,
/// Hold time (ms) after an intent adjustment before re-evaluating.
SchedIntentHoldTimeMs = 0x0008,
/// Resched urgency default. Controls whether `resched_curr()` uses `Eager`
/// (immediate IPI) or `Lazy` (deferred to next tick). UmkaOS-original enum
/// `ReschedUrgency` — see [Section 7.1](07-scheduling.md#scheduler).
/// Range: 0 = Lazy (default), 1 = Eager. ML can shift the default policy
/// per cgroup based on latency sensitivity.
SchedReschedUrgency = 0x0009,
/// SCHED_RR default timeslice, nanoseconds. Bounded
/// [1_000_000 (1 ms), 1_000_000_000 (1 s)]; default 100_000_000
/// (100 ms — the value userspace expects via sched_rr_get_interval(2)).
/// ABI face: `/proc/sys/kernel/sched_rr_timeslice_ms` (Linux-compatible
/// sysctl, [Section 20.9](20-observability.md#kernel-parameter-store)). ML can shorten the quantum for
/// interactive RR mixes or lengthen it for throughput-bound RR batches.
/// See [Section 7.1](07-scheduling.md#scheduler--per-class-tick-functions).
SchedRrTimesliceNs = 0x000A,
/// EEVDF protected-slice fraction, percent. The share of a running entity's
/// virtual slice — measured from the start of that slice — during which it
/// is shielded from lazy preemption; the remaining `100 - pct`% of the slice
/// is the preemptible tail. HIGHER = MORE protection: a batch cgroup runs at
/// 70 (long protected span, fewer context switches, throughput bias) while a
/// latency-sensitive cgroup runs at 20 (short protected span, early
/// preemption points). Read by `set_protect_slice()` (Evolvable). Bounded
/// [10, 90] — the bounds keep both degenerate ends (never protected /
/// protected to the deadline) out of reach; default from `BOOT_PROTECT_PCT`.
/// See [Section 7.1](07-scheduling.md#scheduler--eevdf-algorithm-specification).
SchedProtectFraction = 0x000B,
/// CBS work-conserving steal fraction, percent. The fraction of a
/// sibling's available bandwidth a CPU may steal per attempt (replaces the
/// hardcoded `avail / 2` in `cbs_try_steal()`, Evolvable hot path).
/// Bounded [1, 100]; default 50. See [Section 7.6](07-scheduling.md#cpu-bandwidth-guarantees).
CbsStealFractionPct = 0x000C,
// — Memory Manager (0x0100–0x01FF) —
MemReclaimAggressiveness = 0x0100,
MemPrefetchWindowPages = 0x0101,
MemNumaMigrationThreshold = 0x0102,
MemCompressEntropyThreshold = 0x0103,
MemSwapLocalRatio = 0x0104,
/// OOM victim score adjustment from ML model. Range: [-500, +500].
/// Applied additively to the base OOM score. Bounded to prevent
/// overriding administrator-set oom_score_adj by more than 25% of total range.
/// See [Section 4.2](04-memory.md#physical-memory-allocator--ml-policy-integration-intelligent-oom-victim-selection).
MemOomScoreAdjustment = 0x0105,
/// OOM proactive eviction threshold. When ML predicts OOM within N seconds,
/// trigger proactive soft reclaim (memory.reclaim) on the predicted victim's
/// cgroup before hard OOM occurs. Range: 0 (disabled) to 60 (seconds).
MemOomProactiveEvictionSec = 0x0106,
// — TCP / Network (0x0200–0x02FF) —
NetTcpInitialCwndScale = 0x0200,
NetBbrProbeRttIntervalMs = 0x0201,
NetTcpPacingGainPct = 0x0202,
NetEcnAggressiveness = 0x0203,
// — BlockIo (0x0300–0x03FF) —
BlockIoBatchSizeThreshold = 0x0300,
// — Power Manager (0x0400–0x04FF) —
PowerRaplPackagePowerW = 0x0400,
PowerCpuFreqMinMhz = 0x0401,
PowerAccelPowerCapW = 0x0402,
PowerThermalTargetC = 0x0403,
// — FMA / Observability (0x0500–0x05FF) —
FmaAnomalyAlertThreshold = 0x0500,
FmaHealthCheckIntervalMs = 0x0501,
FmaErrorRateWindowMs = 0x0502,
// — NvmeDriver (0x0600–0x06FF) —
NvmeQueueDepthTuning = 0x0600,
// — NetworkDriver (0x0700–0x07FF) —
NetDriverCoalesceUs = 0x0700,
// — I/O Scheduler (0x0800–0x08FF) —
IoReadaheadPages = 0x0800,
IoQueueDepthTarget = 0x0801,
IoLatencyTargetUs = 0x0802,
// — Gpu (0x0900–0x09FF) —
GpuFreqTargetMhz = 0x0900,
// — Storage (0x0A00–0x0AFF) —
StorageWritebackThresholdPct = 0x0A00,
// — Accel (0x0B00–0x0BFF) —
AccelBatchSizeHint = 0x0B00,
// — VfsLayer (0x0C00–0x0CFF) —
VfsDentryCacheTargetPct = 0x0C00,
// — ContainerMgr (0x0D00–0x0DFF) —
CgroupMemoryReclaimPressurePct = 0x0D00,
// — Kvm (0x0E00–0x0EFF) —
/// Per-HLT halt-poll window, nanoseconds. 0 = polling disabled (every
/// HLT with no pending interrupt blocks immediately). Backing store:
/// `KVM_HALT_POLL_NS` ([Section 18.1](18-virtualization.md#host-and-guest-integration)). This value is
/// auto-tuned by Linux (`halt_poll_ns_grow`/`halt_poll_ns_shrink`
/// adaptive polling, `virt/kvm/kvm_main.c`) because a static default
/// is wrong for most workloads — the canonical proof that this knob
/// is ML-tunable territory. `KvmObs::HaltPollOutcome` is the training
/// signal.
KvmHaltPollNs = 0x0E00,
/// Per-quantum aggregate halt-poll budget as a percentage of the
/// scheduling quantum. Bounds the total idle-accounted poll time per
/// quantum (fairness-exploit containment — see
/// [Section 18.3](18-virtualization.md#kvm-operational--vcpu-scheduling-integration)). Backing store:
/// `KVM_HALT_POLL_BUDGET_PCT`.
KvmHaltPollBudgetPct = 0x0E01,
}
impl ParamId {
/// Decode a wire `u32` (from `PolicyUpdateMsg.param_id`) into a typed,
/// registered `ParamId`, or `None` if it is outside the 16×256 parameter
/// space or names no registered parameter.
///
/// Validation is against `KERNEL_PARAM_STORE`: a slot is set only by
/// `register_param!(ParamId::X, ..)`, so a registered slot whose stored
/// `param_id` equals `v` proves `v` is a real `ParamId` discriminant. This
/// auto-tracks every parameter as it is added — no per-`ParamId` conversion
/// arm to maintain — and rejects both out-of-range and unregistered values
/// (the latter also fail later at handler dispatch).
pub fn try_from_u32(v: u32) -> Option<ParamId> {
// Outside the 16-group × 256-index encoding → definitely not a ParamId.
if v >= (PARAM_GROUP_COUNT as u32) << 8 {
return None;
}
let group = ((v >> 8) & 0x0F) as usize;
let index = (v & 0xFF) as usize;
if !KERNEL_PARAM_STORE.registered[group][index].load(Ordering::Acquire) {
return None;
}
// SAFETY: the Acquire load above pairs with `register()`'s Release
// store, so a `true` flag means `params[group][index]` is initialized.
let stored = unsafe {
KERNEL_PARAM_STORE.params[group][index].assume_init_ref()
};
// Confirm the FULL u32 matches the registered discriminant (the
// group/index decode ignores nothing here since v < 0x1000, but the
// equality also rejects a slot registered for a different id).
if stored.param_id != v {
return None;
}
// SAFETY: `v == stored.param_id`, and `param_id` was written from
// `ParamId::_ as u32` at registration, so `v` is a valid `#[repr(u32)]`
// discriminant of `ParamId`.
Some(unsafe { core::mem::transmute::<u32, ParamId>(v) })
}
}
/// Per-subsystem observation type enums. The `obs_type` field in
/// `KernelObservation` is cast from these enums (u16 discriminant).
/// Scheduler observation types.
#[repr(u16)]
pub enum SchedObs {
/// Task wakeup latency.
TaskWoke = 0,
/// NUMA/load migration decision.
MigrateDecision = 1,
/// Preemption occurred.
PreemptionEvent = 2,
/// Per-CPU runqueue snapshot (every 10ms).
RunqueueStats = 3,
/// EAS placement decision.
EasDecision = 4,
/// Intent-based scheduler feedback: current intent status for a cgroup.
/// Emitted by `intent_observe_status()` on each intent evaluation cycle.
IntentStatus = 5,
/// Latency histogram snapshot for intent-based scheduling evaluation.
/// Emitted periodically (every 60s) by the intent scheduler subsystem.
LatencyHistogram = 6,
}
/// Memory manager observation types.
#[repr(u16)]
pub enum MemObs {
/// Page fault event.
PageFault = 0,
/// Which page was evicted from LRU.
EvictionDecision = 1,
/// NUMA migration result.
NumaMigration = 2,
/// Previously-evicted page faulted again.
RefaultRecord = 3,
/// Memory pressure snapshot (every 1s).
MemPressure = 4,
/// OOM killer invoked — candidate features for victim selection.
/// Features: constraint_type, candidate_count, victim_pid, victim_rss,
/// victim_swap, victim_score, victim_cgroup_id, free_pages, psi_stall_us.
/// See [Section 4.2](04-memory.md#physical-memory-allocator--ml-policy-integration-intelligent-oom-victim-selection).
OomVictimSelection = 5,
/// OOM outcome — emitted 5s after kill with measured recovery metrics.
/// Features: victim_pid, pages_freed, time_to_recovery_ms, service_restart_ms,
/// cascading_kills, cgroup_oom_count_after, was_ml_adjusted, baseline_score.
/// This is the ground-truth feedback signal for the OOM ML model.
OomOutcome = 6,
}
/// TCP / Network observation types.
#[repr(u16)]
pub enum NetObs {
/// Congestion window event.
CongestionEvent = 0,
/// Per-flow statistics snapshot.
FlowStats = 1,
/// Routing decision.
RouteDecision = 2,
}
/// I/O scheduler observation types.
#[repr(u16)]
pub enum IoObs {
/// I/O request completion latency.
IoCompletion = 0,
/// Queue depth snapshot.
QueueDepthStats = 1,
}
/// FMA / Observability observation types.
#[repr(u16)]
pub enum FmaObs {
/// Device health metric update.
FmaHealth = 0,
/// Anomaly score exceeded threshold.
AnomalyAlert = 1,
}
/// Power manager observation types.
#[repr(u16)]
pub enum PowerObs {
/// RAPL energy sample.
EnergySample = 0,
/// Thermal throttle event.
ThermalThrottle = 1,
/// CPU frequency transition.
FreqTransition = 2,
}
/// KVM observation types.
#[repr(u16)]
pub enum KvmObs {
/// Halt-poll window closed — one observation per polled HLT exit,
/// emitted whether the poll succeeded (interrupt arrived inside the
/// window) or expired (vCPU blocked). The success/expiry split at a
/// given window size is exactly the signal an ML policy needs to
/// reproduce — and beat — Linux's grow/shrink heuristic.
HaltPollOutcome = 0,
}
23.1.6 Subsystem Integration Catalog¶
Each subsystem that participates in ML tuning registers its parameters at boot. The tables below define the initial parameter sets and observation types.
23.1.6.1.1 Scheduler (Section 7.1)¶
Observation types (SchedObs):
| obs_type | features[0..9] | Meaning |
|---|---|---|
TaskWoke |
latency_ns, runq_len, cpu, prev_cpu, cgroup_id | Task wakeup latency |
MigrateDecision |
src_cpu, dst_cpu, task_weight, queue_diff, benefit_ns | NUMA/load migration |
PreemptionEvent |
preemptor_prio, preemptee_prio, cgroup_id, — | Preemption occurred |
RunqueueStats |
runq_len, avg_vruntime, nr_throttled, nr_rt, cgroup_id | Per-CPU runqueue snapshot (every 10ms) |
EasDecision |
task_cgroup, chosen_cpu, energy_delta_uw, load_delta, — | EAS placement |
Tunable parameters (SchedParam):
| param_name | default | min | max | decay | Effect |
|---|---|---|---|---|---|
eevdf_weight_scale |
100 | 50 | 200 | 60s | Scale factor for virtual deadline computation (100 = baseline) |
migration_benefit_threshold |
1000 | 100 | 50000 | 30s | Minimum ns benefit to justify task migration |
preemption_latency_budget |
1000 | 100 | 10000 | 30s | Maximum μs a lower-priority task runs before preemption check |
eas_energy_bias |
50 | 0 | 100 | 60s | 0 = performance, 100 = max energy saving in EAS decisions |
cfs_burst_quota_us |
0 | 0 | 100000 | 60s | CFS burst tolerance for cgroup (0 = disabled) |
23.1.6.1.2 Memory Manager (Section 4)¶
Observation types (MemObs):
| obs_type | features[0..9] | Meaning |
|---|---|---|
PageFault |
cgroup, fault_type, addr_band, file_offset_band, prefetch_hit | Page fault event |
EvictionDecision |
cgroup, evicted_page_type, lru_age, refault_distance, — | Which page was evicted |
NumaMigration |
src_node, dst_node, cgroup, pages_moved, benefit_ns | NUMA migration result |
RefaultRecord |
cgroup, file_inode, page_offset, time_since_evict_ms | Previously-evicted page faulted again |
MemPressure |
node, free_pages, anon_pages, file_pages, slab_pages | Memory pressure snapshot (every 1s) |
OomVictimSelection |
constraint_type, candidate_count, victim_pid, victim_rss, victim_swap, victim_score, victim_cgroup_id, free_pages, psi_stall_us | OOM killer invoked — candidate features for ML-adjusted victim selection |
OomOutcome |
victim_pid, pages_freed, time_to_recovery_ms, service_restart_ms, cascading_kills, oom_count_after, was_ml_adjusted, baseline_score | Ground-truth feedback: measured outcome 5s after OOM kill |
Tunable parameters (MemParam):
| param_name | default | min | max | decay | Effect |
|---|---|---|---|---|---|
reclaim_aggressiveness |
100 | 25 | 400 | 30s | LRU reclaim rate relative to baseline |
prefetch_window_pages |
8 | 1 | 128 | 30s | Max pages to prefetch per fault event |
numa_migration_threshold |
200 | 50 | 2000 | 60s | Minimum benefit (ns) per page to trigger NUMA migration |
compress_entropy_threshold |
128 | 64 | 255 | 60s | Page entropy (0–255) above which compression is skipped |
swap_local_ratio |
80 | 0 | 100 | 30s | % of swap that goes to local vs RDMA remote swap (Section 5) |
oom_score_adjustment |
0 | -500 | 500 | 30s | ML-derived OOM score adjustment per candidate. Bounded: cannot override admin-set oom_score_adj by more than 25% of range. See Section 4.2. |
oom_proactive_eviction_sec |
0 | 0 | 60 | 30s | Proactive soft reclaim window (seconds). When ML predicts OOM within this horizon, trigger memory.reclaim on the predicted victim's cgroup before hard OOM occurs. 0 = disabled (pure reactive). |
23.1.6.1.3 TCP / Network (Section 16.8)¶
Observation types (NetObs):
| obs_type | features[0..9] | Meaning |
|---|---|---|
CongestionEvent |
cgroup, cwnd, rtt_us, retransmits, bandwidth_mbps | Congestion window event |
FlowStats |
cgroup, bytes_sent, bytes_recv, rtt_p99_us, loss_pct | Per-flow statistics snapshot |
RouteDecision |
src_addr_band, dst_addr_band, chosen_dev, alternative_dev, latency_us | Routing decision |
Tunable parameters (NetParam):
| param_name | default | min | max | decay | Effect |
|---|---|---|---|---|---|
tcp_initial_cwnd_scale |
10 | 2 | 100 | 30s | Initial congestion window (segments) per cgroup |
bbr_probe_rtt_interval_ms |
10000 | 200 | 60000 | 60s | BBR min-RTT probe interval |
tcp_pacing_gain_pct |
125 | 100 | 200 | 30s | BBR pacing gain percentage |
ecn_aggressiveness |
1 | 0 | 3 | 60s | 0=off, 1=ECT(1), 2=ECT(0), 3=always mark |
23.1.6.1.4 I/O Scheduler (Section 15.18)¶
Tunable parameters (IoParam):
| param_name | default | min | max | decay | Effect |
|---|---|---|---|---|---|
readahead_pages |
32 | 1 | 512 | 30s | Readahead window in pages |
queue_depth_target |
32 | 4 | 1024 | 30s | Target NVMe queue depth per cgroup |
latency_target_us |
0 | 0 | 100000 | 30s | 0 = throughput mode; >0 = latency target (μs) |
23.1.6.1.5 Power Manager (Section 7.7)¶
Tunable parameters (PowerParam):
| param_name | default | min | max | decay | Effect |
|---|---|---|---|---|---|
rapl_package_power_w |
0 | 5 | 400 | 10s | CPU package power cap (W); 0 = no cap |
cpu_freq_min_mhz |
0 | 0 | 10000 | 10s | Minimum CPU frequency; 0 = hardware default |
accel_power_cap_w |
0 | 0 | 1000 | 10s | Per-accelerator power cap; 0 = hardware default |
thermal_target_c |
95 | 60 | 105 | 5s | Thermal throttle target (°C) |
23.1.6.1.6 FMA / Observability (Section 20.1)¶
Tunable parameters (FmaParam):
| param_name | default | min | max | decay | Effect |
|---|---|---|---|---|---|
anomaly_alert_threshold |
80 | 10 | 100 | 60s | Score (0–100) above which FMA raises alert |
health_check_interval_ms |
1000 | 100 | 60000 | 0 | How frequently to poll device health counters |
error_rate_window_ms |
5000 | 100 | 300000 | 0 | Error rate measurement window |
23.1.6.1.7 KVM (Section 18.3)¶
Observation types (KvmObs):
| obs_type | features[0..9] | Meaning |
|---|---|---|
HaltPollOutcome |
poll_ns_cap, polled_ns, budget_remaining_ns, outcome (0=expired, 1=poll success, 2=signal), vm_id | Halt-poll window closed (per polled HLT exit) |
Tunable parameters (KvmParam):
| param_name | default | min | max | decay | Effect |
|---|---|---|---|---|---|
halt_poll_ns |
200000 | 0 | 5000000 | 60s | Per-HLT halt-poll window (ns); 0 = polling disabled |
halt_poll_budget_pct |
10 | 0 | 50 | 60s | Per-quantum aggregate halt-poll budget (% of quantum) |
The default column here is the shipped baseline; if the administrator set a
boot/driver override (umka.driver.umka-kvm.halt_poll_ns,
Section 11.4), umka-kvm
registers the param with the merged admin value as its default_value — so ML
tuning decays back to the ADMIN's chosen baseline, not the shipped one.
23.1.7 Heavy Model Integration Pattern¶
The "big model" pattern enables a Tier 2 driver to call any model — a large transformer,
an LLM, a remote inference service, or a custom RL policy — and feed the results back as
PolicyUpdateMsg entries. The kernel is unaware of where the result came from; it only
sees a bounded parameter update through the standard mechanism.
Data flow (ring buffer structs: PolicyUpdateRingHeader and PolicyUpdateMsg
in Section 23.1; observation rings:
ObservationRing in Section 23.1):
╔═══════════════════════════════════════════════════════════════════╗
║ UmkaOS Core (Tier 0, Ring 0) ║
║ ║
║ [observe_kernel! macro calls] ──► [ObservationRing per CPU] ║
║ ║
║ [KernelParamStore] ◄── [PolicyUpdateMsg ring] ◄── [validation] ║
╚══════════════╬══════════════════════════════╬═════════════════════╝
║ mmap ObservationRings ║ PolicyUpdateMsg
▼ (RW header page / RO slots) ║ (write ring, validated)
╔══════════════════════════════════╗ ║
║ Tier 2 Policy Service Process ║ ║
║ (Ring 3, hardware-isolated) ║ ║
║ ║ ║
║ ┌──────────────────────────┐ ║ ║
║ │ Observation aggregator │ ║ ║
║ │ (100ms/1s/10s windows) │ ║ ║
║ └──────────┬───────────────┘ ║ ║
║ │ feature vector ║ ║
║ ▼ ║ ║
║ ┌──────────────────────────┐ ║ ║
║ │ Inference layer │ ║ ║
║ │ (in-process small model │ ║ ║
║ │ OR call big model ──►──╫────╫─►external)║
║ │ ◄── result ────────────╫────╫─◄──────── ║
║ └──────────┬───────────────┘ ║ ║
║ │ PolicyUpdateMsg ║ ║
║ └────────────────────╫────────────╝
║ ║
╚══════════════════════════════════╝
External big model call — concrete example:
A Tier 2 scheduler policy service runs a 5-minute characterization cycle:
Every 5 minutes:
1. Drain last 5 minutes of Scheduler + Memory ObservationRings into feature matrix
(per-cgroup stats: avg task latency, cache miss rate, memory pressure, IPC, etc.)
2. If local small model is confident (prediction score > 0.85):
→ Apply parameter updates directly (Tier C path, ~100ms latency)
3. If confidence is low OR first boot OR significant workload shift detected:
→ Serialize feature matrix as JSON/protobuf
→ Call external inference service via UNIX socket or HTTP:
POST /analyze { features: [...], cgroup_ids: [...] }
→ Service runs large model (XGBoost, small transformer, RL policy)
(100ms – 5s latency acceptable at this tier)
→ Parse response: { cgroup_id: 42, param_id: "eevdf_weight_scale", value: 130 }
4. For each (param_id, new_value) in response:
→ Validate cgroup ownership (service can only tune cgroups it owns)
→ Submit PolicyUpdateMsg with valid_for_ms = 300_000 (expires in 5 min)
→ Kernel applies on next parameter read (atomic store, ~5 cycles)
5. Log all updates to FMA audit ring ([Section 20.1](20-observability.md#fault-management-architecture))
→ {timestamp, service_id, model_version, param_id, old_value, new_value}
The external service can be: - A local Python/Rust process running PyTorch, XGBoost, or scikit-learn - A remote inference microservice (gRPC or HTTP) running on another node - An LLM with a structured output schema (for workload characterization and root-cause analysis) - A reinforcement learning agent maintaining state across tuning cycles
None of this requires any kernel changes: the kernel only sees PolicyUpdateMsg entries.
23.1.8 Model Weight Update Flow¶
When a Tier 2 service trains or fine-tunes an in-kernel model (Section 22.6), it
ships updated weights via the existing sysfs interface (Section 22.6). The update
is atomic from the kernel's perspective:
Tier 2 online learning loop:
Every N minutes (configurable per service):
1. Extract recent training data from ObservationRings + ground-truth outcomes
(ground truth: observed page refault rates for prefetch model; actual I/O
latencies for I/O scheduler model)
2. Run mini-batch update in Tier 2 userspace (full FP, no kernel restrictions)
3. Quantize new weights to INT8/INT16 using the .umkaml binary format ([Section 22.6](22-accelerators.md#in-kernel-inference-engine--model-binary-format))
4. Validate model offline:
- Pass the model through the load-time validator ([Section 22.6](22-accelerators.md#in-kernel-inference-engine--model-binary-format))
- Run 1000 representative inputs; compare against previous model
- Accept only if accuracy delta > -2% (do not regress more than 2 points)
5. Write to /sys/kernel/umka/inference/models/<model_name>/model.bin
→ Kernel receives write, invokes load-time validator ([Section 22.6](22-accelerators.md#in-kernel-inference-engine--model-binary-format))
→ On validation pass: CAS swap of AtomicModelRef pointer
→ Old model freed after RCU grace period
→ New weights active for next inference call (within microseconds)
6. If validation fails: keep previous model, log failure to FMA ring
KernelModel — the canonical in-kernel inference model representation is defined
in Section 22.6 (KernelModel struct). The fields relevant to
the policy framework's quantized linear inference path are documented here for
context; the engine definition is authoritative.
// umka-nucleus/src/inference/model.rs — see [Section 22.6](22-accelerators.md#in-kernel-inference-engine) for
// the canonical KernelModel struct and KernelModelType enum.
//
// KernelModelType has four variants: DecisionTree, LookupTable, LinearModel,
// TinyNeuralNet. The full `run_inference()` dispatch and per-variant algorithms
// are defined in [Section 22.6](22-accelerators.md#in-kernel-inference-engine). This section documents only
// the quantized linear inference path used by the policy framework.
/// Maximum input feature dimension (enforced at model load time).
/// With output_dim ≤ 16, worst case is 64 * 16 = 1024 MACs = <1μs on modern CPUs.
pub const MODEL_MAX_INPUT_DIM: u16 = 64;
/// Maximum output dimension (enforced at model load time).
pub const MODEL_MAX_OUTPUT_DIM: u16 = 16;
// The following is the LinearModel-specific inference path (called by the
// canonical `KernelModel::run_inference()` when `model_type == LinearModel`).
// It is shown here for context because the ML policy framework primarily uses
// LinearModel for its low-overhead quantized inference.
impl KernelModel {
/// Quantized linear inference: dot product of features with weight matrix + bias.
///
/// **Called only when `self.model_type == KernelModelType::LinearModel`.**
/// For other model types, see [Section 22.6](22-accelerators.md#in-kernel-inference-engine) for `run_decision_tree()`,
/// `run_lookup_table()`, and `run_neural_net()`.
///
/// **Weight element width (normative)**: a LinearModel weight is ONE byte,
/// `i8`. This is the serialized `.umkaml` contract
/// ([Section 22.6](22-accelerators.md#in-kernel-inference-engine--model-binary-format)) as well as the
/// in-memory one — `weight_bytes = input_features * outputs` below is
/// 1 byte per weight, and the load-time validator sizes the weight region
/// by the same arithmetic. A model serialized at any other width reads as
/// garbage here, so the width is not a local implementation choice.
///
/// The width is DECLARED by the owning type, not here: see
/// `KernelModelType::LinearModel` ([Section 22.6](22-accelerators.md#in-kernel-inference-engine)),
/// which carries the normative parameter layout (i8 weights row-major,
/// then i32 biases). This function implements that layout; it does not
/// define it.
///
/// Weight layout in `self.params`:
/// - Bytes `[0 .. input_features * outputs)`: weight matrix W (row-major, i8 quantized).
/// Row `j` (output j) spans bytes `[j * input_features .. (j+1) * input_features)`.
/// - Bytes `[input_features * outputs .. input_features * outputs + outputs * 4)`:
/// bias vector B (each bias is i32, little-endian).
///
/// For each output j: `out[j] = sum(features[i] * W[j][i] for i in 0..input_features) + B[j]`
///
/// Returns output in the caller-provided `output` slice:
/// - If `outputs == 1`: a single scalar result.
/// - If `outputs > 1`: the full output vector. The caller determines interpretation
/// based on the policy context (argmax for classification, first value for regression).
///
/// # Performance bound
///
/// The inference loop is bounded by `input_features * outputs` multiply-accumulate
/// operations. With `MODEL_MAX_INPUT_DIM` (64) and `MODEL_MAX_OUTPUT_DIM` (16),
/// worst case is 1024 MACs = <1μs on modern CPUs. No heap allocation, no floating
/// point — integer-only quantized inference.
fn run_linear_model(&self, input: &[i32], output: &mut [i32]) {
let in_dim = self.input_features as usize;
let out_dim = self.outputs as usize;
let weight_bytes = in_dim * out_dim;
let bias_offset = weight_bytes;
for j in 0..out_dim {
let row_start = j * in_dim;
let mut acc: i32 = 0;
for i in 0..in_dim {
let w = self.params[row_start + i] as i8;
acc = acc.wrapping_add(input[i].wrapping_mul(w as i32));
}
let b_off = bias_offset + j * 4;
let bias = i32::from_le_bytes([
self.params[b_off],
self.params[b_off + 1],
self.params[b_off + 2],
self.params[b_off + 3],
]);
output[j] = acc.wrapping_add(bias);
}
}
}
pub enum ModelError {
/// Input feature vector dimension does not match model.input_features.
InvalidDimensions,
/// Model weights failed integrity check (CRC mismatch).
WeightCorruption,
/// Inference produced out-of-range output.
InferenceOverflow,
}
AtomicModelRef — the kernel's handle on the active in-kernel model:
// umka-nucleus/src/inference/model.rs (continued)
/// RCU-protected reference to the active in-kernel model.
/// Replaced atomically on weight update; old model freed after grace period.
pub struct AtomicModelRef {
pub ptr: AtomicPtr<KernelModel>, // Null = use heuristic fallback
}
impl AtomicModelRef {
/// Hot-path inference: load model pointer, run inference, drop RCU guard.
/// The rcu_read_guard prevents concurrent model replacement from freeing
/// the model while inference is in progress.
///
/// Calls `KernelModel::run_inference()` (see [Section 22.6](22-accelerators.md#in-kernel-inference-engine)),
/// which dispatches to the appropriate algorithm based on `model_type`:
/// `DecisionTree`, `LookupTable`, `LinearModel`, or `TinyNeuralNet`.
pub fn infer(&self, features: &[i32], output: &mut [i32]) -> Option<InferenceResult> {
let _guard = rcu_read_lock();
let model = unsafe { self.ptr.load(Acquire).as_ref()? };
if !model.active.load(Relaxed) { return None; }
let mut yielded = false;
Some(model.run_inference(features, output, &mut yielded))
}
/// Called from sysfs write handler on model.bin write.
/// Validates, then atomically replaces the active model.
///
/// **Serialization**: Callers must hold `MODEL_UPDATE_LOCK` (a global
/// `Mutex<()>` in the sysfs write handler) to prevent concurrent
/// `update()` calls from racing. Without the lock, two concurrent
/// writers could both validate, both `swap()`, and the second swap's
/// "old" pointer would be the first swap's "new" model — which gets
/// freed via `rcu_call_box_drop`, causing the winning model to be
/// freed while still active after the RCU grace period of the
/// losing update.
pub fn update(&self, new_model: Box<KernelModel>) -> Result<(), ModelError> {
// Caller must hold MODEL_UPDATE_LOCK.
new_model.validate()?;
let new_ptr = Box::into_raw(new_model);
let old_ptr = self.ptr.swap(new_ptr, AcqRel);
if !old_ptr.is_null() {
// Free old model after all CPUs pass through a quiescent state.
// Grace period bounds: see [Section 3.1](03-concurrency.md#rust-ownership-for-lock-free-paths).
// Typical grace period: <10ms under normal load; bounded by the
// longest RCU read-side critical section on any CPU.
// ([Section 3.1](03-concurrency.md#rust-ownership-for-lock-free-paths)).
// SAFETY: old_ptr was allocated via Box::into_raw above.
unsafe { rcu_call_box_drop(old_ptr) };
}
Ok(())
}
}
23.1.9 Security and Capability Model¶
CAP_ML_TUNE capability. Applying PolicyUpdateMsg entries to the kernel requires
the Capability::KernelMlTune capability. This is an UmkaOS-specific capability (not a
Linux CAP_*), held by the Tier 2 service process when:
1. The operator has granted it at service registration time, OR
2. The service is one of UmkaOS's reference policy services and is cryptographically signed
Grant mechanism: KernelMlTune is granted to a Tier 2 service via
/etc/umka/ml-policy.d/<service>.toml:
[service.sched-advisor]
binary = "/usr/lib/umka/ml/sched-advisor"
signing_key_id = "sha3-256:abcd..."
grant_capabilities = ["KernelMlTune"]
cgroup_scope = "/system.slice" # limits parameter visibility
signing_key_id and the key is in the .kabi keyring
(Section 12.7), the capability is minted and placed in the
service's capability space with a scope constraint limiting it to cgroup_scope.
Without a matching config entry, the capability is not granted and the service cannot
write parameters.
Without KernelMlTune, a process can read observations from its own cgroup but cannot
write parameter updates.
Bounds enforcement. Every new_value in PolicyUpdateMsg is silently clamped to
[min_value, max_value] before being stored. An ML service that produces out-of-range
values is not rejected — the clamping is the safety mechanism.
Namespace isolation. A containerized Tier 2 service sees only its own cgroup's
parameters and observations. A service in cgroup docker/myapp cannot read memory
pressure observations from system.slice, nor can it set rapl_package_power_w globally.
Global parameter updates require CAP_ML_TUNE + CAP_SYS_ADMIN.
Where these rules are enforced. The grant's cgroup_scope constraint and
the global-update requirement are checked by authorize_policy_update(), which
runs on every submitted PolicyUpdateMsg between structural validation and
dispatch (see the enforcement point above). The check is per-message, not
per-registration, so a grant revoked mid-session stops taking effect at the next
message. Neither validate_policy_update_msg() (structural only) nor the
[min, max] clamp in dispatch_to_subsystem() performs this check: the former
never inspects the registration, and the latter bounds the VALUE a service may
write, never the TARGET it may write to.
Audit log. Every parameter update is logged to the FMA ring (Section 20.1) with:
- {ts_ns, service_id, model_version, param_id, cgroup_id, old_value, new_value}
- Log entries are write-once; tampering with the audit ring requires CAP_SYS_ADMIN
- The FMA ring is accessible to security monitoring tools via umkafs /ukfs/kernel/mlaudit
Adversarial protection:
- Rate limiting: max 1000 PolicyUpdateMsg entries per service per second
- Consistency bounds: if a parameter oscillates by > 50% within 10s, an FMA alert
is raised (may indicate a misbehaving service or adversarial input to the ML model)
- Model versioning: models are refused if their validation accuracy is below 60% on
the standard benchmark set embedded in the kernel binary at build time.
Benchmark set definition (per model type):
| Model Type | Benchmark Inputs | Expected Output | Metric |
|---|---|---|---|
sched (task placement) |
1000 synthetic task arrival patterns (short-burst, sustained-CPU, IO-interleaved, mixed) with known-optimal CPU assignments from offline solver | Optimal CPU ID per task | Accuracy = correct placements / total |
memory (page tiering) |
100 memory access traces (sequential scan, random, hot-set, phase-change, zipfian) with known-optimal tier assignments | Tier ID per page | Accuracy = correct tier decisions / total |
tcp (congestion) |
200 flow traces (datacenter, WAN-lossy, satellite, wifi-variable) with known-optimal cwnd sequences from NS-3 simulation | cwnd within 20% of optimal | Accuracy = within-threshold predictions / total |
power (DVFS) |
50 CPU utilization traces with known Pareto-optimal frequency/power points | Frequency setting within 10% of optimal | Accuracy = within-threshold selections / total |
anomaly (FMA) |
100 metric streams (50 normal, 50 with injected anomalies: step change, drift, spike, flatline, periodic) | Correct anomaly/normal classification | F1 score ≥ 0.60 |
Benchmark inputs are compiled into the kernel as const byte arrays in
umka-nucleus/src/ml/benchmark/ (one module per model type). Total
embedded data: ~2 MiB compressed with LZ4 (block format, not frame format).
LZ4 is chosen for its decompression speed (~4 GB/s on modern CPUs) and
minimal code footprint (~500 lines, no heap allocation required for
decompression). The kernel already includes LZ4 for zram and squashfs support.
Decompression: Benchmark data is decompressed lazily on first model validation (not at boot time). Each model type's benchmark module contains:
/// Compressed benchmark inputs for the scheduler model.
/// LZ4 block format, decompressed size: SCHED_BENCH_DECOMPRESSED_SIZE.
static SCHED_BENCH_COMPRESSED: &[u8] = include_bytes!("sched_benchmark.lz4");
const SCHED_BENCH_DECOMPRESSED_SIZE: usize = 524_288; // 512 KiB
sched benchmark with 1000 task patterns). Total peak memory during
validation: ~512 KiB (only one model type is validated at a time).
The buffer is freed immediately after validation completes.
The validation function runs the candidate model against each benchmark
input and computes the accuracy metric. Models failing the 60% threshold
are rejected with PolicyError::ValidationFailed.
23.1.9.1 Rate Limiter Implementation¶
/// Errors returned by the ML policy subsystem to Tier 2 services.
pub enum PolicyError {
/// Rate limiter rejected the message (too many submissions per second).
RateLimitExceeded,
/// Caller lacks `Capability::KernelMlTune`.
PermissionDenied,
/// Parameter ID not found in `KernelParamStore`.
UnknownParam { param_id: u32 },
// Values outside [min, max] are silently clamped (Section 23.1.8). No error is returned.
/// Service registration failed (duplicate service ID or table full).
RegistrationFailed,
/// Model validation failed (accuracy below 60% threshold or weight corruption).
ValidationFailed,
/// HMAC verification failed on a PolicyUpdateMsg.
HmacInvalid,
}
/// Registration record for a Tier 2 policy service.
/// Created during `ML_POLICY_REGISTER` ioctl. Holds the rate limiter
/// and a reference to the service's consumer vtable.
// Kernel-internal registration record. Not #[repr(C)], not KABI --
// layout is compiler-determined.
pub struct PolicyServiceRegistration {
/// Unique service identifier assigned at registration from a monotonic
/// `AtomicU64` counter; never reused, never recycled.
///
/// u64 because this is a kernel-internal identifier constrained by no
/// external protocol, and it is an identity, not a refcount: it keys the
/// FMA audit trail (`{ts_ns, service_id, ...}`) and each ring's
/// `ObservationCursor::owner`. A u32 would alias after 4.29 billion
/// registrations, which a crash-restart loop reaches inside a 50-year boot,
/// and an aliased id makes two different services indistinguishable in the
/// audit record — the one place where indistinguishable is unacceptable.
/// Allocation starts at 2: 0 and 1 are `CURSOR_SLOT_FREE` and
/// `CURSOR_SLOT_AGGREGATOR`.
pub service_id: u64,
/// Token bucket rate limiter (one per service).
pub rate_limiter: PolicyServiceRateLimiter,
/// Handle for the registering process's `Capability::KernelMlTune`,
/// recorded at `ML_POLICY_REGISTER`.
///
/// A HANDLE, not a validated token: `ValidatedCap<'dispatch>` is RCU-scoped
/// to one dispatch and cannot be stored across messages
/// ([Section 9.1](09-security.md#capability-based-foundation--capability-validation-amortization-validatedcapguard)),
/// and caching a validation result across the life of a registration is
/// exactly what would let a revoked grant keep writing parameters. The
/// submit path re-validates this handle per message. That is affordable
/// precisely because the path is warm — the rate limiter caps it at
/// `POLICY_MSG_RATE_LIMIT_PER_SEC` — so the ~10-20 cycle validation is
/// invisible, and no amortization is worth a stale authority window here.
pub ml_tune_handle: CapHandle,
/// Root of the cgroup subtree this registration may write, taken from the
/// grant's `cgroup_scope` constraint. Every non-global update must target
/// this cgroup or one of its descendants.
pub cgroup_scope: CgroupId,
/// Whether this registration may write GLOBAL parameters
/// (`PolicyUpdateMsg::cgroup_id == 0`). Set at registration only when the
/// caller held `CAP_ML_TUNE` **and** `CAP_SYS_ADMIN`, per the namespace
/// isolation rule below. Kernel-internal struct, not KABI — `bool` is fine.
pub global_authority: bool,
/// File descriptors for the service's shared-memory ring protocol.
pub obs_ring_fd: i32,
pub update_ring_fd: i32,
pub param_store_fd: i32,
/// Kernel's copy of the update ring's capacity in entries. TRUSTED — the
/// bound every drain uses. The `capacity` field inside the mmap'd
/// `PolicyUpdateRingHeader` is service-writable and informational only.
pub update_ring_capacity: u32,
/// Kernel-owned read index into the update ring. Never lives in the shared
/// mapping, so a service cannot rewind or advance the kernel's position.
pub update_ring_read_idx: AtomicU64,
/// Count of drains that observed an impossible `write_idx - read_idx`
/// distance and resynchronized. Non-zero means the service crashed
/// mid-write or is hostile; drives the `PolicyConsumerMisbehaving` alert.
pub ring_desync_count: AtomicU64,
/// PID of the registering process (for audit logging).
pub owner_pid: u32,
/// Per-registration HMAC session key (HMAC-SHA3-256). Generated by the
/// kernel during `ML_POLICY_REGISTER` and returned to the service in
/// `PolicyRegisterResponse.session_key`. The kernel retains this copy
/// to verify the HMAC on every incoming `PolicyUpdateMsg`.
pub session_key: [u8; 32],
}
Each registered Tier 2 service has a PolicyServiceRateLimiter embedded in its
PolicyServiceRegistration record. Enforcement happens before any validation or
dispatch of the PolicyUpdateMsg:
/// Token bucket rate limiter for PolicyUpdateMsg.
/// One per registered Tier 2 service. Cache-line aligned to prevent
/// false sharing when multiple services submit concurrently.
///
/// Uses a fractional nanosecond accumulator to avoid integer division
/// precision loss. The accumulator tracks elapsed nanoseconds that have
/// not yet been converted into tokens. When enough nanoseconds accumulate
/// to produce at least one whole token, they are converted and the
/// remainder is preserved. This gives exact long-term rates for any
/// tokens/sec value without requiring floating point or large intermediate
/// products.
#[repr(C, align(64))]
pub struct PolicyServiceRateLimiter {
/// Current token count (whole tokens, no scaling factor).
pub tokens: AtomicU32,
/// Explicit padding for AtomicU64 alignment (offset 4 -> 8).
pub _pad0: u32,
/// Nanosecond timestamp of last refill attempt.
pub last_refill_ns: AtomicU64,
/// Fractional nanosecond remainder from the last refill that was not
/// enough to produce a whole token. Accumulates across refill calls
/// so that sub-token intervals are never lost.
pub remainder_ns: AtomicU64,
/// Maximum token count (= burst capacity).
pub max_tokens: u32,
/// Explicit padding for u64 alignment (offset 28 -> 32).
pub _pad1: u32,
/// Nanoseconds required to produce one token (= 1_000_000_000 / rate).
/// Stored as the divisor to avoid per-refill division: we divide
/// elapsed_ns by this constant.
pub ns_per_token: u64,
// Total: 40 bytes of fields + 24 bytes align(64) tail padding = 64 bytes.
}
const_assert!(core::mem::size_of::<PolicyServiceRateLimiter>() == 64);
pub const POLICY_MSG_RATE_LIMIT_PER_SEC: u64 = 1_000; // steady-state limit
pub const POLICY_MSG_BURST: u32 = 200; // burst capacity
impl PolicyServiceRateLimiter {
pub const fn new() -> Self {
Self {
tokens: AtomicU32::new(POLICY_MSG_BURST),
_pad0: 0,
last_refill_ns: AtomicU64::new(0),
remainder_ns: AtomicU64::new(0),
max_tokens: POLICY_MSG_BURST,
_pad1: 0,
// 1_000_000_000 ns/sec / 1_000 tokens/sec = 1_000_000 ns per token.
// For any rate R tokens/sec, ns_per_token = 1_000_000_000 / R.
// This is exact for all rates that evenly divide 10^9 and provides
// < 1 ns/token error otherwise (absorbed by the remainder accumulator).
ns_per_token: 1_000_000_000 / POLICY_MSG_RATE_LIMIT_PER_SEC,
}
}
/// Attempt to consume one token. Lock-free.
///
/// Uses Relaxed ordering throughout: the token bucket provides approximate
/// rate limiting. Uses CAS loops for both refill and consume paths
/// to prevent wrapping underflow (which would completely bypass the
/// rate limiter). No inter-CPU visibility guarantees beyond the CAS.
pub fn try_consume(&self) -> Result<(), PolicyError> {
let now_ns = ktime_get_ns(); // CLOCK_MONOTONIC ([Section 7.8](07-scheduling.md#timekeeping-and-clock-management))
// Refill: compute elapsed time, add remainder, convert to whole tokens.
// remainder_ns is best-effort; concurrent refills may grant OR lose
// O(1) tokens due to non-atomic read-modify-write on the remainder.
// This is acceptable: the rate limiter is advisory (prevents ML service
// flooding, not a security boundary). Token loss is bounded at 1 token
// per concurrent refill race.
let last = self.last_refill_ns.load(Relaxed);
let elapsed_ns = now_ns.saturating_sub(last);
let prev_remainder = self.remainder_ns.load(Relaxed);
let total_ns = elapsed_ns.saturating_add(prev_remainder);
let new_tokens = total_ns / self.ns_per_token;
if new_tokens > 0 {
let leftover_ns = total_ns % self.ns_per_token;
// The CAS on last_refill_ns serializes refill intervals.
// A concurrent caller with a slightly later timestamp may win
// a subsequent CAS, producing a second refill for the delta
// interval. This adds at most 1 extra token (delta <
// ns_per_token), bounded by max_tokens.
if self.last_refill_ns.compare_exchange(
last, now_ns, Relaxed, Relaxed
).is_ok() {
self.remainder_ns.store(leftover_ns, Relaxed);
// CAS loop for refill: cap at max_tokens, no lost tokens.
loop {
let cur = self.tokens.load(Relaxed);
let new = (cur + new_tokens as u32).min(self.max_tokens);
match self.tokens.compare_exchange_weak(cur, new, Release, Relaxed) {
Ok(_) => break,
Err(_) => continue,
}
}
}
}
// Consume one token via CAS loop — prevents wrapping underflow.
loop {
let cur = self.tokens.load(Acquire);
if cur == 0 {
return Err(PolicyError::RateLimitExceeded);
}
match self.tokens.compare_exchange_weak(cur, cur - 1, AcqRel, Relaxed) {
Ok(_) => return Ok(()),
Err(_) => continue, // Another thread consumed; retry
}
}
}
}
Enforcement point — the first check in policy_consumer_kabi_submit():
pub fn policy_consumer_kabi_submit(
svc: &PolicyServiceRegistration,
msg: PolicyUpdateMsg,
) -> Result<(), PolicyError> {
svc.rate_limiter.try_consume()?; // Reject immediately if over limit
validate_policy_update_msg(&msg)?; // Then check the message is well-formed
authorize_policy_update(svc, &msg)?; // Then check the SERVICE may write it
dispatch_to_subsystem(svc, msg) // Then dispatch
}
authorize_policy_update definition:
/// Authorization gate for one policy update. This is the enforcement point for
/// the capability and scope rules stated in the Security and Capability Model
/// below; without it a registration granted `cgroup_scope = "/system.slice"`
/// could submit perfectly well-formed, in-bounds updates against any other
/// cgroup, or globally — the structural validator does not look at `svc` at
/// all, and clamping to `[min, max]` bounds the VALUE, never the TARGET.
///
/// Two checks, in order:
///
/// 1. **The grant is still live and still carries write rights.** The stored
/// handle is re-validated through `cap_validate`, which performs the
/// generation and permission checks against the live capability table — so a
/// grant revoked, bulk-revoked, or invalidated by a credential change while
/// the service runs stops taking effect at the NEXT message, not at the next
/// registration. A registration is long-lived by construction, which is
/// exactly why a cached validation result would be the wrong thing to hold.
///
/// 2. **Scope.** `cgroup_id == 0` means a global parameter write and requires
/// `global_authority`, which registration sets only for a caller holding
/// both `CAP_ML_TUNE` and `CAP_SYS_ADMIN`. Any other target must lie inside
/// the registration's `cgroup_scope` subtree; ancestry is tested by walking
/// the target's parent chain, which is bounded by `CGROUP_MAX_DEPTH`
/// ([Section 17.2](17-containers.md#control-groups)). The walk runs under the dispatch guard's RCU read
/// section, so an ancestor cannot be freed mid-walk.
fn authorize_policy_update(
svc: &PolicyServiceRegistration,
msg: &PolicyUpdateMsg,
) -> Result<(), PolicyError> {
let dispatch = KabiDispatchGuard::enter();
let _vcap = cap_validate(svc.ml_tune_handle, PermissionBits::WRITE, &dispatch)
.map_err(|_| PolicyError::PermissionDenied)?;
if msg.cgroup_id == 0 {
if !svc.global_authority {
return Err(PolicyError::PermissionDenied);
}
return Ok(());
}
let mut cur = CGROUP_REGISTRY.get(msg.cgroup_id)
.ok_or(PolicyError::PermissionDenied)?;
for _ in 0..=CGROUP_MAX_DEPTH {
if cur.id == svc.cgroup_scope {
return Ok(());
}
match cur.parent.as_ref().and_then(|w| w.upgrade()) {
Some(p) => cur = p,
None => break, // reached the root, or an ancestor went away
}
}
Err(PolicyError::PermissionDenied)
}
validate_policy_update_msg definition:
/// Structurally validate a policy update message before dispatch.
///
/// Performs only the checks that are independent of the per-service session key.
/// HMAC authentication is verified separately by the update-ring drain / KABI
/// ingress before this point — this function is NOT the authentication boundary.
/// Rejects a message whose:
/// - `msg_type` is not `MSG_TYPE_POLICY_UPDATE` (e.g. a throttle message
/// misrouted onto the update path), or
/// - `param_id` does not decode to a known `ParamId` / `SubsystemId`.
///
/// Returns `Ok(())` if well-formed, else the corresponding `PolicyError`.
/// Value-range enforcement is intentionally NOT done here: out-of-range values
/// are silently CLAMPED at dispatch (see `dispatch_to_subsystem`), never
/// rejected.
fn validate_policy_update_msg(msg: &PolicyUpdateMsg) -> Result<(), PolicyError> {
if msg.msg_type != MSG_TYPE_POLICY_UPDATE {
return Err(PolicyError::ValidationFailed);
}
let param_id = ParamId::try_from_u32(msg.param_id)
.ok_or(PolicyError::UnknownParam { param_id: msg.param_id })?;
SubsystemId::from_param_id(param_id)
.ok_or(PolicyError::UnknownParam { param_id: msg.param_id })?;
Ok(())
}
dispatch_to_subsystem definition:
/// Route an authorized policy update to the target subsystem's parameter store.
/// The subsystem is identified by the `ParamId` encoding in `msg.param_id`
/// (bits [11:8] = subsystem index). Each subsystem registers a static
/// `PolicyApplyFn` at boot; this function looks it up and calls it.
///
/// **Precondition**: the caller has already run `authorize_policy_update` — this
/// function performs no capability or scope check and must never be reached
/// without one.
///
/// Returns `Err(PolicyError::UnknownParam)` if the subsystem index
/// is out of range or has no registered handler.
fn dispatch_to_subsystem(
svc: &PolicyServiceRegistration,
msg: PolicyUpdateMsg,
) -> Result<(), PolicyError> {
// Convert raw u32 wire value to typed ParamId enum.
let param_id = ParamId::try_from_u32(msg.param_id)
.ok_or(PolicyError::UnknownParam { param_id: msg.param_id })?;
let subsystem = SubsystemId::from_param_id(param_id)
.ok_or(PolicyError::UnknownParam { param_id: msg.param_id })?;
let idx = subsystem as usize;
// POLICY_HANDLERS is a static array of Option<PolicyApplyFn>, one per
// SubsystemId, populated at boot by each subsystem's init code.
// Read via Acquire to see the handler written by the registering CPU.
let handler = POLICY_HANDLERS[idx]
.load(Ordering::Acquire)
.ok_or(PolicyError::UnknownParam { param_id: msg.param_id })?;
// Clamp the value to the parameter's registered [min, max] bounds.
let bounds = KERNEL_PARAM_STORE.bounds(param_id);
let clamped_value = msg.new_value.clamp(bounds.min, bounds.max);
// Resolve `valid_for_ms` into the ABSOLUTE revert deadline the handler
// needs. A handler cannot compute this itself: `valid_for_ms` is relative
// to message arrival, and 0 means "use the parameter's registered
// decay_period_ms", which only the store knows. `decay_period_ms == 0` (and
// hence a zero window) means the parameter never auto-reverts, which
// `set_with_expiry` spells `u64::MAX`.
let now_ns = ktime_get_ns(); // CLOCK_MONOTONIC ([Section 7.8](07-scheduling.md#timekeeping-and-clock-management))
let window_ms = if msg.valid_for_ms != 0 {
msg.valid_for_ms as u64
} else {
KERNEL_PARAM_STORE.get(param_id).map_or(0, |p| p.decay_period_ms as u64)
};
let expiry_ns = if window_ms == 0 {
u64::MAX
} else {
now_ns.saturating_add(window_ms.saturating_mul(1_000_000))
};
// Model-sequence admission for GLOBAL writes. The per-cgroup case is
// admitted inside the cgroup write path instead, under `config_lock`,
// because there the watermark lives in the override entry and the decision
// must be atomic with the entry's publication.
if msg.cgroup_id == 0 {
if let Some(p) = KERNEL_PARAM_STORE.get(param_id) {
if !p.admit_model_seq(msg.model_seq) {
return Ok(()); // stale or duplicate — dropped silently
}
}
}
// Audit BEFORE applying, so the record exists even if the handler fails:
// {ts_ns, service_id, model_version, param_id, cgroup_id, old_value,
// new_value} — the audit trail keyed by `svc.service_id` described in the
// Security and Capability Model below.
ml_audit_param_update(svc, param_id, clamped_value, msg.cgroup_id, msg.model_seq);
// Apply the clamped value. The handler writes to the subsystem's
// per-cgroup or global parameter store (AtomicU32/AtomicI32).
handler(&PolicyApplyCtx {
param: param_id,
value: clamped_value,
cgroup_id: msg.cgroup_id,
expiry_ns,
model_seq: msg.model_seq,
})
}
/// Append one parameter-update record to the FMA audit ring
/// ([Section 20.1](20-observability.md#fault-management-architecture)). Write-once; the ring is readable
/// through umkafs at `/ukfs/kernel/mlaudit`.
fn ml_audit_param_update(
svc: &PolicyServiceRegistration,
param: ParamId,
new_value: i64,
cgroup_id: u64,
model_seq: u64,
);
/// Everything a subsystem handler needs to apply one policy update.
/// Kernel-internal; passed by reference on the warm path.
///
/// It carries `expiry_ns` and `model_seq` because dropping them at dispatch
/// left the two downstream contracts unimplementable: a handler that reaches
/// `param.set_with_expiry(stored, expiry_ns)` — the shape
/// [Section 7.10](07-scheduling.md#intent-based-resource-management)'s `apply_policy_update` requires —
/// has no way to invent an expiry, and the per-cgroup half of the
/// model-sequence monotonicity rule has nowhere to compare against.
pub struct PolicyApplyCtx {
/// Target parameter.
pub param: ParamId,
/// Value, ALREADY clamped to the parameter's registered `[min, max]`.
pub value: i64,
/// Target cgroup; 0 = global / kernel-wide.
pub cgroup_id: u64,
/// Absolute monotonic deadline at which the value reverts to its default.
/// `u64::MAX` = permanent (the parameter has no decay window).
pub expiry_ns: u64,
/// Producer sequence of the originating message. Already admitted for
/// global writes; the per-cgroup write path admits it under `config_lock`.
pub model_seq: u64,
}
/// Function pointer type for subsystem parameter application.
/// Called on the warm path (per policy update, not per syscall).
type PolicyApplyFn = fn(ctx: &PolicyApplyCtx) -> Result<(), PolicyError>;
/// Per-subsystem handler table. Indexed by `SubsystemId as usize`.
/// Each entry is set once at boot by the subsystem's init code.
///
/// **Implementation note**: `AtomicOption<fn(...)>` is implemented as
/// `AtomicPtr<()>` with null representing `None`. Function pointers
/// in Rust are pointer-sized thin pointers, so `AtomicPtr` can hold
/// them. The `load()` method returns `Option<fn(...)>` by converting
/// null → None, non-null → Some(transmute). This is sound because
/// all non-null values were stored from valid function pointers.
static POLICY_HANDLERS: [AtomicOption<PolicyApplyFn>; MAX_SUBSYSTEM_ID] =
[const { AtomicOption::none() }; MAX_SUBSYSTEM_ID];
/// Maximum SubsystemId discriminant + 1.
const MAX_SUBSYSTEM_ID: usize = 16;
Rejection behavior: The Tier 2 service receives PolicyError::RateLimitExceeded
(mapped to -EBUSY in the SysAPI layer). The service is responsible for backing off
(exponential backoff recommended). Repeated violations — more than 3 rate-limit errors
per second — trigger an FMA alert (category: PolicyConsumerMisbehaving, service_id).
23.1.10 Reference Policy Services¶
UmkaOS ships the following Tier 2 policy services (all optional, loaded on demand):
| Service | Model | Parameters tuned | Observations consumed | Cadence |
|---|---|---|---|---|
umka-ml-sched |
Gradient-boosted trees (XGBoost) | eevdf_weight_scale, migration_benefit_threshold, eas_energy_bias, resched_urgency |
TaskWoke, RunqueueStats, EasDecision |
Every 60s |
umka-ml-numa |
Gradient-boosted regression | numa_migration_threshold, swap_local_ratio |
NumaMigration, RefaultRecord, MemPressure |
Every 30s |
umka-ml-compress |
Random forest | compress_entropy_threshold, reclaim_aggressiveness |
PageFault, EvictionDecision, MemPressure |
Every 30s |
umka-ml-power |
Contextual bandit (online RL) | rapl_package_power_w, accel_power_cap_w, thermal_target_c |
EAS observations, accel utilization | Every 10s |
umka-ml-anomaly |
Isolation forest | anomaly_alert_threshold, health_check_interval_ms |
FmaHealth device metrics |
Every 5s |
umka-ml-tcp |
Online linear model | tcp_initial_cwnd_scale, bbr_probe_rtt_interval_ms |
CongestionEvent, FlowStats |
Every 5s |
umka-ml-intent |
RL / gradient-boosted | SchedIntentKpLatency, SchedIntentKdLatency, SchedIntentMaxAdjustment, SchedIntentHoldTimeMs |
RunqueueStats, IntentStatus, LatencyHistogram |
Every 60s |
Each service is a standalone Tier 2 driver (~500-2000 LOC) implementing the
policy ring protocol (PolicyRegisterRequest/PolicyRegisterResponse). They are independently upgradable, independently crashable
(Tier 2 restart in ~10ms, Section 11.2), and independently optional.
The umka-ml-sched service supports a "big model" extension point: if the environment
variable UMKA_ML_SCHED_REMOTE_ENDPOINT is set, it will call an external inference
service (Section 23.1.6) for low-confidence workload characterizations, falling back
to its local XGBoost model when the remote endpoint is unavailable.
23.1.11 Performance Impact¶
Observation emission (when at least one consumer is registered):
| Operation | Cycles | Notes |
|---|---|---|
| Static key check (disabled) | 1–3 | Predicted-not-taken branch |
| TSC read | 5–10 | RDTSC or arch-specific |
| Ring buffer write | 10–20 | One cache line write |
| Head publish + mirror | 2–4 | Private counter, then the subscriber-visible copy |
| Overflow probe | 1–3 | ONE load of the cached slowest_tail — constant in subscriber count |
| Total (enabled) | ~30 cycles | ~12ns at 2.5 GHz |
Parameter read (hot path):
| Operation | Cycles | Notes |
|---|---|---|
AtomicI64::load(Relaxed) |
1–3 | Same cost as reading a global variable |
Parameter update (Tier 2 → kernel):
| Operation | Cycles | Notes |
|---|---|---|
| Ring buffer submission | ~20 | From Tier 2 userspace |
| Kernel validation + CAS | ~15 | Including capability check |
| Total end-to-end (async) | ~100μs | Dominated by scheduling latency for async path |
Decay enforcement (once per second, not per tick):
The enforce_param_decay scan visits all registered parameters once per second (guarded
by the decay_due flag set by a 1 Hz periodic timer). With 50 parameters: ~50 comparisons
per second per CPU. At 3 cycles per comparison: ~150 cycles/s = ~60ns/s. Negligible.