Can an LLM Forget the Right Things?

Key takeaways

• : point a normal chat-oriented LLM runtime at a live robot camera and three things break at once: (i) VRAM overflows from a vision stream that never ends, (ii) control-loop deadlines get missed silently, and (iii) a 60Hz camera easily outpaces a much slower reasoning step.
• The fix: an admission controller that estimates a reasoning chunk’s cost with an online exponential moving average and refuses to even start it rather than gamble on the 33ms deadline; a KV cache that evicts the most redundant frame, by cosine similarity, instead of automatically selecting the oldest one; a lock-free double buffer so perception never blocks on a stale backlog.
• The engine room: the entire transformer for Qwen2.5-Coder-1.5B-Instruct: RMSNorm, RoPE, grouped-query attention, SwiGLU — is hand-written in CUDA, no cuBLAS, no libtorch; validated against a real HuggingFace forward pass at ≥0.999 cosine similarity.
• The honest part: this post is an architecture, not a benchmark. Built on a single cloud NVIDIA Hopper GPU (sm_90), the 8GB VRAM ceiling is a modeled budget in the code, not a number measured on a Jetson or any other physical edge board.

TL;DR: Point a normal LLM server at a live robot camera and it will happily flood the VRAM, blow through the control-loop deadlines, and choke on a camera that’s faster than its own brain. vla-edge-backend is a hand-written CUDA runtime that refuses reasoning it can’t finish in 33ms, evicts KV cache by ‘meaning’ instead of just age, and never lets perception block on the reasoning. It’s an architecture, not a benchmark; built on a cloud Hopper GPU, with an 8GB ceiling that’s a design constraint, not a field measurement.

Architecture mental model: 60Hz camera → lock-free double buffer → admission controller (admit / refuse) → [vision encoder → KV manager (semantic eviction) → hand-written CUDA transformer] → action or fallback. Everything below is commentary on one piece of that line.

1. The clock that never stops

Somewhere inside this project, a camera produces a fresh frame every 16.7 milliseconds. It doesn’t care whether anything downstream is ready. It doesn’t pause, it doesn’t retry, and it definitely doesn’t file a support ticket. It just keeps going, 60 times a second, forever, because that’s what a camera bolted to a robot does.

Meanwhile, a small language model wired to a vision encoder has a much harder job: grab the freshest frame it can get, decide what to do about it, and finish that decision inside a 33-millisecond window. Not “usually”, not “on a good day”. Thirty-three milliseconds, for reference, is faster than just the closing half of a single human blink, eyelids take about 50 to 100 milliseconds to snap shut. The robot doesn’t get to blink and think about it.

That number isn’t a vibe. It’s a compile-time constant sitting at the top of the codebase, in include/physical_ai/common.hpp:

constexpr double DEADLINE_MS = 33.0;
constexpr double SAFETY_MARGIN_MS = 2.0;

Every other piece of this runtime exists to either hit that number or fail loudly and on purpose instead of quietly missing it — which, if you’ve ever worked with a deadline-blind system, is already a personality upgrade.

An upfront confession before we go further, because the whole point of starting a new series is refusing to launder one kind of truth as another: this runtime was built and tested on a single NVIDIA Hopper GPU (sm_90) in a cloud dev box. There is no Jetson in this story. The “8GB VRAM ceiling” you’re about to read a lot about is a design constraint baked into the code, a budget the KV manager is built to respect, not a number measured on a robot’s onboard compute. If you came for a benchmark table off a robot arm in a lab, it isn’t here; that’s future work, and I’d rather say so now than bury it at the bottom.

What follows is the architecture of a system that treats a hard deadline and a fixed memory budget as top-priority constraints, instead of doing what most LLM stacks do: pretend neither exists until something falls over, usually expensively.

2. Why your favorite LLM runtime breaks the moment you bolt it to a robot

Take any excellent serving stack the LLM world already has, and point one at a live camera feed instead of a chat window. Three things will go wrong almost at the same time.

VRAM explodes, because chat-oriented runtimes assume a conversation eventually ends. A robot’s camera never sends a “goodbye” token. It produces new visual tokens every frame, forever, and a context window that keeps growing to hold all of them eventually runs out of VRAM on an edge GPU, uncomfortably soon, since edge GPUs don’t have a data center’s spare closet.

Deadlines get missed silently, because a chat completion that takes four seconds instead of two is mildly annoying, while a robot control step that takes 45ms instead of 33ms is a physical action that happened after the moment it was supposed to. Standard inference runtimes have no notion of a deadline at all. They compute as long as computing takes, then hand back an answer whenever it feels like it, like a contractor who “will be there sometime Tuesday.”

The frequencies don’t match, because a camera can push 60Hz and a vision-language-action model doing real reasoning is not going to keep up with that pace. Something has to sit between the two, or perception either blocks the camera (now it’s lying about when things happened) or piles up a backlog of stale frames while reasoning falls further behind reality.

Put it in three lines and it reads like a status report nobody wants to send:

• VRAM: filling up like a group chat nobody has the heart to leave.
• The deadline: missed, quietly, with no one in the runtime even filing the incident.
• The camera and the model: running two different races, at two different speeds, and only one of them has noticed.

None of this is a knock on vLLM, TensorRT-LLM, or llama.cpp’s own server, they’re excellent at what they were built for: serving as many chat-shaped requests as a data center can throughput. A robot’s camera is not a chat-shaped request; it’s a continuous, unstoppable stream with a physical clock duct-taped to it, and that shape of workload asks a different question, not “how many tokens per second,” but “which 33ms window am I inside right now, and can I finish before it closes.”

vla-edge-backend treats those three failure modes as design constraints from line one, instead of retrofitting them onto a chat-shaped runtime later.

3. A camera that never asks permission

The perception side of this runtime has exactly one job: keep publishing the freshest frame it has, at 60Hz, and never, under any circumstances, block waiting for the reasoning side to catch up. Reasoning is allowed to be a little slow. The camera is not allowed to care.

The mechanism is a lock-free single-producer/single-consumer double buffer: two frame slots and one atomic version counter that says which one is currently valid. Think of it as a reverse “musical chairs” game with exactly two chairs and exactly one player at a time: there is always a free chair, so nobody ever has to stand around waiting for the music to stop.

struct PerceptionPipeline {
  PerceptionFrame frame_buffers[2];
  std::atomic<:uint64_t> published_version;
  std::thread producer_thread;
  std::atomic producer_running;
  std::vector<:uint8_t> clip_frame_storage;
  std::uint32_t clip_num_frames;
  std::uint32_t clip_read_cursor;
  double producer_start_time_sec;
  PerceptionFrame last_consumed_frame;
  bool has_consumed_frame;
};

A producer thread writes into whichever buffer isn’t currently being read, then flips the counter:

const int write_buffer_index = static_cast(local_version & 1U);
PerceptionFrame* write_frame = &pipeline->frame_buffers[write_buffer_index];
std::memcpy(write_frame->pixels, &pipeline->clip_frame_storage[frame_offset], frame_byte_count);
write_frame->frame_index = frame_index;
const double elapsed_sec =
    physical_ai_steady_time_sec() - pipeline->producer_start_time_sec;
write_frame->capture_timestamp_sec = elapsed_sec;
++local_version;
pipeline->published_version.store(local_version, std::memory_order_release);

The consumer never blocks. It checks whether the counter moved since it last looked, and helps itself to whatever is freshest:

bool perception_pipeline_consume_latest(PerceptionPipeline* pipeline,
                                        PerceptionFrame* out_frame,
                                        std::uint64_t* last_seen_version) {
  const std::uint64_t current_version =
      pipeline->published_version.load(std::memory_order_acquire);
  if (current_version == 0U) {
    return false;
  }
  if (current_version != *last_seen_version) {
    const int read_buffer_index = static_cast((current_version - 1U) & 1U);
    pipeline->last_consumed_frame = pipeline->frame_buffers[read_buffer_index];
    pipeline->has_consumed_frame = true;
    *last_seen_version = current_version;
  }
  if (!pipeline->has_consumed_frame) {
    return false;
  }
  *out_frame = pipeline->last_consumed_frame;
  return true;
}

No version change, no new copy, just the reader just reuses what it already has instead of waiting around like it’s got nothing better to do. The acquire/release pair on the atomic is the entire safety argument: no mutex, no blocking, no “please hold” music.

Getting an actual 60Hz out of this — instead of “60Hz, roughly, on a good day” — needs a hybrid sleep/spin loop, because sleep_for on most of the operating systems is honest to within about a millisecond, which is a fine margin of error when you have tens of milliseconds to spare and an actively embarrassing one when your entire cycle budget is tens of milliseconds:

void hybrid_sleep_until(const std::chrono::steady_clock::time_point& target_time) {
  while (true) {
    const auto now_time = std::chrono::steady_clock::now();
    if (now_time >= target_time) {
      return;
    }
    const auto remaining = target_time - now_time;
    const auto remaining_ms =
        std::chrono::duration_cast<:chrono::milliseconds>(remaining).count();
    if (remaining_ms > 1) {
      std::this_thread::sleep_for(std::chrono::milliseconds(remaining_ms - 1));
    }
  }
}

Sleep through everything except the last millisecond, then busy-spin the rest of the way. Trading a sliver of CPU for actually landing on time instead of landing “on time, ish.” The exact same trick shows up again, almost word for word, inside the admission controller’s cycle loop. Same problem, same fix, no shame in reusing a good idea twice.

4. Teaching the brain to say “no”

This is the part of the runtime that commits to a genuinely unusual decision: when the runtime isn’t confident it can finish in time, it does not just give its best shot and hope for the best. It refuses to start. No optimism. No “we’ll make it up on decode.” Just a flat no, delivered before any GPU cycle is wasted finding out the hard way. It is, as far as I can tell, the one part of this codebase with cleaner boundaries than most humans manage on a Monday morning.

Here’s the actual decision, short enough to read in one breath:

bool admission_controller_should_admit(const AdmissionEmaState* ema_state,
                                       double elapsed_since_cycle_start_ms) {
  if (!ema_state->has_bootstrap_sample) {
    return true;
  }
  const double remaining_budget_ms =
      DEADLINE_MS - elapsed_since_cycle_start_ms - SAFETY_MARGIN_MS;
  const double estimated_chunk_ms =
      static_cast(ema_state->ema_prefill_per_token_ms) *
          static_cast(VISION_TOKENS_PER_FRAME) +
      static_cast(ema_state->ema_decode_per_token_ms) *
          static_cast(ACTION_DECODE_TOKENS);
  return estimated_chunk_ms <= remaining_budget_ms;
}

To put it in plain language: take what’s left of the 33ms budget, subtract a 2ms safety margin, and compare that against how expensive the next reasoning chunk (32 tokens of prefill, 8 tokens of decode) is estimated to be, based on recent history. Doesn’t fit? Don’t start. Fall back to repeating the last action instead of gambling on a miss.

One wiring detail worth being precise about: in the current main loop, this check runs right at the top of the cycle, before any GPU work has started and the call site always passes elapsed_since_cycle_start_ms = 0.0. So today the comparison is really “does my estimated cost fit inside DEADLINE_MS − SAFETY_MARGIN_MS,” checked once per cycle, not continuously re-checked mid-chunk. And it’s a heuristic guard on an estimate, not a hard guarantee as a chunk that looked safe can still run long, and the runtime just logs that as a deadline miss rather than pretending it’s impossible. Confidence, not clairvoyance.

The estimate comes from an exponential moving average — boring, ancient, and extremely effective:

ema_state->ema_prefill_per_token_ms =
    (1.0f - ADMISSION_EMA_ALPHA) * ema_state->ema_prefill_per_token_ms +
    ADMISSION_EMA_ALPHA * measured_prefill_per_token_ms;
ema_state->ema_decode_per_token_ms =
    (1.0f - ADMISSION_EMA_ALPHA) * ema_state->ema_decode_per_token_ms +
    ADMISSION_EMA_ALPHA * measured_decode_per_token_ms;

ADMISSION_EMA_ALPHA is 0.2f, meaning new measurements get 20% of the vote, history keeps the other 80%. Prefill and decode get separate EMAs on purpose, timed as two distinct boundaries rather than one number split in half. Prefill is a 32-token parallel pass; decode is eight one-token-at-a-time passes, each with its own transformer forward and its own argmax. Different shapes, different costs, no pretending otherwise. The very first cycle always gets admitted, because there’s no history yet to distrust and every decision after that is grounded in something the runtime measured about its own hardware, not a number copied off a spec sheet.

A small aside for anyone who’s spent time around telecom networks: “admission control” isn’t a term this project invented. Cellular and ATM networks have been deciding whether to accept a new call without wrecking everyone else’s quality of service for decades. A robot’s 33ms cycle asking “can I admit this chunk without breaking my one promise” is the exact same decision, just made 30-ish times a second instead of once per phone call.

5. The memory problem: forgetting on purpose

Every vision frame processed costs 32 slots in a KV cache with room for exactly N_MAX_TOKENS = 4096 slots total, shared across the system prompt, all retained frames, and a few transient action tokens during decode. Do the division and the ceiling shows up fast: worst case, this cache holds floor(4096 / 32) = 128 frames, and at roughly one admitted frame per 33ms cycle, it fills up in a matter of seconds. After that, every new frame means an old one has to go and the interesting decision is which one.

FIFO, the obvious baseline, always evicts the oldest frame, no questions asked. It’s the nihilist of eviction policies: everything is temporary, being young is the only crime.

// FIFO eviction baseline — evicts the single oldest retained frame-block on each overflow step.
bool kv_manager_fifo_evict_if_needed(KvManager* manager,
                                     int slots_needed,
                                     double eviction_timestamp_sec) {
  // Repeat oldest-frame eviction until enough free physical slots exist for the incoming frame.
  while (manager->num_free_slots < slots_needed) {
    if (manager->num_retained_frames <= 0) {
      return false;
    }
    // FIFO always removes the temporally oldest retained frame — no similarity computation.
    kv_manager_evict_oldest_frame(manager, eviction_timestamp_sec);
  }
  return true;
}

Semantic eviction is more discerning. It asks “which frame is least likely to be missed,” not “which frame is oldest.” Every retained frame already has a pooled 1536-dimensional embedding from its trip through the vision encoder. The policy walks every adjacent pair of retained frames, computes cosine similarity, and evicts the older half of whichever pair is most similar — the frame most redundant with its neighbor, not the frame that’s simply been around longest:

int best_pair_index = 0;
float best_similarity = -2.0f;
for (int pair_index = 0; pair_index < manager->num_retained_frames - 1; ++pair_index) {
  const float* embedding_a =
      &manager->pooled_embeddings[pair_index * HIDDEN_SIZE];
  const float* embedding_b =
      &manager->pooled_embeddings[(pair_index + 1) * HIDDEN_SIZE];
  const float similarity = kv_manager_cosine_similarity_host(embedding_a, embedding_b);
  if (similarity > best_similarity) {
    best_similarity = similarity;
    best_pair_index = pair_index;
  }
}
// ... evict the older frame of best_pair_index, shift retained frames down, rebuild indices ...

Think of it as evicting whichever frame has a near-identical roommate, rather than whichever frame simply moved in first. If two consecutive frames look almost identical — the arm hasn’t moved, the scene hasn’t changed — you lose almost nothing dropping one. If a frame captures something genuinely different, it survives, because there’s no similarly-boring neighbor to pair it with and blame. That’s “saliency retention” designed straight into the eviction rule instead of bolted on as an afterthought. Cosine similarity itself is nothing exotic — a 1536-dimensional dot product and two norms, computed on the host:

float kv_manager_cosine_similarity_host(const float* embedding_a, const float* embedding_b) {
  float dot_product = 0.0f;
  float norm_a = 0.0f;
  float norm_b = 0.0f;
  for (int hidden_index = 0; hidden_index < HIDDEN_SIZE; ++hidden_index) {
    const float value_a = embedding_a[hidden_index];
    const float value_b = embedding_b[hidden_index];
    dot_product += value_a * value_b;
    norm_a += value_a * value_a;
    norm_b += value_b * value_b;
  }
  const float denominator = std::sqrt(norm_a) * std::sqrt(norm_b);
  if (denominator < 1.0e-8f) {
    return 0.0f;
  }
  return dot_product / denominator;
}

Sliding window, the third policy, comes with an honesty note written directly into its own source file — I’m quoting it, not paraphrasing, because it says it better than I would:

// SLIDING-WINDOW EQUIVALENCE NOTE (§6.1):
// With this project's fixed uniform 32-token vision frame-block size and identical overflow
// behavior, sliding_window and fifo produce mathematically equivalent eviction outcomes:
// both always evict the temporally oldest retained frame when at capacity. This file implements
// sliding_window as a distinct proactive code path (cap enforced before insert) per the spec,
// rather than aliasing fifo — report this equivalence plainly in the final benchmark report (§9).

Since every frame costs exactly the same 32 slots, “keep the last N frames” and “evict the oldest whenever full” turn out to be the same policy in two different outfits like sliding window showed up to the costume party dressed as FIFO and nobody at the door noticed. It still gets its own code path and it evicts proactively, before an insert, once the frame count hits a cap, rather than reactively during an insert when slots run out, but the end state matches FIFO exactly. That’s not a bug hiding under a rock. It’s the honest mathematical result of a uniform frame size, documented in the file that implements it instead of discovered by a confused reader three months later.

6. What’s actually running on the GPU

Everything so far has been about scheduling. This section is about the thing being scheduled: a transformer, written entirely by hand, with no cuBLAS, no libtorch, and no ONNX runtime standing between the code and the silicon.

The model is Qwen2.5-Coder-1.5B-Instruct, and its dimensions are compile-time constants pulled from the model’s own config, not hand-derived guesses that will bite someone at 2 a.m.:

constexpr int HIDDEN_SIZE = 1536;
constexpr int NUM_LAYERS = 28;
constexpr int NUM_ATTENTION_HEADS = 12;
constexpr int NUM_KV_HEADS = 2;
constexpr int HEAD_DIM = 128;
constexpr int INTERMEDIATE_SIZE = 8960;
constexpr int VOCAB_SIZE = 151936;

Twenty-eight decoder layers. Twelve query heads sharing just two KV heads — a 6-to-1 grouped-query-attention ratio. That ratio is a big part of why the KV cache fits a fixed 4,096-slot budget at all: with only 2 KV heads instead of 12, every cached token costs a fraction of what full multi-head attention would. The whole KV cache, across all 28 layers, both K and V, at the full 4,096-slot ceiling, comes to roughly 235MB — comfortably inside the 8GB budget. The KV cache was never going to be the thing that blew the memory ceiling. A vision stream that never ends was always the actual culprit.

Turning a camera frame into tokens. A 128×64 RGB frame gets carved into 16×16 patches — 32 per frame, exactly matching VISION_TOKENS_PER_FRAME — and each patch gets linearly projected into a 1536-dimensional vector. One block per patch, one thread doing all the work per block (no fancy parallelism needed for something this small):

__global__ void vision_patch_embed_kernel(float* __restrict__ patch_embeddings,
                                          float* __restrict__ pooled_embedding,
                                          const std::uint8_t* __restrict__ frame_rgb,
                                          const float* __restrict__ vision_projection_weight,
                                          const float* __restrict__ vision_projection_bias,
                                          const int frame_width,
                                          const int frame_height,
                                          const int patch_size,
                                          const int hidden_size) {
  const int patch_index = blockIdx.x;
  const int patches_per_row = frame_width / patch_size;
  const int patch_row = patch_index / patches_per_row;
  const int patch_col = patch_index % patches_per_row;

  // Flatten patch pixels row-major, RGB interleaved per pixel — matches Python prep contract.
  float flattened_patch[768];
  int flat_index = 0;
  for (int pixel_row = 0; pixel_row < patch_size; ++pixel_row) {
    for (int pixel_col = 0; pixel_col < patch_size; ++pixel_col) {
      const int image_row = patch_row * patch_size + pixel_row;
      const int image_col = patch_col * patch_size + pixel_col;
      const int pixel_offset = (image_row * frame_width + image_col) * 3;
      for (int channel_index = 0; channel_index < 3; ++channel_index) {
        const float normalized_pixel =
            (static_cast(frame_rgb[pixel_offset + channel_index]) / 127.5f) - 1.0f;
        flattened_patch[flat_index++] = normalized_pixel;
      }
    }
  }

  // Linear projection: patch_embedding[hidden] = W[hidden,768] @ flat_patch + bias.
  float* output_patch_ptr = patch_embeddings + patch_index * hidden_size;
  for (int hidden_index = 0; hidden_index < hidden_size; ++hidden_index) {
    float accumulated = vision_projection_bias[hidden_index];
    for (int flat_dim_index = 0; flat_dim_index < 768; ++flat_dim_index) {
      accumulated += flattened_patch[flat_dim_index] *
                     vision_projection_weight[hidden_index * 768 + flat_dim_index];
    }
    output_patch_ptr[hidden_index] = accumulated;
  }
  // ... pooled-mean accumulation into pooled_embedding across all 32 patches omitted here ...
}

Worth being upfront: the projection weights are generated once at startup from a deterministic PRNG (splitmix64 feeding a Box-Muller transform), not trained on anything. Same seed, same weights, every run — which makes the systems behavior fully reproducible, while the vision encoder itself stays exactly what it is: a very confident intern on day one. The shapes are correct, the paperwork is filed on time, memory footprint checks out to the byte — nobody has taught it what it’s actually looking at yet. A deliberate scope cut, not something I’m trying to cover up.

The attention kernel is the part of the codebase with the clearest fingerprints of an engineering decision being argued with and revised. The comment above it tells the whole story:

// Grouped-query causal attention: one CUDA block per query head (gridDim.x = num_query_heads),
// with ATTENTION_BLOCK_THREADS cooperating threads per block instead of the previous single-
// thread implementation. This replaces an O(1)-thread, fully serialized kernel (which forced
// every dot product, softmax reduction, and weighted value-sum onto one lane with zero warp-
// level memory-latency hiding) with a block-parallel reduction that scales with num_valid_slots.

An earlier version ran everything on a single thread per head — easy to audit, and about as parallel as a single-lane toll booth on a Friday morning. The current version launches one block per query head (128 cooperating threads each) and moves through attention in three synchronized phases. Threads spread across KV slots for the dot product instead of piling onto lane zero:

float thread_max_logit = -1.0e30f;
for (int slot_list_index = threadIdx.x; slot_list_index < num_valid_slots;
     slot_list_index += blockDim.x) {
  const int physical_slot = valid_slot_indices[slot_list_index];
  const float* key_head_ptr =
      key_cache + (static_cast<:size_t>(kv_head_index) * N_MAX_TOKENS + physical_slot) *
                      head_dim;
  float dot_product = 0.0f;
  for (int dim_index = 0; dim_index < head_dim; ++dim_index) {
    dot_product += query_head_ptr[dim_index] * key_head_ptr[dim_index];
  }
  const float scaled_logit = dot_product / sqrtf(static_cast(head_dim));
  my_logits_scratch[slot_list_index] = scaled_logit;
  thread_max_logit = fmaxf(thread_max_logit, scaled_logit);
}

Two shared-memory tree reductions later (max, then softmax sum), the weighted value-sum hits a genuinely fun problem: every thread owns a disjoint set of KV slots, but all of them write into the same head-dimension output vector, 128 coworkers all reaching for the same shared whiteboard at once. The fix is atomicAdd straight into shared memory: fine on any GPU since Kepler, and much simpler than hand-rolling a second reduction just to avoid a little contention:

for (int dim_index = 0; dim_index < head_dim; ++dim_index) {
  atomicAdd(&shared_output[dim_index], attention_weight * value_head_ptr[dim_index]);
}

Because every query head’s attention is fully independent of every other head’s, all twelve blocks can genuinely run across different streaming multiprocessors at once, which is the entire performance argument for tearing up the old single-thread kernel and writing this one.

The rest of the layer: RMSNorm, RoPE, SwiGLU, every matmul behind every projection follows the same hand-written, no-library rule, run once per layer for 28 layers, twice per reasoning chunk: a 32-token parallel prefill for the vision frame, then eight sequential decode steps. That mismatch in shape is exactly why prefill and decode earned separate EMA estimators back in section 4, treating a 32-wide parallel pass and eight one-token passes as the same kind of work would just make the deadline estimate worse.

Proving it isn’t lying to you. Writing your own transformer by hand is a wonderful way to introduce a subtle sign error into RoPE and not notice for three weeks. The project’s insurance policy is a correctness gate that runs the hand-written stack against a real HuggingFace forward pass, on the same weights and the same tokens, and demands they agree:

if (hidden_cosine < 0.999f) {
  std::fprintf(stderr, "hidden cosine %.6f < 0.999 thresholdn", hidden_cosine);
  return false;
}
if (max_hidden_abs_diff > 5.0e-2f) {
  std::fprintf(stderr, "hidden max abs diff %.6f > 5e-2 thresholdn", max_hidden_abs_diff);
  return false;
}

Cosine similarity of at least 0.999 against the HuggingFace output, max per-dimension difference under 0.05. Not a benchmark but a much more basic claim: this hand-written math is actually computing the thing it says it’s computing, not just producing numbers that look plausible and run fast, though I built the scaffolding for a hard 33ms deadline, and honestly, the transformer itself is currently ~100x too slow to live inside it (~ 3.1s). Everything else in this post depends on that gate passing first.

7. The honest list of what this is not (yet)

If you’ve read this far expecting a table of latency numbers: there isn’t one, on purpose. Consider this the part of the post where the runtime sits down, makes eye contact, and lists its own limitations before you can find them yourself.

1. No measured performance numbers in this post. No deadline-miss rate, no p50/p95/p99 latency, no fallback rate, no peak-VRAM measurement — even though the runtime is fully instrumented to compute every one of them, and tools/run_benchmarks.py is wired to run all three eviction policies back to back. This post is the architecture; the measurements are separate, undated work.
2. Built and tested on a Hopper GPU (sm_90), not an edge board. CMakeLists.txt hardcodes CMAKE_CUDA_ARCHITECTURES 90. No Jetson, no DRIVE, no physical robot anywhere in this project’s development loop.
3. The 8GB VRAM ceiling is modeled, not measured. VRAM_CEILING_BYTES is a constant a sampler thread checks every 100ms against a KV cache sized to fit comfortably underneath it is a design target, not a wall this has actually been crashed into.
4. The vision encoder is untrained. Same PRNG-seeded weights every run, real shapes and memory footprint, zero labeled examples ever seen. Any claim about what it perceives would be fiction; this post only claims things about the systems engineering, never about perception quality.
5. Sliding window and FIFO are the same policy here, mathematically, as section 5 already confessed still worth repeating because a less honest project would quietly bury it.
6. The correctness gate covers the transformer, not the whole pipeline. It says nothing about the admission controller’s cost estimates or the semantic policy’s behavior on real footage. That’s what a future benchmarking pass is for.

None of this changes the architectural claim that a hard deadline, a fixed memory ceiling, and a mismatched sensor frequency can all be first-class design constraints instead of afterthoughts. It just means the claim, for now, is about the shape of the system, not a number on a chart.

8. Where this leaves things

This is the first post in what I’m calling “Physical AI Systems” — not because I have a five-part roadmap already written (I don’t, and I’m not inventing one just to sound organized), but because the questions this project raised are bigger than one repo. What does it actually take for a model to operate under a clock it cannot negotiate with? How much of “robotics AI” is just systems engineering wearing a friendlier name?

If you build inference infrastructure for a living: go ask whatever’s serving your models, honestly, what happens when the input never stops and the output has a deadline. If the answer is “it just keeps computing until it’s done,” you’ve found the same gap this project is trying to close.

If you build robots for a living: I’d genuinely like to hear how far 4096 tokens and 8GB are from your real hardware budget, and whether a semantic eviction policy would survive contact with real footage instead of a replayed clip.

And if this is your first real look inside an inference runtime: you now know what an admission controller, a KV eviction policy, and a grouped-query attention kernel each do, and why a robot needs all three running at once. Most tutorials skip straight to model.generate(), as if the hard part were saying please. This is what’s underneath that call, for a workload that isn’t allowed to be late.

Now go find out what your own inference stack does when nobody’s watching the clock.

Disclaimer: The illustrations in this article were generated with the help of AI image tools. They are illustrative, not photographic, and any labels visible inside the images are stylized rather than authoritative — refer to the article body and the code itself for precise function names, constant values, and architecture details. The article text, code references, and constant values are not AI-generated.