How to Utilize OKF Efficiently to Enable Knowledge Exchange Among LLMs

• The pattern. This is Google’s Open Knowledge Format skeleton — a Markdown file with a YAML frontmatter block — repurposed for agent hand-off. The repo’s frontmatter carries one extra load-bearing field the general OKF spec does not define: token_pointer, an absolute path to the pre-computed .npy array in shared memory. Human-readable body, machine-readable pointer.
• The mechanism. Three Qwen2.5-Coder models of different sizes (7B / 3B / 1.5B) cannot share a KV cache — they have different architectures. But they can share pre-computed token IDs, because the whole Qwen2.5-Coder family ships one identical BPE vocabulary. This repo tokenizes once, hands off the integer array through /dev/shm/qwen_tokens/, and lets every downstream agent skip its own tokenizer entirely on the input side.
• The numbers. Median of 7 trials per prompt, 3 blocks, greedy decoding, 64 new tokens: on the 3B model, mean baseline TTFT drops from 69.3 ms to 49.9 ms — a 28.0% reduction. On the 1.5B model, from 49.6 ms to 30.9 ms — a 37.8% reduction. Both models pass the coherence heuristic on every sample. Full pipeline wall clock is 41.3 s end to end (Agent 1: 3.9 s, Agent 2: 18.7 s, Agent 3: 15.7 s).
• The guardrail. Feeding a downstream model an integer array that meant a different subword under its own vocabulary does not crash anything. It generates a fluent, coherent-looking, completely wrong report. So before any agent trusts another agent’s integers, this pipeline runs a full ~151,936-entry get_vocab() dict equality check — not a vocab_size comparison, the real thing.
• What this does NOT claim. Short-block regime (few-hundred-token blocks). No custom CUDA — this is orchestration on top of transformers‘ existing model.generate(input_ids=...) API. Tokenizer equivalence is verified for the exact three checkpoints this repo pins, not a family-wide standing guarantee.

TL;DR up front, so you can leave with the point: if you have ever wired three or more LLM-based agents from the same model family into a pipeline that fans out over one shared document, your CPU is running the exact same Byte-Pair Encoding merges over the exact same characters two or three times in a row, because each agent’s tokenizer is a stateless newborn that has no idea the previous agent already produced the same integer array. This post is about a small pipeline of three Qwen2.5-Coder models (7B, 3B, 1.5B) where the upstream agent tokenizes once, drops a NumPy array of int64 token IDs into /dev/shm/qwen_tokens/, and every downstream agent calls model.generate(input_ids=...) directly on that array. It also — and this is where the actually interesting engineering lives — refuses to let anyone else in the pipeline trust that array until it has confirmed, byte for byte, that every model in the chain agrees on what those integers mean. This is orchestration, not a CUDA kernel. But if you have ever debugged an LLM pipeline that produced fluent, on-topic, wrong-in-a-different-way-every-run output, you already know the shape of the problem this piece of infrastructure is designed to prevent.

Github repo: https://github.com/AnubhabBanerjee/inter-llm-tokf

---

1. A confession: your second agent is doing your first agent’s homework, twice

Let me dramatise the moment this whole repo is about.

Imagine you have three LLM agents chained together. Agent 1 is a big model, it reads a design document. Agent 2 is a mid-sized model, it evaluates part of it. Agent 3 is a small model, it writes the final report. All three of them come from the same model family — same tokenizer, same vocabulary, same everything above the hidden layers — just at three different sizes. Since you are not made of H100s, and running a 7B model three times when a 1.5B model will do for the last step would be, frankly, rude to your GPU.

Now watch what happens on a naive setup:

You: “Agent 1, please read this design document and pass the relevant sections to Agent 2.”

Agent 1 (7B): “On it. Loading tokenizer. Running BPE over the whole document. Sections split. Handing off the interesting sections to Agent 2 as strings. ✅”

You: “Great. Agent 2?”

Agent 2 (3B): “Hello, I am a beautiful, stateless newborn. Loading my own tokenizer. Running BPE over the same characters Agent 1 already ran BPE over three seconds ago. Writing an evaluation.”

You: “Wait, you have the exact same tokenizer as Agent 1.”

Agent 2 (3B): “I do?”

You: “Yes. You are literally in the same model family. Same vocabulary, same subword IDs, same everything.”

Agent 2 (3B): “That’s nice. Anyway, I have re-tokenized the input from scratch and I am ready to generate. Please stand by. 🫡”

You: “…and Agent 3?”

Agent 3 (1.5B): “Loading tokenizer. Running BPE over Agent 2’s output—”

You: “You know what, forget I asked.”

That is the joke, and it is the dirty secret of every multi-agent LLM pipeline that fans out over one shared piece of text using models from the same family. The tokenizer is not the bottleneck — a fast Rust-backed BPE tokenizer is not slow, and I will not lie to you and pretend it is. But the tokenizer is redundant work, and how many times you do redundant work is not a function of how fast the redundant work is. It is a function of how many downstream consumers you fanned out to.

The point of this piece of infrastructure, and the whole reason it took more than a fifteen-line patch, is that the moment you decide to skip the tokenizer on the downstream side, you have inherited a correctness problem that the tokenizer was previously doing for you. The rest of this post is what that looks like when you draw it out honestly, and the one runtime check that is doing all the load-bearing work.

2. Why three sizes at all? (a one-minute crash course on which layer is actually shared)

Skip this if you already know. For everyone else, here is the short version.

The three models in this pipeline are Qwen/Qwen2.5-Coder-7B-Instruct, Qwen/Qwen2.5-Coder-3B-Instruct, and Qwen/Qwen2.5-Coder-1.5B-Instruct. Same architecture family, same tokenizer, three different sizes. The reason they are three different sizes and not one big one is deliberately telecom-flavored, because that is the world I actually came from: the concrete example this repo is built against is a design document proposing that a chain of LLM agents help a mobile core network’s operations team reason about a new control-plane feature — specifically, bolting MCP (Model Context Protocol) and A2A (Agent-to-Agent protocol) style orchestration onto the existing 5G Service-Based Interface. The plan calls for a large “Architect” agent that structures the document, a mid-sized “Protocol Engineer” that evaluates the interesting sections, and a small “Edge Analyst” that produces deployment-ready latency guidance — small enough to run at a far-edge site next to a UPF.

Three sizes, three roles, one pipeline.

Now, one structural fact drives the entire design: you cannot share a KV cache across these three models. Different sizes mean different hidden_size values — 3584 for the 7B, 2048 for the 3B, 1536 for the 1.5B. The shape of a KV cache is derived directly from that number, so there is no reinterpreting one model’s cache as another’s. That door is closed, permanently, by the math.

What is not closed is the tokenizer. Qwen2.5-Coder ships one BPE vocabulary across its entire size range — the whole family is documented to agree on the same integer-to-subword mapping. So while you can’t share activations between differently-sized models, you absolutely can share token IDs, provided — and this “provided” is doing a lot of work, more on that in a minute — every model in the chain really does use that same vocabulary.

If you have read enough distributed-systems papers to be dangerous, this shape is familiar. Two network functions on the same message bus don’t get to assume they agree on message semantics just because they’re both plugged into the same bus. Two models in the same family don’t get to assume they agree on hidden states just because they agree on vocabulary. Different layer, same discipline: find the exact layer of the stack where interoperability is actually guaranteed, and refuse to assume it holds one layer higher just because the layers are adjacent.

The tokenizer is that layer. Everything above it is a shape mismatch. Everything at or below it, if we’re lucky and if we check, is a free integer array.

3. OKF: the “just hand off the integers” pattern

Here is the pitch in five bullets:

1. Agent 1 loads only the 7B model’s tokenizer — never its weights. It splits the document, tags each section, and tokenizes each section.
2. It saves each section’s token IDs as a NumPy int64 array into /dev/shm/qwen_tokens/. That is a RAM-backed tmpfs mount, not disk, so reading it back is a memcpy, never a seek.
3. It also writes one Markdown file per section into okf_workspace/. The Markdown body is the section’s human-readable text. The YAML frontmatter carries the metadata — block_id, tags, token_pointer, token_count, tokenizer_model_id, etc.
4. Agent 2 (the 3B model) reads the frontmatter, follows token_pointer into shared memory, loads the .npy, and calls model.generate(input_ids=...) directly on the loaded tensor. No tokenizer call on the input side.
5. Agent 2 tokenizes its own output (that text has, by definition, never been tokenized before — nothing to reuse), saves that array to shm, writes another OKF file, and Agent 3 (1.5B) does the same trick again.

A quick introduction on the “OKF” (for those who don’t know yet)

OKF stands for Open Knowledge Format, and before you read the frontmatter block below, one thing is worth being honest about.

The Open Knowledge Format is a published spec — Google Cloud shipped v0.1 in June 2026 and v0.2 is now the current version (see GoogleCloudPlatform/knowledge-catalog on GitHub). Its pitch is intentionally minimal: a bundle is a directory of UTF-8 Markdown files, each file is one concept, and each file carries a YAML frontmatter block plus a Markdown body. The only frontmatter field the spec requires is type — a short human-readable string like BigQuery Table, Playbook, or Attested Computation. Everything else is optional metadata. It is a format, not a platform: no schema registry, no SDK, no central authority. If you can cat a file, you can read OKF.

This repo’s okf/ reuses that exact skeleton — one Markdown file per unit of work, YAML frontmatter plus a human-readable body — but interprets it for a job the general spec was not written for: an agent-to-agent hand-off of pre-tokenized integer arrays. So this repo’s required frontmatter fields are not Google’s type; they are block_id, source_agent, stage, title, tags, token_pointer, token_count, tokenizer_model_id, and created_at (see utils/okf_parser.py‘s REQUIRED_FRONTMATTER_KEYS). The load-bearing one is token_pointer — an absolute path into /dev/shm/qwen_tokens/ — which has no equivalent in the general OKF spec because Google’s OKF was designed for durable knowledge sharing, not for a shared-memory hand-off between short-lived agent processes on the same GPU host. Put plainly: this repo’s files are not valid Google-OKF bundles as-is (they lack type, they add token_pointer); the repo is conforming in spirit — same Markdown+YAML aesthetic, same “standardise the interoperability surface, not the content model” instinct — with one domain-specific required field bolted on. This post keeps the repo’s terminology because that is what the source code and the generated files actually use.

With that out of the way, here is the schema in the wild — the actual frontmatter block from okf_workspace/block_004_routing_and_signaling_integration_points.md, unedited:

---
block_id: block_004_routing_and_signaling_integration_points
source_agent: agent_1_architect
stage: 1
title: Routing and Signaling Integration Points
tags:
- routing
- signaling
- security
- deployment
token_pointer: /dev/shm/qwen_tokens/block_004_routing_and_signaling_integration_points.npy
token_count: 3437
tokenizer_model_id: Qwen/Qwen2.5-Coder-7B-Instruct
created_at: '2026-08-04T12:41:39.249368+00:00'
---

The load-bearing field is token_pointer. Everything else — source_agent, stage, tags, token_count, tokenizer_model_id, created_at — exists to support routing and provenance decisions around that one array. Agent 2 filters the workspace by tag (routing or signaling, both trigger it). Agent 3 filters by source agent (agent_2_protocol_eval, so it never accidentally picks up its own output on a re-run). The tokenizer_model_id field is there so a future audit can cross-check per-file which tokenizer actually produced the bytes at that path, instead of trusting one pipeline-start assertion for all eternity.

One more architectural detail worth calling out: each agent is a separate OS process. src/run_pipeline.py launches them via subprocess.run, one at a time. That is deliberate, not lazy: a CUDA context only releases its VRAM back to the driver when the process holding it exits. So running three multi-GB models sequentially inside one process would leak each prior model’s VRAM into the next agent’s memory budget unless every caller remembered to manually del model; torch.cuda.empty_cache() — and even that is not always sufficient to fully reclaim CUDA context overhead. Subprocess isolation makes VRAM release unconditional and automatic. On a single-GPU box, this is what lets the 7B, then the 3B, then the 1.5B each get the whole card to themselves in turn, without ever needing all three resident in memory simultaneously.

4. The actual save/load code, all six meaningful lines of it

Now the code that does the actual hand-off. From utils/token_manager.py, verbatim:

def save_token_array(token_ids: torch.Tensor, block_name: str) -> Path:
    ...
    token_ids_as_numpy_int64 = token_ids.detach().cpu().numpy().astype(TOKEN_ARRAY_DTYPE)
    destination_path = QWEN_TOKENS_SHM_DIR / f"{block_name}.npy"
    np.save(destination_path, token_ids_as_numpy_int64, allow_pickle=False)
    return destination_path

That is the write half. Three lines that actually move data. QWEN_TOKENS_SHM_DIR is /dev/shm/qwen_tokens, a RAM-backed tmpfs mount. TOKEN_ARRAY_DTYPE is np.int64, matching torch’s default torch.long, specifically so the load side never needs a casting step. And allow_pickle=False is there because a .npy file with allow_pickle=True will happily deserialise and execute pickled Python objects from disk — unnecessary attack surface for an array that is, by definition, pure numeric data.

Here is the read half:

def load_token_array(pointer_path: Path) -> torch.Tensor:
    ...
    token_ids_as_numpy_int64 = np.load(pointer_path, allow_pickle=False)
    if token_ids_as_numpy_int64.dtype != TOKEN_ARRAY_DTYPE:
        raise TypeError(...)
    return torch.from_numpy(token_ids_as_numpy_int64)

Also three meaningful lines. np.load reads back the exact .npy header (which embeds dtype, shape, and byte-order, all explicit), the defensive dtype check refuses to silently .astype() if some future code path ever writes something other than int64 into this namespace, and torch.from_numpy(...) shares memory with the NumPy array — zero-copy, since token IDs from this point forward are never mutated in place by any agent.

That is the entire on-wire format. A NumPy .npy file, int64, on a RAM-backed mount. If you were expecting something exotic, sorry to disappoint you.

The last piece of the puzzle is what a downstream agent actually does with the loaded tensor. From utils/model_loader.py, the two entry points that Agent 2 and Agent 3 can call — the naive baseline, and the optimized path. Look at them side by side, because the whole optimization is one function call’s worth of difference:

def generate_from_text(model, tokenizer, prompt_text, max_new_tokens):
    ...
    wall_clock_start = time.perf_counter()

    encoded_prompt = tokenizer(prompt_text, return_tensors="pt")

    input_ids = encoded_prompt["input_ids"].to(model.device)
    attention_mask = encoded_prompt["attention_mask"].to(model.device)

    return _generate_and_measure_ttft(
        model, tokenizer, input_ids, attention_mask, wall_clock_start, max_new_tokens
    )

Baseline. Clock starts before tokenizer(...) runs, so the tokenizer-encode cost this pipeline exists to skip is fully included in the reported TTFT. That is not accidental — it is deliberately honest. If the baseline started its clock after tokenization, the comparison would understate the real savings and pretend the tokenizer was free. It is not free. It is fast, but it is not free.

Now the optimized side:

def generate_from_token_ids(model, tokenizer, token_ids, max_new_tokens):
    ...
    wall_clock_start = time.perf_counter()

    input_ids = token_ids.unsqueeze(0).to(model.device)
    attention_mask = torch.ones_like(input_ids)

    return _generate_and_measure_ttft(
        model, tokenizer, input_ids, attention_mask, wall_clock_start, max_new_tokens
    )

The clock also starts here, with no tokenizer call preceding it — the whole point of the comparison. token_ids was already produced by an upstream agent’s tokenizer, already saved into shm, already loaded off shm. All this function does before starting the model is unsqueeze a batch dimension and copy the array to the GPU. The tokenizer argument is still passed in, but only because _generate_and_measure_ttft needs it to supply pad_token_id and to decode the output tokens back to text — the input side genuinely never hits the tokenizer.

The single-line difference between those two functions — one line, tokenizer(prompt_text, ...) — is the entire savings. It sounds almost too small to write an article about. Keep reading, because the failure mode on the other side of “almost too small” is not small at all.

5. The part where I stopped trusting the vendor docs

Here is the sentence from my own project notes that made me nervous enough to write code instead of just shipping the pipeline: “Qwen2.5-Coder is documented to share one tokenizer across the whole family.” Documented. By whom? Checked how recently? What happens to three agents’ worth of generated text if that turns out to be true for six of the seven sizes and subtly not true for the one I picked?

A tokenizer mismatch here doesn’t crash anything. That’s the scary part. model.generate(input_ids=[1234, 5678, ...]) doesn’t know or care whether 1234 meant the same subword to whoever produced it as it means to the model about to embed it. It will happily run a forward pass on integers that decode to complete nonsense under its own vocabulary, and it will happily generate a fluent-looking continuation of that nonsense. You get a confidently wrong report, not an error. Your tokenizer: not the bottleneck. Your assumptions about your tokenizer: entirely the bottleneck.

So before any agent is allowed to trust a token array it didn’t produce itself, this runs — from utils/env_checks.py:

def verify_tokenizer_equivalence(
    model_ids: tuple[str, ...] = PIPELINE_MODEL_IDS,
) -> None:
    ...
    loaded_tokenizers = {
        model_id: AutoTokenizer.from_pretrained(model_id) for model_id in model_ids
    }

    reference_model_id = model_ids[0]
    reference_tokenizer = loaded_tokenizers[reference_model_id]
    reference_vocab_size = reference_tokenizer.vocab_size

    reference_vocab = reference_tokenizer.get_vocab()

    for candidate_model_id in model_ids[1:]:
        candidate_tokenizer = loaded_tokenizers[candidate_model_id]

        if candidate_tokenizer.vocab_size != reference_vocab_size:
            raise RuntimeError(
                f"Tokenizer vocab_size mismatch: {reference_model_id} has "
                f"vocab_size={reference_vocab_size}, but {candidate_model_id} "
                f"has vocab_size={candidate_tokenizer.vocab_size}. Token IDs "
                "produced by one are not safe to feed into the other's "
                "embedding layer."
            )

        if candidate_tokenizer.get_vocab() != reference_vocab:
            raise RuntimeError(
                f"Tokenizer vocabulary mismatch between {reference_model_id} "
                f"and {candidate_model_id}: at least one token string maps "
                "to a different integer id between the two. Direct token "
                "injection across these models would silently corrupt "
                "downstream generations."
            )

        if candidate_tokenizer.special_tokens_map != reference_tokenizer.special_tokens_map:
            raise RuntimeError(
                f"Special-tokens map mismatch between {reference_model_id} "
                f"({reference_tokenizer.special_tokens_map}) and "
                f"{candidate_model_id} ({candidate_tokenizer.special_tokens_map})."
            )

Three checks, deliberately layered.

The first check is vocab_size. It exists purely so a mismatch here produces a short, immediately-readable error naming the two integers that disagree, instead of forcing whoever is debugging this to diff two ~151,936-entry dicts by hand to find that the sizes alone differ.

The second check — the load-bearing one — is full dictionary equality on get_vocab(). Not a vocab_size comparison. A full dict != dict over the entire ~151,936-entry mapping of every subword string to every integer id. Two tokenizers can have identical sizes and still disagree about what integer 42 means. This is the check that would catch a “shuffled id assignment for even a single subword” mismatch, which is exactly the kind of failure that produces fluent nonsense downstream instead of a loud error.

The third check is special_tokens_map. A model’s chat template and stopping behavior depend on these exact strings/ids matching too — a correct main vocabulary with a divergent EOS id, for example, would make a downstream agent’s generate() call fail to stop at the boundary Agent 1 intended.

I wanted the actual guarantee, not the cheap proxy for it. Ran it against the real triplet before writing another line of pipeline code, and it held: Qwen2.5-Coder-7B-Instruct, Qwen2.5-Coder-3B-Instruct, and Qwen2.5-Coder-1.5B-Instruct all agree, byte for byte. Good. But “it held, this time, for this triplet” is a very different sentence from “it’s documented to hold,” and only one of those two sentences belongs in a pipeline you’re going to run unattended.

6. The receipts

Same 3 sections of the design doc (the ones Agent 1’s keyword scan tagged routing or signaling — block_002 at 1948 tokens, block_003 at 2292 tokens, block_004 at 3437 tokens). Same greedy decoding. Max 64 new tokens for the timed comparison. One throwaway warm-up call absorbed before any timed measurement so cuBLAS’s first-call kernel selection doesn’t contaminate the numbers. Median of 7 repeated trials per block, to smooth out millisecond-scale scheduling and GPU-clock jitter.

Straight from scripts/benchmark.py‘s output:

=== Benchmarking Qwen/Qwen2.5-Coder-3B-Instruct ===
  Metric 1 (TTFT reduction): mean_baseline=69.3 ms, mean_injection=49.9 ms, reduction=28.0% -- PASS
  Metric 2 (semantic fidelity): PASS

=== Benchmarking Qwen/Qwen2.5-Coder-1.5B-Instruct ===
  Metric 1 (TTFT reduction): mean_baseline=49.6 ms, mean_injection=30.9 ms, reduction=37.8% -- PASS
  Metric 2 (semantic fidelity): PASS

[benchmark] ALL ACCEPTANCE METRICS PASSED

In table form:

The interesting bit is not that both models got faster — of course they did, they stopped doing redundant work. The interesting bit is why the 1.5B model’s percentage reduction is noticeably bigger than the 3B model’s, even though the absolute number of milliseconds saved is roughly comparable. The explanation is in the repo’s own README, and it is worth quoting because it’s the kind of thing that trips people up if they only read the table:

The tokenizer’s CPU cost is the same string, tokenized once, regardless of which model reads the result — but GPU forward-pass latency scales with model size. For the smaller 1.5B model, that GPU-side floor is lower, so the (roughly fixed) tokenizer cost it avoids is a larger fraction of its total time-to-first-token.

That is also, incidentally, why blindly growing the input document further does not push the reduction toward 100%. Past a certain input length, GPU compute time itself starts growing too, and the percentage plateaus rather than climbing indefinitely. The savings scale with how much text you would otherwise redundantly re-tokenize, times how many downstream agents share that same input, divided by how big each downstream model’s own forward pass is. On a short single-hop demo, you get double-digit percent. On a large source document fanned out to many downstream agents of the same family, you pay the BPE cost once instead of N times, which is exactly the regime the plan was built for.

The semantic-fidelity side of the receipts is a heuristic, on purpose. Two ratios: printable-character ratio ≥ 0.98, and unique-word ratio ≥ 0.25 across the sample’s tokens. Cheap enough to run on every generation, calibrated to catch the specific “garbage output” failure mode a tokenizer mismatch or byte-order bug produces — degenerate repetition of one token, or a wall of non-printable control-character noise — not a general quality judgment. Every sample from every model passed. Read more details about the results here.

7. Wrap: the actually interesting part was the guardrail

The interesting part of this project was never “skip the tokenizer, it’s slow.” Tokenizers, especially the fast Rust-backed kind, are not the bottleneck anyone thinks they are — the numbers above prove that themselves. Saving 20 ms of TTFT is nice. It is not the point.

The interesting part was building the one piece of infrastructure that makes skipping the tokenizer safe: a runtime check that refuses to let one agent trust another agent’s integers until it has actually confirmed they speak the same language, byte for byte, vocabulary entry for vocabulary entry. That check is what turns “20 ms faster” from a footgun into a reliable engineering move. Without it, you have a pipeline that is fast when it works and confidently wrong when it doesn’t, and no clean way to tell which one you’re currently living in.

Every multi-agent pipeline that passes state between models is making an assumption like this somewhere, usually silently. Sometimes it’s about tokenizer vocabularies. Sometimes it’s about hidden-state dimensions. Sometimes it’s about the meaning of a particular chat-template string. Sometimes it’s about which side of an RPC boundary the retries live on. Mine just happens to be about BPE integer-to-subword mappings, because that is what this repo’s optimization strategy leans on. Yours is somewhere else. Go find it. It’s probably not documented either.

If you want to reproduce the numbers, python scripts/benchmark.py on a CUDA GPU with enough VRAM for a bf16 3B checkpoint will do it. If you want to reproduce the pipeline itself against your own input, drop your document into data/raw_input.txt and python src/run_pipeline.py walks through the three stages, cleans up shm on the way out, and leaves three OKF files behind in okf_workspace/.

Small pipeline. Modest numbers. One load-bearing check. That’s the whole shape of it.

Disclaimer: The illustrations in this article were generated using AI (Claude Opus 4.8). They are illustrative, not photographic, and any labels visible inside the images are stylized rather than authoritative — refer to the article body and the code itself for precise function names, metric values, and architecture details.