Insights & Updates

Moonshot AI recently released Kimi K3, the largest open-weight model available as of mid-2026, and the first open model to reach the 3-trillion-parameter class. While most labs chase scale by simply adding more hardware, Moonshot took a different path. Instead of just building a bigger model, they redesigned how the model remembers things. That redesign is really the whole story of K3. The Road to K3 The story starts with Kimi K1.5 , which focused on scaling reinforcement learning and improving the model’s basic reasoning ability. By mid-2025, the team released Kimi K2 , a roughly 1-trillion-parameter model that pushed further on architecture and training pipelines. Not long after, Kimi K2.5 added native multimodal support and stronger agentic skills, meaning the ability to use tools and carry out multi-step tasks with less human guidance. As these models grew, they ran into a bottleneck that every large language model eventually hits: memory. The Memory Problem Standard transformer models remember everything. Every time the model generates a new word, it looks back at every previous word in the conversation. This lookup relies on something called the KV cache, which stores a representation of every token seen so far. The problem is that this cache grows with the conversation. Storage needed for it scales linearly with context length, but the total computation needed to keep re-checking that growing cache scales quadratically. In plain terms, double the conversation length and you don’t just double the cost, you roughly quadruple it. At the scale of a multi-trillion-parameter model with a million-token context window, that cost becomes something only the largest labs can absorb. To build a genuinely massive model without that cost exploding, Moonshot needed the model to forget selectively, rather than hoarding every detail forever. Kimi Linear: The Testbed In October 2025, Moonshot released Kimi Linear , a smaller 48-billion-parameter model built to test this idea. This is where they developed Kimi Delta Attention (KDA) , the mechanism that would later be scaled up to power K3. Here’s a simple way to picture the difference between standard attention and KDA: Standard attention is like a notebook. Every new piece of information gets written on a fresh page, and the notebook keeps growing forever. KDA is like a whiteboard with a fixed amount of space. When new information arrives, nothing new gets added. Instead, the model checks what has changed and updates the board in place. Using this approach, Kimi Linear cut memory use by around 75% compared to standard attention, with only a small trade-off in raw capability. The Delta Rule: How KDA Actually Forgets The mechanism behind KDA is called the Delta Rule , and it is what lets the model manage a fixed-size memory instead of a constantly growing one. Instead of stacking new entries on top of old ones, the Delta Rule updates memory in place. If a meeting moves from Tuesday to Thursday, the model does not keep both facts around. It erases “Tuesday” and writes “Thursday” into the same memory slot. Two learned values control this process for each attention head: β (writing strength) : controls how strongly new information overwrites what is already stored. α (forgetting factor) : controls how much of the old state is kept versus allowed to fade. KDA takes this further by giving each attention head 128 separate forgetting dials, rather than one shared setting. This lets the model be very selective. It can drop small talk and filler almost immediately, while holding on tightly to a specific name or date from hundreds of thousands of tokens earlier in the conversation. This “erase then write” approach also prevents new information from interfering with older memories, which was one of the main weaknesses of earlier linear attention designs. KDA itself builds on earlier research into Gated DeltaNet, which first introduced the idea of learning how much information to write into a recurrent memory state. Kimi K3’s Core Architecture Kimi K3 is a 2.8-trillion-parameter model, currently the largest open-weight model released, and Moonshot describes it as the first open model in the 3-trillion-parameter class. It supports a 1-million-token context window and understands text, images, and video within a single model, using a vision encoder called MoonViT-V2 (about 401 million parameters) to handle image and video input, alongside a roughly 160,000-token vocabulary. A few architectural pieces work together to make this possible. According to Moonshot’s technical report, K3 has 93 attention layers in total, and 69 of them use KDA to keep memory use flat regardless of context length. Moonshot reports that this delivers noticeably faster decoding at million-token context lengths compared to standard attention. To actually see why KDA saves so much memory, it helps to look at what a normal attention layer is doing under the hood, then see what KDA changes. In standard attention, every token produces a key vector k_t and a value vector v_t, and both get stored forever in the KV cache. When the model wants to generate the next token, it compares the current query q_t against every single key seen so far: o_t = Σ (for all j ≤ t) softmax(q_t · k_j / √d) · v_j That sum runs over every previous token j. That's exactly why the cache keeps growing, and why the cost keeps climbing the longer the conversation gets. KDA replaces that entire running list of keys and values with one fixed-size matrix, call it S. Think of S as a compressed lookup table, roughly d_k rows by d_v columns, that never grows no matter how long the sequence gets. It's updated at every step using the Delta Rule: S_t = S_(t-1) · diag(α_t) · (I − β_t k_t k_t^T) + β_t k_t v_t^T Dense-looking, but each piece has a plain job: (I − β_t k_t k_t^T) is the erase step. It looks at the direction k_t points in and clears out whatever value was previously stored along that direction. β_t controls how aggressively it erases and rewrites, that's the writing strength from earlier. β_t k_t v_t^T is the write step, putting the new value v_t into that same freed-up slot. diag(α_t) is the forgetting gate, applied first. It decides how much of the old state survives at all, channel by channel, before the new information gets written in. So at every token, the model does an erase-then-write on a fixed-size memory, rather than appending a new page to a notebook. That’s the whiteboard idea, written out as math. The output at each step is just a lookup against this compressed state: o_t = S_t^T · q_t No sum over the past, no growing list. Just one matrix-vector multiply against a state that stays the same size whether the conversation is 1,000 tokens or 1,000,000. KDA’s real trick is that α isn't one number for the whole layer. K3 gives every attention head 128 separate α values, one per channel. So the model isn't choosing between "remember everything" and "forget everything" as one blanket setting. It can let filler and small talk decay almost instantly in some channels, while other channels hold a specific name or date for hundreds of thousands of tokens, all inside the same fixed-size state. Gated Multi-Head Latent Attention (Gated MLA), for the rest. KDA alone risks losing exact details over very long stretches, since it’s built for efficient forgetting rather than perfect recall. So the remaining 24 layers use Gated MLA instead, an evolution of the Multi-Head Latent Attention architecture originally introduced by DeepSeek, which compresses the KV cache into a smaller latent representation. In K3, these layers are interleaved with the KDA layers at roughly a 3-to-1 ratio: KDA handles efficient, fixed-size memory, while Gated MLA acts as a compressed full-recall path that preserves exact, token-by-token history for details that can’t be allowed to fade. The way MLA compresses things is a genuinely different trick from KDA, not just a smaller version of the same idea. Instead of storing full key and value vectors for every token, MLA first compresses the token’s hidden state h_t into a small latent vector: c_t = W_c · h_t where W_c is a down-projection that squashes h_t into something much smaller. Only c_t gets cached, not the full key and value. When the model actually needs to attend, it reconstructs keys and values from that latent on the fly: k_t = W_k · c_t v_t = W_v · c_t then runs normal softmax attention over those reconstructed keys and values. Because c_t is so much smaller than a full key or value vector, the cache footprint per token drops sharply, even though the model is technically still keeping one entry per token, unlike KDA, which keeps none. The gated part adds one more control on top: a learned gate g_t, a sigmoid that outputs something between 0 and 1, decides how much of this layer's attention output actually passes forward: output_t = g_t ⊙ Attention(q_t, K, V) That gate lets the model dial the exact-recall path up or down depending on whether a given token actually needs it, instead of always applying it at full strength. Attention Residuals (AttnRes). In deep models, information from early layers can get diluted as it passes through many later layers, sometimes called PreNorm dilution. AttnRes acts as a drop-in replacement for standard residual connections, letting later layers selectively pull in representations from earlier layers instead of relying only on a single accumulated stream. Moonshot reports this delivers around 25% better training efficiency for well under 2% additional compute cost. The dilution problem has a simple mathematical root. In a normal deep model, each layer just adds its own output on top of the previous one: x_l = x_(l-1) + F_l(x_(l-1)) Do that 93 times, and whatever the earliest layers contributed gets buried under 90-plus rounds of addition and renormalization. Its relative weight in the final signal keeps shrinking, layer after layer. AttnRes gives later layers a more direct line back, letting layer l pull from a learned mix of everything before it, not just the layer right before it: x_l = F_l( Σ (for all i < l) w_(l,i) · x_i ) where the weights w are learned. A layer near the end of the network can still "hear" a layer near the start almost directly, instead of that signal fading out through 80-something layers in between. Stable LatentMoE. K3 uses a Mixture-of-Experts design with 896 experts, of which only 16 are activated per token. Out of the 2.8 trillion total parameters, Moonshot’s technical report states that roughly 104 billion are active for any given token. This sparsity is what allows the model to reach such a large total parameter count while keeping the actual compute per token manageable. The routing math is fairly direct once it’s written out. For every token, a small router network scores all 896 experts: r = W_r · h_t Only the top 16 scores get kept, turned into weights via softmax, renormalized just among that group: g_i = softmax(r)_i, for i in top-16 The token’s output is then a weighted sum of what those 16 experts individually produce: output_t = Σ (for i in top-16) g_i · E_i(h_t) The other 880 experts do nothing for that token. This is exactly why 2.8 trillion total parameters and 104 billion active parameters can both be true at once: the total is the size of every expert’s weights added together, but the compute for any single token only ever touches 16 of them. Keeping usage even across 896 experts is its own problem, since a router left alone will happily send most tokens to a handful of favorite experts and starve the rest. That’s what Quantile Balancing is built to fix, by setting each expert’s routing bias directly from where its score falls in the overall distribution, instead of bolting on a separate loss term to nudge things toward balance. Put together, Moonshot says these architectural changes, combined with updates to training and data, give K3 roughly 2.5x the overall scaling efficiency of K2, meaning it converts a given amount of compute into more usable capability than its predecessor. Training and Optimization Details Running stable training at 2.8 trillion parameters with only 16 of 896 experts active required a few extra tricks beyond the core attention design. Quantile Balancing. With so few experts active per token, keeping the workload evenly spread across all 896 experts is a real challenge. Instead of the usual approach of adding an auxiliary loss term to nudge routing toward balance, Quantile Balancing sets each expert’s routing bias directly from router-score quantiles, so load balancing falls out of the routing math itself rather than needing a separate, sensitive hyperparameter to tune. Per-Head Muon. Muon is a training optimizer that has become popular for large-scale model training. Moonshot extended it so that attention heads are optimized independently rather than as one block, which the team says gives more adaptive learning at this scale. Sigmoid Tanh Unit (SiTU). This is a custom activation function used in place of more common choices like GeLU or SwiGLU, intended to give the model finer control over activations. Quantization-aware training. Rather than training at full precision and quantizing afterward, K3 is trained with quantization awareness starting from the supervised fine-tuning stage. The released weights use the MXFP4 format with MXFP8 activations, which keeps the model runnable on a wider range of hardware without a separate, lossy quantization step after training. Benchmarks and Real-World Performance At launch on July 16, 2026, Kimi K3 took the top spot on Arena’s WebDev leaderboard (a human blind-vote benchmark for AI-generated front-end code) with a score of 1,679, ahead of Claude Fable 5 and GPT-5.6 Sol at the time. It also led on several sustained coding and agentic benchmarks, including Program Bench, SWE Marathon, BrowseComp, and OmniDocBench. Moonshot has been direct that K3 does not lead across the board. The company’s own materials describe it as trailing the strongest proprietary models, including Claude Fable 5 and GPT-5.6 Sol, on overall performance and on benchmarks like FrontierSWE and HLE-Full, even while outperforming other tested models on its evaluation suite. As is typical with vendor-reported benchmarks, results depend heavily on the exact test harness, reasoning settings, and context management used, so these numbers are best read as a general signal of strength rather than an exact ranking. Release Timeline and Background Kimi K3 comes from Moonshot AI, a Beijing-based company founded in 2023 by Yang Zhilin, who studied computer science at Tsinghua University, completed a machine learning PhD at Carnegie Mellon, and worked on earlier long-context research such as Transformer-XL and XLNet before starting Moonshot. Long-context modeling has been a consistent focus for the company since its earliest products, and KDA is best understood as the latest step in that same line of work. K3 was announced on July 16, 2026, with the full open-weight release following on July 27, 2026, distributed as roughly 96 shards totaling around 1.56 terabytes, under a custom Kimi K3 License. It is available through Kimi.com, Kimi Work, Kimi Code, and the Kimi API, and Moonshot has also contributed a KDA implementation to the vLLM community to support self-hosted deployment. Why This Matters The throughline across K1.5, K2, K2.5, Kimi Linear, and now K3 is a shift away from “just add more compute” and toward teaching models to manage memory the way a person might: keep the important stuff precise, let the unimportant stuff fade, and avoid paying to re-read the entire conversation from scratch every time something new is said. That combination of selective memory (KDA), targeted exact recall (Gated MLA), better information flow across depth (AttnRes), and aggressive but carefully balanced sparsity (Stable LatentMoE) is what let Moonshot push past the 2-trillion-parameter mark on open weights while keeping the model usable at a million tokens of context. Inside Kimi K3: How Moonshot AI Built the Largest Open-Source Model was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.

Library tested: headroom-ai v0.31.0 Goal: Shrink the big junk that LLMs read (JSON, logs, tool dumps, RAG docs) before it hits the model Important: Every LLM answer in this article is mocked . Every token number comes from a real local headroom compression run on my machine. Experiments were performed by Cursor Agents. These are also my personal views and findings from my own setup, not a definitive benchmark. I may have gotten configuration details wrong or missed something that skewed or degraded some of these results Quick summary Headroom sits between your app and the LLM. It compresses tool outputs and other fat context. In my tests: What is Headroom, in plain words? Imagine your AI agent asks a database tool for “sales last week.” The tool returns a giant JSON blob. Most of it is repeated fields, tags, long descriptions, and boring rows. The model still has to read all of it . You pay for those tokens. The answer often only needed a few useful rows. Headroom is a token optimization layer. It tries to: Detect what the content is (JSON, logs, code, text, etc.) Pick a compressor that fits that type Send a smaller version to the LLM Keep originals in a retrieve store ( CCR ), so the model can ask for full detail if needed. Those weird <<ccr:...>> / hash=... strings in compressed output are the lookup keys, not corruption You can use it as: a Python function: compress(messages) a LangChain wrapper: HeadroomChatModel, tool wrappers, memory, retrievers a proxy / MCP server (not the focus of this article) How I tested I did not call OpenAI/Anthropic for answers. Compression: real headroom-ai==0.31.0 LLM replies: a fake MockChatModel that returns fixed text Token counting target model name: gpt-4o (used by Headroom's counters) LangChain: real integrations (HeadroomChatModel, HeadroomToolWrapper, memory, document compressor) That means: Savings numbers = real “The model said …” = fake on purpose (so the article is reproducible) Install pip install headroom-ai langchain-core langchain-community tiktoken # LangChain extra (if you want the documented extra set): # pip install "headroom-ai[langchain]" What does compression look like? (before vs after) I ran before/after captures. Below are the real shapes Headroom produced. Example A — Full chat with a tool call (sales JSON) Tokens: 8,055 to 3,887 ( 51.74% saved , 4,168 tokens) Before (what you would send without Headroom) Same 4 roles. The fat part is only the tool message. [ { "role": "system", "content": "You are a careful assistant. Use tool data." }, { "role": "user", "content": "Summarize sales and top SKUs." }, { "role": "assistant", "content": "", "tool_calls": [ { "id": "call_1", "type": "function", "function": { "name": "fetch_data", "arguments": "{}" } } ] }, { "role": "tool", "tool_call_id": "call_1", "content": "{\n \"query\": \"sales last week\",\n \"total\": 40,\n \"results\": [\n {\n \"id\": 0,\n \"sku\": \"SKU-0000\",\n \"name\": \"Product 0\",\n \"category\": \"Electronics\",\n \"price\": 10.0,\n \"stock\": 100,\n \"tags\": [\"tag-0\", \"tag-1\", \"tag-2\", \"tag-3\", \"tag-4\", \"tag-5\", \"tag-6\", \"tag-7\"],\n \"description\": \"High quality product with durable materials, fast shipping, and a 30-day return policy. High quality product with durable materials, fast shipping, and a 30-day return policy. High quality product with durable materials, fast shipping, and a 30-day return policy. \",\n \"metrics\": {\"views\": 0, \"clicks\": 0, \"conversion\": 0.0}\n },\n {\n \"id\": 1,\n \"sku\": \"SKU-0001\",\n \"name\": \"Product 1\",\n ... 38 more almost-identical objects ...\n }\n ]\n}" } ] After (what Headroom actually returned) Notice: system / user / assistant tool_call stay the same only the tool content changes pretty JSON becomes a compact table-like string repeated long description text becomes a CCR pointer like <<ccr:2facef870b78,string,264B>> [ { "role": "system", "content": "You are a careful assistant. Use tool data." }, { "role": "user", "content": "Summarize sales and top SKUs." }, { "role": "assistant", "content": "", "tool_calls": [ { "id": "call_1", "type": "function", "function": { "name": "fetch_data", "arguments": "{}" } } ] }, { "role": "tool", "tool_call_id": "call_1", "content": "{\"query\":\"sales last week\",\"total\":40,\"results\":\"[40]{category:string,description:string,id:int,metrics.views:int,metrics.clicks:int,metrics.conversion:float,name:string,price:float,sku:string,stock:int,tags:json}\\nElectronics,<<ccr:2facef870b78,string,264B>>,0,0,0,0.0,Product 0,10.0,SKU-0000,100,[tags...]\\nApparel,<<ccr:2facef870b78,string,264B>>,1,12,3,0.01,Product 1,11.37,SKU-0001,99,[tags...]\\nHome,<<ccr:2facef870b78,string,264B>>,2,24,6,0.02,Product 2,12.74,SKU-0002,98,[tags...]\\n...\"}" } ] How to read the after format in plain words: [40]{category:string,...} means "there are 40 rows; here are the column types" each next line is one row as CSV-ish values <<ccr:...>> means "full description text is stored elsewhere; ask to retrieve if needed" So the model still sees SKU, price, stock, metrics. It just does not re-read the same marketing paragraph 40 times. Wait, is [80]{level:string,...} a bug? No. That string is the real compressed output. It looks broken the first time you see it, but it is Headroom's default csv-schema format (SmartCrusher's lossless compaction renderer). What you are seeing is not corrupted JSON. Headroom deliberately rewrote a fat array of objects into: one shape declaration dense CSV-like rows optional CCR hash markers for anything it parked off-prompt Decode the declaration line From my JSON-logs experiment, the tool content became: {"logs":"[80]{level:string,meta.region:string,meta.attempt:int,msg:string,request_id:string,service:string,ts:string} INFO,us-east-1,0,ok path=/api/items/0,req-00000,checkout,2026-07-16T14:30:00Z ... FATAL,...,payment-service DB pool exhausted...,req-00067,... ..."} Why bother? Pretty JSON repeats "level":, "service":, braces, quotes, and indentation on every object. Models already understand CSV. Paying those field names 80 times is waste. Headroom also supports other renderers (json, markdown-kv), but csv-schema is the default because it is the token-cheapest of the lossless formats. Decode the hashes (CCR) You will also see markers like: <<ccr:2facef870b78,string,264B>> and footers like: [301 lines compressed to 7. Retrieve more: hash=1d4fbe6309448a101b916c1e] or LangChain-style sentinels: {"_ccr_dropped": "<<ccr:baf8f8ba3c6b 27_rows_offloaded>>"} These are not garbage . They are CCR : Compress, Cache, Retrieve. Compress: shrink what the model sees now Cache: store the original blob locally, keyed by that hash Retrieve: if the model needs the missing detail, it calls a tool Headroom injects: { "name": "headroom_retrieve", "parameters": { "hash": "2facef870b78", "query": "optional BM25 search inside the cached blob" } } In proxy / wrapped-client mode, Headroom intercepts that tool call, returns the original (about 1ms from local cache), and the turn continues. Your app often never sees the retrieve hop. How it gets used in practice: So the hash is a pointer , not the content. The compressed prompt stays small; the truth stays available. Important: experiment scripts measure compression only (mocked LLM answers). They prove the markers appear and tokens drop. A live model actually calling headroom_retrieve is the next step when you wire Headroom's proxy or CCR-enabled client, not something which I have tried. Example B — Plain logs (needle in a haystack) Tokens: 4,527 to 160 ( 96.47% saved ) Before (preview) 2026-07-16 INFO ok 0 xxxxxxxxxxxxxxxxxxxx 2026-07-16 INFO ok 1 xxxxxxxxxxxxxxxxxxxx 2026-07-16 INFO ok 2 xxxxxxxxxxxxxxxxxxxx 2026-07-16 INFO ok 3 xxxxxxxxxxxxxxxxxxxx ... hundreds more INFO lines ... 2026-07-16 FATAL payment DB timeout pool exhausted replica_lag_ms=8421 ... more INFO lines ... After (full output, this is the entire compressed message) 2026-07-16 INFO ok 77 xxxxxxxxxxxxxxxxxxxx 2026-07-16 INFO ok 78 xxxxxxxxxxxxxxxxxxxx 2026-07-16 INFO ok 79 xxxxxxxxxxxxxxxxxxxx 2026-07-16 FATAL payment DB timeout pool exhausted replica_lag_ms=8421 2026-07-16 INFO ok 80 xxxxxxxxxxxxxxxxxxxx 2026-07-16 INFO ok 81 xxxxxxxxxxxxxxxxxxxx 2026-07-16 INFO ok 82 xxxxxxxxxxxxxxxxxxxx [294 lines omitted: 1 ERROR, 300 INFO] [301 lines compressed to 7. Retrieve more: hash=1d4fbe6309448a101b916c1e] This is the clearest “wow” example. The FATAL line stays. Most INFO noise becomes one summary footer plus a retrieve hash (hash=1d4fbe63...). That hash is the CCR cache key: keep the tiny window in-prompt; fetch the full 301 lines only if the model asks. Example C — JSON logs inside a tool message Tokens: 5,658 to 3,126 ( 44.75% saved ) Before (pretty objects) { "logs": [ { "ts": "2026-07-16T14:30:00Z", "level": "INFO", "service": "checkout", "request_id": "req-00000", "msg": "ok path=/api/items/0", "meta": {"region": "us-east-1", "attempt": 0} }, { "ts": "2026-07-16T14:30:07Z", "level": "FATAL", "service": "checkout", "request_id": "req-00067", "msg": "payment-service DB pool exhausted timeout replica_lag_ms=8421", "meta": {"region": "us-east-1", "attempt": 1} } ] } After (schema + dense rows; FATAL still visible) This is the string that looks “wrong” in the text until you know the grammar. It is correct, see the decoder section above. {"logs":"[80]{level:string,meta.region:string,meta.attempt:int,msg:string,request_id:string,service:string,ts:string} INFO,us-east-1,0,ok path=/api/items/0,req-00000,checkout,2026-07-16T14:30:00Z INFO,us-east-1,1,ok path=/api/items/1,req-00001,checkout,2026-07-16T14:30:01Z ... FATAL,us-east-1,1,payment-service DB pool exhausted timeout replica_lag_ms=8421,req-00067,checkout,2026-07-16T14:30:07Z INFO,us-east-1,2,ok path=/api/items/68,req-00068,checkout,2026-07-16T14:30:08Z ..."} Same story: keep the signal, drop repeated JSON punctuation and field names. The model still sees level, msg, request_id, just once in the header, then as CSV cells. Example D — LangChain tool wrapper crush (compress_tool_result_with_metrics) Tokens: 10,001 to 3,297 ( 67.03% saved ) Items: 50 to 24 kept in the visible JSON, plus a CCR drop marker Before (start of payload) { "query": "sales last week", "total": 50, "results": [ { "id": 0, "sku": "SKU-0000", "name": "Product 0", "category": "Electronics", "price": 10.0, "stock": 100, "tags": ["tag-0", "tag-1", "tag-2", "tag-3", "tag-4", "tag-5", "tag-6", "tag-7"], "description": "High quality product with durable materials, fast shipping, and a 30-day return policy. ...", "metrics": {"views": 0, "clicks": 0, "conversion": 0.0} } // ... 49 more rows ... ] } After (kept rows + offload marker) { "query": "sales last week", "total": 50, "results": [ {"id": 0, "sku": "SKU-0000", "name": "Product 0", "category": "Electronics", "price": 10.0, "...": "..."}, {"id": 1, "sku": "SKU-0001", "name": "Product 1", "category": "Apparel", "price": 11.37, "...": "..."}, {"id": 12, "sku": "SKU-0012", "name": "Product 12", "...": "..."}, {"id": 40, "sku": "SKU-0040", "name": "Product 40", "...": "..."}, {"id": 49, "sku": "SKU-0049", "name": "Product 49", "...": "..."}, {"_ccr_dropped": "<<ccr:baf8f8ba3c6b 27_rows_offloaded>>"} ] } This is the LangChain-agent friendly shape: still valid-ish JSON, fewer rows, and an explicit note that 27 rows were offloaded to CCR. One-sentence takeaway Headroom does not rewrite your system prompt or delete the tool call. It shrinks the tool result body (and similar fat blocks), usually into csv-schema ([N]{cols} + rows) and/or CCR hash pointers, or keeps errors in logs and summarizes the rest. Weird-looking does not mean wrong; those hashes are how the model can pull originals back. Part 1: The core API, compress() This is the simplest path. from headroom import compress import json rows = [{"id": i, "name": f"Product {i}", "desc": "quality item " * 20} for i in range(100)] payload = json.dumps({"results": rows}) messages = [ {"role": "system", "content": "You are a careful assistant."}, {"role": "user", "content": "Summarize sales and top SKUs."}, { "role": "assistant", "content": "", "tool_calls": [{ "id": "call_1", "type": "function", "function": {"name": "fetch_data", "arguments": "{}"}, }], }, {"role": "tool", "tool_call_id": "call_1", "content": payload}, ] result = compress(messages, model="gpt-4o") print(result.tokens_before, "->", result.tokens_after) print("saved %", round(100 * result.compression_ratio, 2)) print(result.transforms_applied) My real result for big sales JSON Before: 20,041 tokens After: 9,573 tokens Saved: 10,468 tokens ( 52.23% ) Transforms: router:protected:user_message, router:mixed:0.01 Runtime: about 1.08 seconds for that case So: more than half the tokens gone, and the user question stayed protected. Part 2: Where Headroom performed best 1) JSON tool outputs Big arrays of similar objects are Headroom’s happy place. SmartCrusher keeps a smaller set of rows / signal and offloads the rest (CCR style). User-message JSON (forcing user compression): result = compress( [{"role": "user", "content": "Please analyze:\n" + big_json}], model="gpt-4o", compress_user_messages=True, target_ratio=0.3, ) My result: 15,995 to 7,627 tokens 52.32% saved Practical takeaway: turn optimization on for fat JSON tool traffic. 2) Plain logs when the log router wakes up result = compress( [{"role": "user", "content": plain_logs}], model="gpt-4o", compress_user_messages=True, protect_recent=0, # important in my run target_ratio=0.2, ) My result: 4,527 to 160 tokens 96.47% saved Transform: router:log:0.04 That is the “needle in a haystack” story: keep the FATAL line, drop endless INFO noise. 3) Logs stored as JSON arrays If your logging tool returns JSON, SmartCrusher helps a lot even when plain-text log detection fails. My result: 10,558 to 5,786 tokens 45.20% saved 4) Multi-turn agent history (savings stack) I simulated 3 tool turns. LLM replies were mocked. Compression was real each turn. Cumulative tokens saved across turns: 20,945 Mocked answers still “made sense” for the demo (sales summary, FATAL DB timeout advice, create_order explanation). That does not prove answer quality on a live model, only that the pipeline runs end-to-end. Part 3: Where it did not help (also real) These are as important as the wins. Recent code stayed protected (0% saved) code_search_default: 1402 -> 1402 (0.0%) transforms: ['router:protected:recent_code'] Headroom often refuses to crush fresh code on purpose. That is safer for coding agents, but it means “code dump in context” is not always a token win. Tiny JSON (0% saved) small_json: 56 -> 56 (0.0%) No point compressing 20 characters. Some “realistic” structured text logs (0% saved) My pretty timestamps + service= + request_id= logs on the default tool path: 8261 -> 8261 (0.0%) transforms: ['router:protected:user_message'] Same family of content as the 96% log win, but detection/settings differed. On Windows, Headroom also warned it used a pure-Python content detector by default. Lesson: measure your real payloads. Do not assume every log file gets 90% off. Plain repetitive English text on default tool path (0% saved) 2046 -> 2046 (0.0%) Default live-zone / protection rules mattered more than “this text looks repetitive to a human.” Part 4: LangChain integration (all the common cases) Case A: Wrap a chat model from langchain_core.language_models.chat_models import BaseChatModel from langchain_core.messages import AIMessage, HumanMessage, SystemMessage, ToolMessage from langchain_core.outputs import ChatGeneration, ChatResult from headroom.integrations import HeadroomChatModel from headroom import compress class MockChatModel(BaseChatModel): model_name: str = "mock-gpt-4o" @property def _llm_type(self): return "mock-chat" def _generate(self, messages, stop=None, run_manager=None, **kwargs): text = "Total revenue is led by Electronics and Apparel." return ChatResult(generations=[ChatGeneration(message=AIMessage(content=text))]) llm = HeadroomChatModel(MockChatModel()) # use llm.invoke(messages) as usual Measured compression for the same sales tool context: 12,045 to 5,777 tokens 52.04% saved Mocked reply: “Total revenue is led by Electronics and Apparel. Top SKUs look like SKU-0001 style items from the compressed rows.” So LangChain wrapping + Headroom compression is a clean fit for tool-heavy chats. Case B: Wrap tools (agent sweet spot) I used classic LangChain Tool objects with HeadroomToolWrapper (more reliable in my run than @tool StructuredTool wrapping). from langchain_core.tools import Tool from headroom.integrations import HeadroomToolWrapper, compress_tool_result_with_metrics search_tool = Tool( name="search_database", func=lambda query: big_json_string, description="Search sales DB", ) wrapped = HeadroomToolWrapper(search_tool, min_chars_to_compress=500) out = wrapped("sales last week") Database tool (real metrics): Tool metrics collector after 2 calls: total invocations: 2 total compressions: 1 total chars saved: 49,356 search_database compressed; plain fetch_logs text did not JSON logs via compress_tool_result_with_metrics: 8,411 to 2,276 tokens 72.94% saved items 120 to 41 Mocked agent line used in the demo: “I used the compressed tool outputs. Sales look healthy; checkout FATAL is a DB pool timeout.” Case C: Memory history wrapper from langchain_community.chat_message_histories import ChatMessageHistory from headroom.integrations import HeadroomChatMessageHistory base = ChatMessageHistory() history = HeadroomChatMessageHistory( base, compress_threshold_tokens=800, keep_recent_turns=2, model="gpt-4o", ) My run after 12 bulky turns: messages in history: 24 total chars: 16,952 compression_count: 0 total_tokens_saved: 0 So the API is there, but my synthetic “context-data-N” spam did not trigger compression the way JSON tools did. Another reminder: wire it up, then measure . Case D: RAG / document compressor from headroom.integrations import HeadroomDocumentCompressor compressor = HeadroomDocumentCompressor( max_documents=4, min_relevance=0.1, prefer_diverse=True, ) kept = compressor.compress_documents(docs, query) Query: “What is Python used for in AI and web development?” It dropped cooking, sports, gardening, travel, which is exactly what you want for this query. How the pipeline thinks (simple picture) Your agent / LangChain app | | tool JSON, logs, RAG chunks, files v Headroom router |-- JSON -> SmartCrusher |-- logs -> Log compressor |-- code -> code-aware path (often protected if "recent") |-- text -> other compressors / no-op / protect rules v Smaller messages (+ CCR retrieve option) v LLM provider (in this article: mocked) Two ideas that showed up in my transforms list: Protection (router:protected:user_message, router:protected:recent_code): keep some content untouched Crushing (router:smart_crusher, router:log, router:mixed): shrink the rest That mix is why results vary by content type. Pros Huge wins on fat JSON tools: I saw about 52% on full chat compress and about 78% on direct tool crush. Extreme wins on compressible logs: 96.47% in the best log case. LangChain-native pieces: chat model, tools, memory, retriever compressor. Works without changing your whole stack: compress(messages) is enough to start. Local / open approach: you can test offline with mocked LLMs (like I did). CCR mindset: compression can offload rows instead of forever deleting truth. Multi-turn stacking: my 3-turn sim saved 20,945 tokens across compress calls. RAG filtering helps quality and cost: 12 docs to 4 relevant ones, 67% fewer tokens. Cons / challenges Not every payload compresses. Code and tiny messages often save 0%. Detection is picky. One log format got 96%; a more “realistic” format got 0% on the default path. Defaults protect recent content. That is good for safety, bad if you expected automatic crushing everywhere. You may need flags like compress_user_messages=True or protect_recent=0. Memory wrapper did not save tokens in my synthetic chat. Do not assume “I wrapped it” equals savings. Compression takes time. My big JSON case took about 1.08s. Usually cheaper than paying for 10k extra tokens, but not free. tiktoken / vocab loading can stall the first time; Headroom may fall back to estimates unless cache is warm. Where Headroom is the best fit Use it when your LLM bill is driven by inputs that are huge and repetitive : Coding / ops agents with big tool returns SQL / search tools returning hundreds of rows JSON APIs, Elasticsearch-like dumps, inventory feeds SRE assistants reading long logs (especially if log routing triggers) RAG pipelines that retrieve too many chunks Multi-tool LangGraph / LangChain agents where tool spam fills the context every turn Skip or be careful when: Prompts are already small You need every code token untouched (and protection already keeps them) Legal/medical text where you must validate compression with evals first You cannot accept any risk of dropping a rare but important row (unless you use retrieve/CCR well) Simple decision guide Is the expensive part TOOL OUTPUT or RAG DOCS? | |-- JSON arrays / DB rows -----> YES, start here (I saw 50-78% saves) |-- JSON logs -----------------> YES (I saw ~45-73%) |-- Plain logs ----------------> YES, but tune flags; measure twice |-- Recent source code --------> Maybe little/no save (protected) |-- Tiny messages -------------> No Full LangChain “agent-shaped” pattern I recommend from langchain_core.tools import Tool from headroom.integrations import HeadroomChatModel, HeadroomToolWrapper from headroom import compress # 1) wrap model (use your real ChatOpenAI in production) llm = HeadroomChatModel(your_chat_model) # 2) wrap fat tools db_tool = Tool(name="search_database", func=search_db, description="DB search") wrapped_tools = [HeadroomToolWrapper(db_tool, min_chars_to_compress=1000)] # 3) optional: compress message lists yourself between graph nodes result = compress(messages, model="gpt-4o") messages = result.messages print("saved", result.tokens_saved) In production, replace MockChatModel with ChatOpenAI / ChatAnthropic. Keep the same wrappers. Cost intuition (using my numbers, not a vendor invoice) Suppose input costs about $2.50 per 1M tokens (example price only). Saving 10,468 tokens on one fat JSON call is about $0.026 once Saving 20,945 tokens over 3 agent turns is about $0.052 once That looks small, until the agent runs thousands of times a day. Then it becomes rent money. Latency side: my compress calls were often 30ms to 270ms , with the biggest JSON near 1s . Still usually worth it versus shipping 2x tokens to a frontier model. Final verdict Headroom is a strong token optimization library for LangChain apps that drown in tool JSON and retrieved docs. In my local run with headroom-ai 0.31.0 : best everyday win: JSON tools (about 52–78%) best peak win: plain logs with log routing (about 96%) best product pattern: wrap tools + compress between agent turns biggest caveat: protection + detection mean some inputs save 0% If your agent’s context is mostly clean chat text, Headroom will not wow you. If your agent’s context is mostly tool sludge , it can pay for itself quickly. Repro notes All numbers in this article were produced by local scripts that: Build mocked tool payloads Run real compress() / LangChain Headroom wrappers Use MockChatModel for answers Write results.json (aggregate metrics) and before_after.json (actual prompt text before/after) Library version recorded: 0.31.0 . LLM mode: fully_mocked . If you want to show stakeholders the “look,” open before_after.json and compare before_messages vs after_messages for json_tool_chat, plus plain_logs.after_full. Further reading GitHub: https://github.com/headroomlabs-ai/headroom Docs: https://headroomlabs-ai.github.io/headroom/ LangChain guide: https://headroomlabs-ai.github.io/headroom/langchain/ I Tried To Cut My LangChain Agent Tokens With Headroom — Here Is What Actually Happened was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.

A Short Note Before Starting This is not a technical article. It is simply a reflection on the last year of my life. A lot happened during this one year — academics, internships, research, hackathons, interviews, and many moments in between. Some things worked out exactly as hoped, while others did not. Looking back now, it feels like this year changed the way I think about work, growth, success, failure, and even uncertainty. How It All Started Recently turned 22, and the last year felt like everything was moving at once. There was always something happening. One week was spent preparing for interviews, the next week working on a research paper, and after that there would be a hackathon, an exam, or a project deadline waiting around the corner. The funny thing is that none of this was part of some grand plan. There was no roadmap, no carefully designed timeline, and no perfect strategy. Most of the time, it was simply about moving from one opportunity to another and trying to make the most of it. At that point, it felt like life was just busy. Looking back now, it feels more like a year where growth happened quietly in the background without being noticed. What the Year Looked Like From the outside, the year probably looked very organized. Attended ICPC Asia West Regionals, which is an experience that will stay in memory for a long time. Interned at Arctic Wolf, got the opportunity to build some amazing things and met some amazing engineers as well. That experience showed that engineering is much more than writing code. It is about understanding problems, making decisions, and taking ownership. Towards the end of the internship, it was converted into a PPO, which felt like a nice reminder that consistent effort eventually pays off. Alongside that came around five hackathons. Each one exposed new ways of thinking and building. Some ideas worked, some failed, but every event taught something new. Two research papers were published during the year. More than anything else, research taught patience. Unlike coding contests or projects where results come quickly, research often takes months before there is any clear outcome. Somewhere in between all of this, reached Knight on LeetCode. Interned at IISc and got a deeper look into research and problem-solving. Also worked at an early-stage startup (ProjektAnalytics.) where there was direct interaction with clients. I learnt a lottt here. Learnt how to interact with clients and built products, UX and optimizations. That environment was very different because there was no buffer between the problem and the solution. Requirements had to be understood, built, and delivered with very little room for mistakes. Projects were built that felt far too ambitious at the beginning. Interviews happened at companies that once seemed completely out of reach. There was also the chance to attend events like YC Startup School and GitHub Constellation and meet people building incredible things. When written down, it all sounds very structured. Living through it felt anything but structured. When Things Did Not Go As Planned Of course, not everything worked out. There were times when everything seemed to be moving in the right direction, and then suddenly something would fall apart. An interview would not go well. A project would fail. A research idea would lead nowhere. Plans that looked solid would completely change. Earlier, there was always a need to have everything under control. If something was due tomorrow, it felt better to finish it two days earlier. Preparation brought comfort. This year changed that. Calculated risks started replacing perfect preparation. Sometimes decisions had to be made without having complete certainty. Sometimes important things were prioritized while other things were left unfinished. Many of those choices felt risky at the time. Some failed. Some worked out surprisingly well. Over time, that completely changed how risk is viewed. Rejections and What They Taught Me One of the hardest parts of the year was dealing with rejections, especially in final interview rounds. Those rejections feel different because the finish line is right there. Weeks of preparation go into them. Multiple rounds get cleared. It starts to feel real. And then suddenly it ends. For a long time, there was an attempt to find the exact reason behind every rejection. Maybe one answer could have been better. Maybe more preparation was needed. Eventually, that way of thinking stopped making sense. Sometimes everything possible is done, and things still do not work out. That does not always mean failure. Sometimes it is timing. Sometimes it is fit. Sometimes the outcome is simply outside personal control. Understanding that made moving forward easier. Trying to Build in a Fast-Moving World A lot of time was also spent trying to build AI products and turn them into something useful. The challenge was rarely the building itself. The real challenge was speed. Many times, weeks would go into building an idea, only for a large company to release a similar feature for free shortly after. At first, that felt frustrating. Later, it became one of the most valuable lessons of the year. Building is only one piece of the puzzle. Timing matters. Positioning matters. Understanding the problem deeply matters. A great product is not always the one with the most features. Learning to Figure Things Out Alone There were many situations where getting stuck felt unavoidable, whether in code, research, or important decisions. The easiest option was always asking someone else for help. But something interesting happened whenever extra time was spent trying to solve problems independently. It was never just one problem being solved. New ways of thinking started developing. Different approaches became clearer. Confidence grew slowly. That kind of learning stays much longer than simply being given an answer. The Role of People One thing that stood out during the year was how much the people around us shape our thinking. Through internships, hackathons, conferences, and events, there was a chance to meet some incredibly talented people. People building startups. People publishing research. People solving problems at a very high level. Being around them naturally raises expectations. Not because anyone tells you to do more, but because you start realizing what is possible. Some really good friendships were built along the way too, and they made the journey much more enjoyable. How My View of Interviews Changed A few years ago, interviews seemed like a test where the only thing that mattered was getting the correct answer. Now they feel very different. The best interviews often feel more like conversations. They are about how problems are approached, how ideas are explained, and how clearly experiences can be communicated. Technical skills still matter. But thinking clearly and communicating clearly matter just as much. How I Look at Results Now The biggest lesson from this year is probably this: Results matter, but thinking about them all the time does not help. There was a time when a lot of energy went into worrying about outcomes. Interview results, contest ratings, paper decisions, internship applications everything felt important. Now it feels different. The focus has shifted more towards the process. If progress is happening, even slowly, there is usually a feeling that the result will eventually take care of itself. Maybe not immediately. Maybe not in the exact way expected. But eventually. Some things will work out and some will not. That is simply how life works. What matters most is being able to look back and honestly say that the effort was real. That there was genuine work behind whatever happened. That feeling lasts much longer than any result. Looking Back I actually haven’t achieved every single thing which was planned, but I'm super grateful for every single opportunity I got. This year did not provide a secret formula for success. It did not teach how to control outcomes. Instead, it taught something much simpler. Do the work. Keep showing up. Take risks when they make sense. Accept that things will go wrong sometimes. Learn from failures. Keep moving forward. The more this year is reflected on, the more one thing becomes clear: growth rarely feels dramatic while it is happening. Most of the time, it just feels like another busy day, another deadline, another problem to solve. Only later do you realize how much those ordinary days changed you. Onto the next year now. Unlike 22, this one feels much more uncertain. There are plans, of course, but there are also many things that are completely unknown. Right now, there is no clear picture of what the next two or three months will look like, and honestly, that is not a bad thing. And yeah, I am not more a “college student” Over the past year, one thing became very clear: some of the best opportunities came from places that were never part of the plan. The internship, the research, the hackathons, and the people met along the way – many of those moments were impossible to predict beforehand. There is a quote that I came across recently: “Life is what happens while you are busy making other plans.” The more I think about it, the more true it feels. The moments that make life exciting are often the ones where something is happening — an interview result waiting to come out, a research paper decision, a contest rating update, a new opportunity, or even a challenge that was never expected. There is always something around the corner, and that uncertainty is what makes the journey interesting. So while there is no idea what the next few months will bring, there is something to look forward to. More challenges, more risks, more lessons, and hopefully a few stories worth writing about when 23 comes around.

Have you ever wondered how your phone recognizes your face to unlock itself, even if you are wearing a hat, changing your hairstyle, or sitting in a dark room? It has never seen that exact photo of you before, yet it instantly knows it’s you. How does it do that? The secret weapon behind this magic is something called Contrastive Loss . At first, it sounds like a scary math term. But don’t worry! It is actually a super simple and clever concept. To understand it, we just need to look at how it differs from regular AI learning. Regular AI: “What Is This?” Imagine you are building an AI model to recognize animals. You show it pictures of cats, dogs, and birds. In standard machine learning (which uses something called classification loss ), the AI’s job is to look at a picture and guess the correct label. If you show it a dog and it says “Dog,” it gets a gold star. If it says “Cat,” it gets corrected. The AI is basically asking itself one question over and over again: “What is this?” It draws invisible lines in its brain to separate cats from dogs. Contrastive Learning: “How Similar Are These Two Things?” Contrastive learning has a completely different goal. Instead of guessing a label, the AI looks at pairs of images at the same time. Instead of asking “What is this?”, the AI asks: “How similar are these two inputs?” Think of it like a game of matching magnets: The Pull (Similar Pairs): If you show the AI two different pictures of the same golden retriever, the AI recognizes they are a match. The contrastive loss function tells the model to pull their digital representations (called embeddings) closer together. The Push (Different Pairs): If you show the AI a picture of a golden retriever and a picture of a cat, it recognizes they do not match. The loss function tells the model to push them far apart. The Margin Rule: When pushing different things apart, the AI doesn’t need to push them away forever. Once they are separated by a safe distance (called a margin ), the model stops pushing. It says, “Okay, these are clearly different enough,” and moves on. Why This is a Huge Deal By focusing on similarity instead of labels , the AI creates a beautiful, organized map in its mind. All the pictures of you naturally cluster together in one corner, pictures of your friend cluster in another corner, and pictures of cats stay far away. Because the AI learns the concept of “sameness” rather than just memorizing what a dog looks like, it becomes incredibly versatile. This is exactly how powerful technology works today, including: Face Recognition: Matching your face to your ID. Image Search: Finding similar clothes online when you upload a photo. Recommendation Systems: Showing you videos similar to the ones you just watched. Summary to Go Next time you think about machine learning, just remember this simple shortcut: Classification answers: “What is this?” Contrastive Learning answers: “How similar are these two things?” If you want to see how this works behind the scenes in actual Python code, check out this clean NumPy implementation here: https://github.com/saqlain2204/TensorTonic-Solutions/blob/main/contrastive-loss/contrastive-loss.py

Where It Started There was a time when I was very consistent with solving DSA problems. Every day, without fail, I would sit and solve at least one problem. That routine slowly changed how I think and how I approach problems in general. Now, I have become a little inconsistent because of work, but I still understand the value of what that daily habit gave me and why it mattered. It Is Not About Coding Forget that there is a programming language. A problem is just a problem. It is like a brain teaser that needs to be understood and broken down. You read it, sit with it, and think about what it is asking. There is no need to rush to a keyboard. You can imagine solving it on paper or just in your head. The goal is simple: how would you approach this problem as a human, without thinking about syntax or code. The Real Skill: Approach The most important part of solving a problem is not coding, it is the approach. When you read a question, your brain should start forming a direction. What is given, what is needed, and what are the possible ways to get there. You are not trying to be perfect, you are trying to build a path. If you are able to form a reasonable approach within 5–6 minutes of reading the problem, then you are on the right track. That clarity matters much more than writing the final code. Coding Comes Later Once the approach is clear, coding becomes a simple step. At that point, the programming language is just a tool to express what you have already thought through. You are translating your logic into code. If your thinking is clear, the code will follow naturally. If your thinking is unclear, coding will feel confusing and messy. This is why the real work always happens before typing anything. Daily Practice Builds Thinking When I was solving problems every day, this process became natural. I did not feel stuck when I saw a new question because I trusted my thinking. Even if I did not know the solution immediately, I knew how to start and how to move forward step by step. That confidence did not come from remembering solutions, it came from repeatedly trying to build them on my own. Struggle Is the Point Some problems will feel difficult, and that is where the real value lies. You will try different ideas, and many of them will fail. That is not wasted effort. That is the process that forces your brain to adapt and improve. Easy problems might feel good in the moment, but hard problems are the ones that actually make a difference in how you think. Some Days Will Be Bad There will be days when nothing works. A problem that you could have solved on another day might suddenly feel hard. You may read it multiple times and still not get any clear idea. This is normal. It does not mean your ability has gone away. It just means your brain is not in the right state at that moment. What matters is continuing the process, not judging yourself based on one day. Patterns Will Come Naturally After solving enough problems, patterns begin to appear. You start recognizing similar structures and ideas across different questions. Things that once felt new start to feel familiar. This does not happen because you tried to memorize, it happens because you have seen and worked through enough variations over time. Thinking in the Age of AI With the rise of AI, writing code is becoming easier and faster. Many people may start relying on it heavily, which can make thinking more similar and limit new ideas. But this also creates an advantage for those who focus on thinking. If you build strong problem-solving skills and use AI only as a tool for execution, you can move much faster. You focus on the idea and the approach, and let AI help with the implementation. This combination can lead to building things that you might not have attempted otherwise. Why This Still Matters Solving DSA problems is not just about preparing for interviews. It is a way to train your brain to think clearly and stay patient. It teaches you how to handle confusion, how to break down complex situations, and how to keep moving even when things are not obvious. These are skills that apply everywhere, not just in coding. Final Thought Treat every problem like a puzzle. Do not rush to code. First focus on understanding and building an approach. If you keep doing this regularly, your brain stays active, sharp, and ready to solve problems in any situation.

If you work in AI, you’ve probably lived through this nightmare: your model runs perfectly on your laptop, but the moment you try to move it to a server or share it with a teammate, everything breaks. You start seeing red error messages about “missing libraries” or “wrong CUDA versions.” This is exactly why Docker exists. In AI engineering, Docker isn’t just a “nice-to-have” tool; it’s the standard way to make sure your work actually works everywhere. It creates a secure, isolated “bubble” for your code so that it doesn’t matter what computer you’re using — the environment stays exactly the same. Understanding the Core Concepts To really get how Docker works, you need to understand three main things: the Dockerfile, the Image, and the Container. First, there is the Dockerfile . Think of this as your master recipe. It’s a simple text file where you write down every single thing your AI needs to run. You’ll list the version of Python you want, the libraries like PyTorch or Scikit-learn, and any specific settings your code requires. Next is the Docker Image . When you tell Docker to “build” your recipe, it creates an Image. This is a frozen, unchangeable snapshot of your entire setup. It contains your code and all the tools needed to run it. Because this snapshot never changes, you can send it to a friend or upload it to the cloud, and they will get the exact same setup you have. Finally, we have the Docker Container . This is the Image brought to life. When you “run” an image, it becomes a container. It’s an isolated process that runs on your computer but stays separated from your other files. This isolation is a lifesaver because it allows you to run two different projects with completely different requirements on the same machine without them ever interfering with each other. What Happens If You Don’t Use Docker? Trying to build AI without Docker is like trying to build a house on shifting sand. You will eventually hit three major technical walls. The first is Dependency Conflicts . One project might need an old version of a library, while another needs the newest version. If you install them both directly on your computer, they will fight, and your code will crash. Docker solves this by giving every project its own private room to live in. The second issue is Infrastructure Mismatch . Your laptop might run Windows or Mac, but most AI servers run a version of Linux. Subtle differences in how these systems handle files or memory can cause your AI model to act differently or fail entirely. Docker removes this mystery by making the environment identical across all systems. The third problem is Scaling . If your AI tool becomes popular and you need to run it on 100 different servers, you can’t manually set up each one. Without Docker, you are stuck doing manual labor instead of using automation to grow your project. Scaling Up with Docker Compose Modern AI projects are rarely just a single Python script. Usually, you have a “stack” — maybe a Python API to handle requests, a database to store user info, and a fast memory tool like Redis to speed things up. This is where Docker Compose comes in. Instead of starting each part of your project one by one, Docker Compose lets you define your entire system in a single file called docker-compose.yml. With one simple command, you can launch your whole infrastructure. It automatically connects all your different containers so they can talk to each other, making your local development feel exactly like a professional production environment. Essential Features for AI Success There are a few specific technical features that make Docker perfect for AI work. One is Volumes . Usually, when you turn off a container, anything you changed inside it disappears. Volumes allow you to “link” a folder on your real computer to a folder inside the container. This means you can edit your code or save your trained model weights, and they will stay safe on your hard drive even if the container is deleted. Another is Port Mapping . Docker containers are locked tight for security. If your AI model is running a web server inside the container, you won’t be able to reach it from your browser unless you “map” a port. This creates a bridge that lets data travel from your computer into the container. Lastly, and most importantly for deep learning, is GPU Support . Normally, containers can’t “see” your computer’s hardware. But by using the NVIDIA Container Toolkit, you can pass your GPU’s power directly into the Docker container. This lets you train massive models at full speed while keeping all the messy driver installations tucked away inside the container. The Professional Workflow The best way to use Docker is to make it part of your daily routine. You start by writing your Dockerfile using a solid base (like an official PyTorch image). You build that image to lock in your environment. Then, you use a Compose file to link your AI to any databases you need. By following this path, you ensure that your AI models are portable, professional, and ready for the real world. You’ll spend less time fixing “version errors” and more time actually building intelligent systems.

In traditional software engineering, things are simple. You write code that says 2 + 2. You tell the computer, “The answer must be 4.” If the computer says 4, it passes the test. If it says 5, it fails. This is called deterministic logic. It is “Yes” or “No”. AI does not work like that. AI is probabilistic. This means it’s more about “Maybe” or “Probably”. 1. There is No “Right” Answer If you ask an AI to write a poem about a cat, there are a million “correct” poems. One might be funny, one might be sad, and one might be short. How do you write a computer program to “grade” a poem? You can’t. Because the “right” answer is a matter of opinion, not math. This makes it very hard to know if a new version of an AI is actually better than the old one. 2. The “Vibe” Check vs. Hard Data Right now, many AI engineers use the “Vibe Check”. They type a prompt, look at the answer, and say, “Yeah, that looks pretty good”, But you cannot build a global business on a “vibe”. What if the AI is good at poems but starts lying about medical facts? What if it works today but gets “lazy” tomorrow? 3. The “LLM-as-a-Judge” Problem Since humans are too slow to read thousands of AI answers, we use another AI to grade the first AI. AI Unit A writes an answer. AI Unit B (The Judge) gives it a score from 1 to 10. But here is the catch: The Judge can be wrong too. The judge might have its own biases. This is like having a student grade their own classmate’s homework it’s not always fair. 4. Small Changes, Big Messes In a normal app, if you change one line of code, you know exactly what will happen. In AI, if you change one tiny setting, the entire personality of the AI can change. We call this regression. Keeping track of these tiny changes across millions of possible conversations is a nightmare. How We Solve It? To fix this, we use specific methods to turn “vibes” into “numbers”. Here are the most popular ways to do it: Method 1: The “Rubric” Prompt Just like a teacher, we give the AI a grading sheet. We tell it exactly what a “10 out of 10” looks like. Example Rubric Prompt: You are a Quality Assurance bot. Grade the following AI response on a scale of 1-5: 1. Accuracy: Does it contain lies? 2. Tone: Is it polite and professional? 3. Conciseness: Did it use too many words? AI Response: <AI Response/> Source Text: <Original Data/> Give a score and a one-sentence reason for the score. Method 2: RAGAS If you build an AI that reads your company’s PDF files (RAG Engines), you need to make sure it isn’t making things up. RAGAS checks the “faithfulness” of the AI. Method 3: Semantic Similarity Instead of checking if the words are exactly the same, we check if the meaning is the same. We use maths to see how close the AI’s answer is to a “golden answer” provided by a human. We can have a golden dataset. A Related Challenge: “Prompt Drift” One thing many people forget is that AI models change over time. OpenAI or Anthropic might update their models, and suddenly, the prompt that worked yesterday doesn’t work today. This is called prompt drift. To solve this, engineers now use unit tests for prompts. They run the same prompt through the AI every single morning to see if the “vibe” or the “accuracy” has dropped. Example of a simple Python-style test check: def test_ai_response(): user_query = "What is our refund policy?" ai_output = call_ai_model(user_query) # We check if the AI mentioned the most important keyword assert "30 days" in ai_output, "AI forgot to mention the 30-day limit!" assert "receipt" in ai_output.lower(), "AI forgot to ask for a receipt!" The Big Picture: Why This Matters Until we solve “evaluation”, AI will stay a bit like a “black box”. We know it’s powerful, but we are a little bit afraid to trust it with really important jobs like surgery, flying planes, or legal contracts because we don’t have a “thermometer” to measure its accuracy perfectly.

For a long time, it felt like progress meant writing more code. Many of us start our careers thinking that more features, more commits, and more output are the only metrics that matter. But over time, that idea usually breaks. You eventually realize that the real work happens before you ever touch the keyboard. What matters most is understanding the system. Most work happens before writing In real-world projects, especially when building from scratch, very little time is actually spent writing new lines. Most of the effort goes into: Defining the problem: Seeing the goal clearly before the first line is typed. Reviewing the foundation: Understanding what already exists in the codebase. The “Why”: Investigating why something was built a certain way. The “How”: Mapping out how it works internally. The Decision: Determining whether a piece should be reused or replaced. Writing code comes much later. When you spend the time to understand first, the implementation becomes the easiest part of the day. Writing becomes easy when understanding is strong There is a clear pattern in software development. When understanding of a system is weak: Every new feature feels confusing and forced. Bugs keep appearing in unexpected, “unrelated” places. Small changes break the entire build. However, when understanding is strong, the solution becomes obvious. There are fewer stressful decisions to make, and the code feels lighter and calmer. In my experience, the hard part is never the typing, the hard part is thinking clearly. Reading code builds system awareness Reading code forces you to pay attention to the details that actually matter: How data flows through the whole system. Where specific responsibilities are placed. What assumptions were made by the developers who came before you. What constraints do you have to work with. Slowly, a mental picture forms. You begin to see what belongs together, what should stay separate, and what should not be changed casually. This level of awareness rarely comes from simply creating new files. Building from scratch makes this clear When you start a system from zero, the stakes feel higher. Every choice has a ripple effect: Every new feature adds long-term maintenance cost. Every decision made today affects your flexibility next year. Every shortcut is a debt that will eventually be collected. Before writing a single line, the most productive move is to clarify requirements and explore alternatives. To me, a system is mostly a collection of decisions, the code is just the final result. Decide as late as possible A key principle of Lean Software Development is: Decide as late as possible. This does not mean delaying the work. It means delaying irreversible decisions until you have enough information to make them correctly. By keeping options open while your understanding evolves, you avoid boxing yourself into a corner. Reading existing code and studying the system is what makes this strategy possible. Principles that reward “Thinking First” Great software is often the result of following principles that prioritize thought over raw typing speed: DRY (Don’t Repeat Yourself): You can only reuse code if you’ve taken the time to know it exists. KISS (Keep It Simple): Simplicity comes from deep clarity, not cleverness. YAGNI (You Aren’t Gonna Need It): Many features feel “necessary” only when the system isn’t yet understood. Separation of Concerns: You can only set clear boundaries when you understand everyone’s role. Where AI fits realistically AI speeds up writing, but it cannot replace understanding. When you understand your system, AI is a massive help. It generates the repetitive parts and reduces mechanical effort. But when you don’t understand the system, AI becomes a liability. It amplifies wrong assumptions and hides shallow reasoning behind a wall of generated text. AI works best as an extension of clear thinking, not a substitute for it. Output vs. Understanding Typing feels like progress because you can see the lines growing on the screen. Understanding feels slow because it happens entirely in your head. But stable, scalable systems come from fewer assumptions and better decisions. They come from a clear structure that only arrives after you have spent time reading and thinking. Code is just a tool. Understanding is the real skill.

We build an agentic RAG system from scratch. The idea is to go beyond a simple retrieval pipeline and create a system where a language model can decide when it needs external knowledge, retrieve it using hybrid search, and then reason over it in a structured, stateful workflow. The end result is a clean separation between ingestion, retrieval, orchestration, and execution. Before diving into the code, we first establish the environment, dependencies, and project layout. Environment variables Set up the environment variables in a .env file in the root directory of the project. The variables are: PINECONE_API_KEY: used to authenticate with Pinecone. PINECONE_INDEX_NAME: the name of the Pinecone index used for hybrid retrieval. GROQ_API_KEY: used to access the Groq-hosted LLM. Loading these at runtime keeps secrets out of version control and allows the same code to run across different environments without modification. Dependencies Web ingestion and parsing : requests, beautifulsoup4 Vector storage and retrieval : pinecone Embedding and inference : Pinecone hosted embedding models, Groq LLM Agent orchestration : langgraph, langchain Configuration management : python-dotenv Directory structure AGENTIC RAG ├── src │ ├── embeddings.py │ ├── tools.py │ └── workflow.py ├── .env ├── graph.png ├── main.py └── README.md embeddings.py handles ingestion and indexing. Raw web content is fetched, chunked, embedded, and stored in Pinecone. tools.py exposes hybrid retrieval as a callable tool that the agent can invoke. workflow.py defines the agentic control flow using a state graph, deciding when to retrieve and when to respond. main.py acts as the entry point, running the workflow end to end and visualizing the graph. graph.png is a rendered view of the agent workflow, useful for understanding control flow. .env stores configuration and secrets. README.md documents usage and setup. # src/embeddings.py import os from dotenv import load_dotenv import uuid load_dotenv() from pinecone.grpc import PineconeGRPC as Pinecone from pinecone import ServerlessSpec import requests from bs4 import BeautifulSoup def fetch_and_chunk(url, chunk_size=300): """Fetches the content from a URL and chunks it into pieces.""" response = requests.get(url) soup = BeautifulSoup(response.text, "html.parser") paragraphs = [p.get_text() for p in soup.find_all("p") if p.get_text(strip=True)] text = "\n".join(paragraphs) chunks = [] for i in range(0, len(text), chunk_size): chunk = text[i:i+chunk_size] if chunk.strip(): chunks.append(chunk) return chunks def get_all_url_chunks(urls, chunk_size=300): """Fetches and chunks all URLs, returns list of dicts with 'chunk_text'.""" all_chunks = [] for url in urls: chunks = fetch_and_chunk(url, chunk_size=chunk_size) for chunk in chunks: all_chunks.append({"chunk_text": chunk, "source_url": url}) return all_chunks URLS = [ "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/", "https://lilianweng.github.io/posts/2024-07-07-hallucination/", "https://lilianweng.github.io/posts/2024-04-12-diffusion-video/", ] data = get_all_url_chunks(URLS) print(data) PINECONE_API_KEY = os.getenv("PINECONE_API_KEY") pc = Pinecone(api_key=PINECONE_API_KEY) index_name = "hybrid-index" if not pc.has_index(index_name): pc.create_index( name=index_name, vector_type="dense", dimension=1024, metric="dotproduct", spec=ServerlessSpec( cloud="aws", region="us-east-1" ) ) def get_batched_embeddings(pc, data, model, batch_size=96): all_embeds = [] for i in range(0, len(data), batch_size): batch = [d['chunk_text'] for d in data[i:i+batch_size]] result = pc.inference.embed( model=model, inputs=batch, parameters={"input_type": "passage", "truncate": "END"} ) all_embeds.extend(result) return all_embeds dense_embeddings = get_batched_embeddings(pc, data, model="llama-text-embed-v2", batch_size=96) sparse_embeddings = get_batched_embeddings(pc, data, model="pinecone-sparse-english-v0", batch_size=96) desc = pc.describe_index(name=index_name) host = desc['host'] index = pc.Index(host=host) records = [] for d, de, se in zip(data, dense_embeddings, sparse_embeddings): records.append({ "id": str(uuid.uuid4()), "values": de['values'], "sparse_values": {'indices': se['sparse_indices'], 'values': se['sparse_values']}, "metadata": {'text': d['chunk_text']} }) index.upsert( vectors=records, ) We start by building a pipeline that takes raw web content and prepares it for hybrid search. First, we load configuration from environment variables and set up the libraries needed for scraping web pages, generating embeddings, and storing vectors. Next, we extract content from the web. For each URL, we fetch the page and parse the HTML, keeping only paragraph text. This removes navigation elements, scripts, and other markup noise. We then combine the extracted text and split it into fixed-size chunks. Chunking is done at the character level so each piece stays within embedding limits while still preserving local context. Any empty or whitespace-only chunks are discarded. We repeat this process across multiple URLs, resulting in a single collection of text chunks. Each chunk is stored together with its source URL, which later allows us to trace retrieved content back to the original document. Once the text is ready, we initialize the Pinecone client. We check whether the target index already exists and create it if it does not. The index is configured for dense vector similarity using dot product and deployed in a serverless environment, which fits large-scale semantic retrieval workloads. We then generate embeddings in batches to keep the process efficient and within request limits. For every text chunk, we create two representations: A dense embedding that captures semantic meaning using a transformer-based model. A sparse embedding that captures lexical information for keyword-style matching. Using both representations gives us a hybrid setup, where semantic similarity and exact-term relevance can be combined during retrieval. After embedding, we package each chunk into a vector record. We assign a unique ID, attach both dense and sparse embeddings, and store the original text as metadata. This makes it possible to retrieve relevant vectors and reconstruct the underlying text later. Finally, we upsert all records into the Pinecone index. At this point, the content is fully indexed and ready to be used for hybrid retrieval and downstream tasks such as retrieval-augmented generation. The flow stays focused and linear: we move from raw web pages to a production-ready hybrid vector index, without mixing in any querying or generation logic. # src/tools.py import os import uuid from langchain_core.tools import tool from pinecone import Pinecone from dotenv import load_dotenv load_dotenv() PINECONE_API_KEY = os.getenv("PINECONE_API_KEY") INDEX_NAME = os.getenv("PINECONE_INDEX_NAME") pc = Pinecone(api_key=PINECONE_API_KEY) desc = pc.describe_index(name=INDEX_NAME) host = desc["host"] index = pc.Index(host=host) @tool def hybrid_retriever_tool(query: str, top_k: int = 3) -> str: """Hybrid search on Pinecone: returns top answer(s) for a query.""" dense_query_embedding = pc.inference.embed( model="llama-text-embed-v2", inputs=[query], parameters={"input_type": "query", "truncate": "END"} ) sparse_query_embedding = pc.inference.embed( model="pinecone-sparse-english-v0", inputs=[query], parameters={"input_type": "query", "truncate": "END"} ) d = dense_query_embedding[0] s = sparse_query_embedding[0] query_response = index.query( top_k=top_k, vector=d['values'], sparse_vector={'indices': s['sparse_indices'], 'values': s['sparse_values']}, include_values=False, include_metadata=True ) if not query_response.matches: return "No relevant results found." results = [] for match in query_response.matches: results.append({ "page_content": match.metadata.get("text", ""), "metadata": match.metadata }) return "\n".join([doc["page_content"] for doc in results]) if __name__ == "__main__": query = "What is reward hacking?" print(hybrid_retriever_tool.run(query)) We move from indexing to retrieval and expose hybrid search as a reusable tool. We define a hybrid retriever as a tool so it can be directly plugged into agent-based or RAG pipelines. The tool accepts a natural language query and a top_k parameter that controls how many relevant chunks we want back. When a query comes in, we embed it twice. First, we generate a dense embedding that captures the semantic meaning of the query. Then, we generate a sparse embedding that captures lexical signals such as exact terms and keywords. These two embeddings represent the same query from complementary perspectives. With both representations ready, we perform a hybrid query against Pinecone. The dense vector drives semantic similarity, while the sparse vector reinforces keyword relevance. We explicitly request metadata in the response so we can reconstruct the original text chunks while skipping raw vector values to keep the response lightweight. If the search returns no matches, we fall back to a clear message indicating that nothing relevant was found. Otherwise, we iterate over the matched vectors, extract the stored text from metadata, and aggregate the retrieved content into a single response. This format is intentionally simple so it can be passed directly to a language model for answer generation. Finally, a small execution block shows how the tool can be invoked with a sample query. This closes the loop and demonstrates the full retrieval flow: query → embeddings → hybrid search → text reconstruction. Together with the indexing step, this completes the hybrid RAG pipeline. We first prepare and store knowledge using dense and sparse embeddings, and then retrieve it using the same hybrid signals at query time. # src/workflow.py import os from dotenv import load_dotenv load_dotenv() from langgraph.graph import StateGraph, MessagesState, START, END from langgraph.prebuilt import ToolNode from langgraph.checkpoint.memory import MemorySaver from .tools import hybrid_retriever_tool from langchain_groq import ChatGroq from langchain_core.messages import HumanMessage class Workflow: def __init__(self): self.api_key = os.getenv("GROQ_API_KEY") self.llm = ChatGroq(model="llama-3.3-70b-versatile", api_key=self.api_key) self.tools = [hybrid_retriever_tool] self.llm = self.llm.bind_tools(self.tools) self.tools = ToolNode(self.tools) self.workflow = self._build_workflow() def _build_workflow(self): graph = StateGraph(MessagesState) graph.add_node("agent", self._call_model) graph.add_node("tools", self.tools) graph.add_edge(START, "agent") graph.add_conditional_edges( "agent", self._should_continue, { "tools": "tools", "end": END, }, ) graph.add_edge("tools", "agent") checkpointer = MemorySaver() return graph.compile(checkpointer=checkpointer) def _call_model(self, state): messages = state['messages'] response = self.llm.invoke(messages) return {"messages": [response]} def _should_continue(self, state): last_message = state["messages"][-1] if last_message.tool_calls: return "tools" return "end" def run(self, query): result_state = self.workflow.invoke({"messages": [HumanMessage(content=query)]}, config={"configurable": {"thread_id": 42}} ) print("\n--- Conversation Log ---") for i, msg in enumerate(result_state['messages']): print(f"Message {i}: {msg}") tool_calls = getattr(msg, 'tool_calls', None) if tool_calls is None and isinstance(msg, dict): tool_calls = msg.get('tool_calls', None) if tool_calls: print(f" Tool Calls: {tool_calls}") print("--- End of Log ---\n") final_message = result_state['messages'][-1] if isinstance(final_message, dict): return final_message.get('content', '') return getattr(final_message, 'content', '') We now tie everything together by introducing an agentic workflow that can reason, decide when to retrieve information, and maintain conversational state. We start by initializing the core components needed for orchestration. This includes LangGraph for defining the control flow, a memory checkpointer for persistence, and the hybrid retriever tool we built earlier. The language model is initialized using Groq with a large, instruction-capable LLaMA variant, which is well suited for tool calling and multi-step reasoning. Next, we bind the retriever tool to the language model. It allows the model to explicitly decide when external knowledge is required and to invoke the retriever as part of its reasoning loop, instead of relying purely on parametric memory. We then construct the workflow as a stateful graph. The graph operates over a message-based state, where each node receives the conversation history and appends new messages. We define two core nodes: An agent node, responsible for calling the language model. A tools node, responsible for executing any tool calls requested by the model. The execution flow starts at the agent. After the model responds, we evaluate whether it has issued any tool calls. If it has, control is routed to the tools node, which executes the requested tools and appends their outputs back into the message state. Control then returns to the agent so it can continue reasoning with the new information. If no tool calls are present, the workflow terminates. This conditional routing is what gives the system its agentic behavior. The model is not forced to retrieve information on every query; instead, it retrieves only when it determines that external context is needed. To support multi-turn interactions, we attach a memory checkpointer to the graph. This allows the workflow to persist state across invocations using a thread identifier, enabling continuity and conversational grounding rather than stateless question answering. Finally, we expose a run method that drives the entire process. A user query is wrapped as a human message and injected into the workflow. As the graph executes, we log each message and any associated tool calls, making the reasoning and retrieval steps transparent. The final response is extracted from the last message in the state and returned as plain text. At this point, the full RAG loop is complete. We ingest and index knowledge, retrieve it using hybrid search, and integrate it into an agentic workflow where the model can reason, retrieve, and respond in a controlled, stateful manner. # main.py from src.workflow import Workflow if __name__ == "__main__": wf = Workflow() query = "What is reward hacking?" answer = wf.run(query) wf.workflow.get_graph().draw_mermaid_png( output_file_path="graph.png" ) print("Final Answer:") print(answer) We finish by adding a simple entry point that runs the entire system end to end. We start by importing the workflow we defined earlier. This workflow already encapsulates the agent logic, tool usage, memory, and control flow, so the main script stays intentionally minimal. Inside the main block, we instantiate the workflow. This initializes the language model, binds the hybrid retriever tool, and compiles the LangGraph state machine. We then define a user query and pass it to the workflow’s run method. At this point, the full pipeline is triggered: the agent processes the question, decides whether retrieval is needed, invokes the hybrid retriever if required, and produces a final response grounded in the indexed knowledge. After execution, we visualize the workflow itself. By exporting the graph as a Mermaid diagram, we get a clear structural view of how control flows between the agent node, the tools node, and the termination condition. This is useful for debugging, documentation, and explaining the agent’s behavior in a reproducible way. Finally, we print the final answer returned by the workflow. This confirms that the system is working as expected and provides a clean separation between orchestration logic and user-facing output. Building an Agentic RAG System with Pinecone Hybrid Vector Search and LangGraph (With Code) was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.

This article walks through a practical, end-to-end process for fine-tuning a Qwen vision-language model to extract structured text from images. The goal is simple and specific: take images containing technical content and convert them into clean, usable text. The focus is on reliability and correctness rather than generic OCR or broad vision tasks. The entire workflow is designed to run on Kaggle using a single NVIDIA T4 GPU. Memory constraints are treated as a hard limit, so the setup relies on QLoRA, 4-bit quantization, gradient checkpointing, and parameter-efficient fine-tuning through Unsloth. The model is trained on computer-science–focused image–text pairs and formatted as an instruction-following conversation. This ensures the model learns to respond predictably when given an image and a clear extraction instruction. The final fine-tuned and merged model is publicly available on Hugging Face: https://huggingface.co/vanishingradient/qwen-docs-finetuned The rest of the article breaks down the code step by step, explaining why each component exists and how it enables vision fine-tuning on limited GPU hardware. !pip install unsloth trl transformers huggingface_hub !pip install -U bitsandbytes These commands set up the full software stack required to fine-tune a Qwen vision-language model for text extraction. Unsloth is installed to load Qwen models in a memory-efficient way and to enable parameter-efficient fine-tuning methods like LoRA. Without it, training vision models of this size would exceed typical GPU limits. TRL is included to provide supervised fine-tuning utilities. It handles the training loop, loss computation, batching, and gradient updates, removing the need to implement training logic manually. Transformers supplies the actual Qwen model classes, tokenizers, and image processors. Unsloth operates on top of Transformers, so this library is mandatory. Hugging Face Hub enables downloading pretrained models and datasets and later uploading the fine-tuned model or adapters. BitsAndBytes is upgraded to enable low-precision computation such as 4-bit and 8-bit weights. This drastically reduces GPU memory usage while keeping training stable, which is essential when working with vision-language models. Together, these libraries form the minimum required environment to fine-tune Qwen for extracting text from images on constrained hardware. from unsloth import FastVisionModel import torch model, tokenizer = FastVisionModel.from_pretrained( "unsloth/Qwen2-VL-2B-Instruct-bnb-4bit", load_in_4bit = True, # Use 4bit to reduce memory use. False for 16bit LoRA. use_gradient_checkpointing = "unsloth", # True or "unsloth" for long context ) This code loads a pretrained Qwen vision-language model in a way that is optimized for low memory usage. FastVisionModel is Unsloth’s wrapper around Hugging Face vision-language models. It modifies how the model is loaded so that training and inference use less GPU memory without changing the model’s behavior. The model being loaded is a 2-billion-parameter Qwen vision-language instruction model. It is already prepared for vision inputs and text outputs, which is exactly what is needed for extracting text from images. load_in_4bit = True tells the loader to keep the model weights in 4-bit precision. This greatly reduces GPU memory consumption and makes it possible to fine-tune the model on smaller GPUs. If this were set to false, the model would load in higher precision and require much more memory. use_gradient_checkpointing = "unsloth" enables gradient checkpointing using Unsloth’s optimized implementation. Instead of storing all intermediate activations during training, some are recomputed when needed. This trades extra computation for much lower memory usage, which is important when working with long image-text sequences. The output of this step is the model itself and its tokenizer. These two objects are the foundation for all later steps: preprocessing inputs, training, and running inference. model = FastVisionModel.get_peft_model( model, finetune_vision_layers = True, # False if not finetuning vision layers finetune_language_layers = True, # False if not finetuning language layers finetune_attention_modules = True, # False if not finetuning attention layers finetune_mlp_modules = True, # False if not finetuning MLP layers r = 16, # The larger, the higher the accuracy, but might overfit lora_alpha = 16, # Recommended alpha == r at least lora_dropout = 0, bias = "none", random_state = 3407, use_rslora = False, # We support rank stabilized LoRA loftq_config = None, # And LoftQ # target_modules = "all-linear", # Optional now! Can specify a list if needed ) This step converts the base Qwen model into a parameter-efficient fine-tuning setup using LoRA. Instead of updating all original model weights, LoRA adds small trainable matrices on top of the existing layers. This keeps memory usage low, speeds up training, and avoids damaging the pretrained knowledge of the model. The first four flags control which parts of the model are allowed to learn: Finetuning the vision layers lets the model adapt how it reads visual features from images. This is important for text extraction because the visual patterns of characters, formulas, and layouts matter. Finetuning the language layers lets the model improve how it generates text from those visual features, such as producing clean, structured output. Finetuning the attention modules allows the model to better align image regions with text tokens, which is critical for accurate OCR-style behavior. Finetuning the MLP modules allows deeper transformations inside each layer, giving the model more flexibility to learn task-specific patterns. The LoRA-specific parameters control how much the model can change: r is the rank of the LoRA adapters. Higher values give the model more learning capacity but increase memory usage and the risk of overfitting. lora_alpha scales the LoRA updates. Keeping it equal to or larger than r stabilizes training and is a common best practice. lora_dropout is set to zero to avoid randomly dropping LoRA updates, which is usually fine for structured tasks like text extraction. bias = "none" means no bias terms are trained, keeping the fine-tuning minimal and stable. random_state fixes the initialization seed so results are reproducible. use_rslora and loftq_config are disabled here, meaning standard LoRA is used without rank stabilization or quantization-aware initialization. After this step, only a small fraction of parameters are trainable. The model remains lightweight, but it is now capable of learning image-to-text extraction behavior from your dataset. from datasets import load_dataset dataset = load_dataset( "vidore/vidore_v3_computer_science", 'corpus', split="test" ) dataset = dataset.remove_columns( [c for c in dataset.column_names if c not in ["image", "markdown"]] ) This code loads the dataset that will be used to teach the model how to extract text from images. The dataset is downloaded from Hugging Face using the Datasets library. It contains computer science–related images paired with their ground-truth text in markdown format. This pairing is exactly what a vision-to-text model needs during fine-tuning. The split="test" argument selects a specific subset of the dataset. In practice, this split is used here as training data, which is common when experimenting or when the dataset already contains high-quality labels. After loading, unnecessary columns are removed. Only the image column and the markdown column are kept. The image column contains the visual input that the model will look at, such as screenshots, diagrams, or rendered documents. The markdown column contains the expected textual output. This is what the model is trained to produce when it sees the corresponding image. Removing unused columns reduces memory usage and simplifies data handling. It also ensures that later preprocessing and training steps operate only on the inputs and outputs that matter for image-to-text learning. instruction = "Write the markdown representation for this image." def convert_to_conversation(sample): conversation = [ { "role": "user", "content" : [ {"type" : "text", "text" : instruction}, {"type" : "image", "image" : sample["image"]} ] }, { "role" : "assistant", "content" : [ {"type" : "text", "text" : sample["markdown"]} ] }, ] return { "messages" : conversation } This part prepares each training example in the exact format that the Qwen vision-language model expects. The instruction is a short, fixed prompt that tells the model what task to perform. Here, it explicitly asks the model to write the markdown version of the image. Keeping the instruction simple and consistent helps the model learn a clear image-to-text mapping. The convert_to_conversation function transforms one dataset sample into a chat-style conversation. Qwen is trained as an instruction-following model, so it learns best when data is structured as a dialogue rather than raw input-output pairs. The user message contains two pieces of content. The text part provides the instruction, and the image part provides the visual input. This tells the model that the image is what it should reason over to answer the instruction. The assistant message contains only text. This text is the ground-truth markdown extracted from the image. During training, the model is optimized to generate this response when given the user message. The function returns a dictionary with a messages key, which matches the format expected by supervised fine-tuning trainers. This step is critical because it aligns the dataset structure with the model’s original instruction-tuning format, making learning stable and predictable. converted_dataset = [convert_to_conversation(sample) for sample in dataset] This line applies the conversation formatting to the entire dataset. Each sample in the dataset is passed through the conversion function, which wraps the image and its markdown text into a structured user–assistant conversation. The result is a list where every element represents one complete training example in chat format. Each example contains the instruction, the image, and the expected textual response. This step is necessary because the training process does not operate on raw images and labels directly. It expects a sequence of messages that mirrors how the model was originally trained to follow instructions. After this transformation, the dataset is ready to be fed into the fine-tuning trainer without additional restructuring. FastVisionModel.for_inference(model) # Enable for inference! image = dataset[2]["image"] instruction = "Write the markdown representation for this image." messages = [ {"role": "user", "content": [ {"type": "image"}, {"type": "text", "text": instruction} ]} ] input_text = tokenizer.apply_chat_template(messages, add_generation_prompt = True) inputs = tokenizer( image, input_text, add_special_tokens = False, return_tensors = "pt", ).to("cuda") from transformers import TextStreamer text_streamer = TextStreamer(tokenizer, skip_prompt = True) _ = model.generate(**inputs, streamer = text_streamer, max_new_tokens = 128, use_cache = True, temperature = 1.5, min_p = 0.1) This code runs inference with the pretrained model before any fine-tuning. Its purpose is to establish a baseline and verify that the pipeline works end to end. The model is first switched to inference mode. This disables training-specific behavior and enables optimizations needed for generation, such as caching attention states. An image is selected directly from the dataset. This is the same type of image that will later be used for fine-tuning, so the behavior observed here reflects the model’s raw capability on the target domain. A single instruction is defined. This instruction matches the one used during training later, ensuring the comparison is valid. The input is constructed as a chat-style message. The user message contains an image placeholder followed by text. The placeholder tells the tokenizer that an image will be provided alongside the text prompt. The chat template converts this structured message into the exact token sequence the Qwen model expects. The generation prompt is added so the model knows it must now produce an assistant response. The tokenizer is then called with both the image and the formatted text. This step converts the image into visual embeddings and the text into token IDs, packages them together, and moves everything to the GPU. A text streamer is used to print tokens as they are generated instead of waiting for the full output. This is useful for observing generation behavior in real time. The generation call produces markdown text from the image. max_new_tokens limits the length of the output. use_cache speeds up generation by reusing attention states. temperature controls randomness; a higher value makes outputs more varied and less deterministic. min_p applies nucleus-style filtering to avoid extremely low-probability tokens. Because this is run before fine-tuning, the output reflects the model’s generic understanding. Any errors, hallucinations, or formatting issues observed here are what fine-tuning is meant to correct. from unsloth.trainer import UnslothVisionDataCollator from trl import SFTTrainer, SFTConfig FastVisionModel.for_training(model) # Enable for training! trainer = SFTTrainer( model = model, tokenizer = tokenizer, data_collator = UnslothVisionDataCollator(model, tokenizer), # Must use! train_dataset = converted_dataset, args = SFTConfig( per_device_train_batch_size = 2, gradient_accumulation_steps = 4, warmup_steps = 5, max_steps = 30, # num_train_epochs = 1, # Set this instead of max_steps for full training runs learning_rate = 2e-4, logging_steps = 1, optim = "adamw_8bit", weight_decay = 0.001, lr_scheduler_type = "linear", seed = 3407, output_dir = "outputs", report_to = "none", # For Weights and Biases # You MUST put the below items for vision finetuning: remove_unused_columns = False, dataset_text_field = "", dataset_kwargs = {"skip_prepare_dataset": True}, max_length = 2048, ), ) This block sets up the actual fine-tuning process for the vision-language model. The model is first switched to training mode. This enables gradient computation, disables inference-only optimizations, and prepares the LoRA adapters to be updated. The data collator is a critical component for vision models. It takes each conversation, correctly aligns images with text tokens, pads sequences, and builds batches that the model can train on. Using the Unsloth-provided collator is required because standard text collators do not understand image inputs. The trainer is created using supervised fine-tuning. This training method teaches the model to produce the assistant response given the user message, exactly matching the conversation format prepared earlier. The batch size controls how many samples are processed at once on each GPU. A small value is used to stay within memory limits. Gradient accumulation simulates a larger batch size by accumulating gradients over multiple steps before updating weights. This improves stability without increasing memory usage. Warmup steps slowly increase the learning rate at the start of training, preventing sudden large updates that could destabilize the model. The training length is controlled by a fixed number of steps. This is useful for quick experiments. For full training runs, epochs can be used instead. The learning rate defines how aggressively the LoRA parameters are updated. A relatively high value is acceptable because only a small number of parameters are being trained. Logging is set to run frequently so training progress can be monitored closely. The optimizer uses an 8-bit variant of AdamW, which reduces memory usage while maintaining training quality. Weight decay applies mild regularization to reduce overfitting. The learning rate scheduler linearly decreases the learning rate over time, helping the model converge smoothly. The output directory is where checkpoints and logs are saved. External reporting is disabled to keep the setup minimal. The final group of arguments is mandatory for vision fine-tuning. Unused columns must not be removed because image data is not stored in standard text fields. The dataset text field is left empty because the trainer reads from the structured message format instead. Dataset preparation is skipped because the data is already in the correct conversational form. The maximum sequence length sets an upper bound on combined image and text tokens. After this setup, the trainer is fully configured to fine-tune Qwen to convert images into structured markdown text. trainer_stats = trainer.train() This line starts the fine-tuning process. Calling train() tells the trainer to begin updating the LoRA parameters using the prepared dataset and configuration. From this point onward, the model repeatedly sees image–instruction pairs and learns to generate the correct markdown output. During training, only the LoRA adapters are updated. The original pretrained weights remain frozen. This keeps training stable and memory-efficient. The trainer automatically handles forward passes, loss computation, backpropagation, gradient accumulation, learning-rate scheduling, and checkpointing based on the configuration defined earlier. The returned object contains training statistics such as loss values and step counts. These numbers are mainly used to confirm that training ran correctly and that the loss decreased over time. After this step completes, the model is no longer a generic pretrained model. It is now adapted specifically for extracting structured markdown text from images in the target domain. try: model.push_to_hub_merged("USERNAME/qwen-docs-finetuned", tokenizer, token = "<your hf token>") excpet: print("Please check the username and the hf token") This line uploads the final fine-tuned model to Hugging Face. The model is first merged. This means the trained LoRA adapters are permanently combined with the original base model weights. After merging, the model behaves like a normal standalone Qwen model and no longer depends on separate adapter files. The repository name defines where the model will live on Hugging Face. It becomes the public or private location from which the model can be downloaded and used later. The tokenizer is uploaded alongside the model so that anyone using it applies the exact same text and image processing logic. This is required for correct inference. The token is a Hugging Face access key that authorizes the upload. Without it, the model cannot be pushed to the account. After this step, the fine-tuned vision model is fully portable. It can be loaded directly for inference, shared with others, or used as a base for further fine-tuning. from transformers import AutoModelForVision2Seq, AutoProcessor, TextStreamer import torch from PIL import Image model_id = "USERNAME/qwen-docs-finetuned" # Load model (4-bit, fits on 16GB VRAM) model = AutoModelForVision2Seq.from_pretrained( model_id, torch_dtype=torch.float16, device_map="auto", trust_remote_code=True, load_in_4bit=True, ) processor = AutoProcessor.from_pretrained( model_id, trust_remote_code=True ) image = Image.open("/kaggle/input/python-images/images.jpg") messages = [ { "role": "user", "content": [ {"type": "image"}, {"type": "text", "text": "Convert this image to markdown format."} ] } ] text = processor.apply_chat_template( messages, tokenize=False, add_generation_prompt=True ) inputs = processor( text=[text], images=[image], return_tensors="pt" ).to("cuda") streamer = TextStreamer( processor.tokenizer, skip_prompt=True ) _ = model.generate( **inputs, streamer=streamer, max_new_tokens=1024, temperature=0.1, ) This code shows how to run inference using the fine-tuned model after it has been uploaded. The model is loaded directly from Hugging Face using the merged checkpoint. Because the LoRA adapters were merged earlier, this behaves like a normal standalone Qwen vision model. The model is loaded in 4-bit precision to keep GPU memory usage low. This allows the model to fit comfortably on a 16 GB GPU while still producing accurate outputs. Automatic device mapping places layers on the available GPU without manual configuration. The processor is loaded alongside the model. It combines the tokenizer and image processor and ensures that images and text are transformed in exactly the same way as during training. An image is loaded from a local path. This is the visual input the model will convert into markdown. The path is intentionally left as a placeholder so it can be replaced with any local image. The input is structured as a chat message. The user message contains an image placeholder and a text instruction. This matches the format used during fine-tuning, which is essential for correct behavior. The chat template converts the structured message into the exact prompt format expected by Qwen. The generation prompt tells the model that it should now produce an assistant response. The processor then converts both the image and the formatted text into tensors and moves them to the GPU. A text streamer is used to print the generated markdown token by token as the model produces it. During generation, the maximum number of tokens limits the output length. A low temperature is used to make the output more deterministic and structured, which is desirable for markdown extraction. This final step demonstrates how the fine-tuned model can be used as a drop-in image-to-markdown extractor in a real application. Ref: Unsloth fine tuning notebooks Fine-Tuning Qwen for Image-Text-to-Text Task on a Single T4 GPU Using Unsloth and TRL was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.