Take one clip from CaReBench. A man in a black tank top sits by a window, brick wall behind him, wooden door to his right. He flicks a lighter, brings the flame to a pipe, and draws on it. Ask SmolVLM2 to describe the clip and it first turns each of the 12 sampled frames into hundreds of visual tokens, then reads your question, then answers. Ask a second question and it does all of that again, because the frames have to sit in the context window every time.

A few dozen frames is already tens of thousands of tokens before you have typed a word, and past the model’s capacity it does not fail gently: it starts emitting repetitive text that has nothing to do with the video. Frames2LoRA, which I worked on with Sarvesh Baskar and Dinesh Manocha at UMD, takes the video out of the context entirely. It reads the frames once, writes what it saw into the model’s weights as a small adapter, and from then on every question is answered from text alone.

The idea in one picture

Video in context (base model) 12 frames context window … query hundreds of visual tokens per frame Frozen SmolVLM2 Answer Every question re-encodes the video tens of thousands of tokens for a few dozen frames Frames2LoRA 12 frames Perceiver hypernetwork LoRA adapter A, B · rank 16 one forward pass, once per video context window query 0 visual tokens Frozen SmolVLM2 + adapter Answer internalize once, ask as many times as you like
Left: the base model keeps every frame in its context and pays for it on every question. Right: Frames2LoRA generates a LoRA adapter from the frames once; the frozen model plus adapter answers with zero visual tokens in context.
Key ideaLoRA adapters are normally trained with gradient descent. Frames2LoRA predicts one instead: a hypernetwork (a network whose output is another network's weights) looks at the video once and emits the adapter in a single forward pass. Attach it to the frozen VLM and the model "knows" the video without a single visual token in its prompt.

Doc-to-LoRA showed this works for text documents. Video is harder in three ways. The token volume per example is orders of magnitude larger, the compression is cross-modal (visual content has to become perturbations to a language model’s weights), and video varies along frame count and resolution, axes that text does not have. The rest of this post follows the pipe-lighting clip through the method, one stage at a time.

Walkthrough: one CaReBench clip, from frames to an answer with no frames

Walkthrough: internalizing the pipe-lighting clip and asking it a question

1. Sample the frames and add an instruction

CaReBench clip: a man lights a pipe by a window 12 frames sampled uniformly, longest edge 384 px f1 f2 f3 f4 f5 f6 f7 f8 f9 f10 f11 f12 lighter flicks on → flame meets the pipe → he sets the lighter down and smokes Internalization instruction i a fixed text prompt that tells the encoder to take the video in; not the user's question video v = the 12 frames. The user's question p does not enter yet. Visual tokens in context when the question is asked Base model 12 × hundreds every frame, every question Frames2LoRA 0 frames seen once, by the encoder

We sample 12 frames uniformly from the clip, longest edge 384 px, and pair them with a fixed internalization instruction (a prompt that tells the encoder to take the video in; it is not the user’s question). For the base model, these 12 frames are the expensive part: each becomes hundreds of visual tokens, and all of them ride along with every question. In Frames2LoRA they will be seen exactly once, by the encoder in the next step.

f1f2f3…f12+internalization instruction

2. Run the frozen encoder and keep every layer's hidden states

12 frames v instruction i Frozen SmolVLM2 E vision encoder + text decoder layer 0 layer 1 layer 2 ⋮ layer L − 1 h₀ S tokens × D dims text-side states after layer 0 h_L−1 C = stack(h₀ … h_L−1) shape L × S × D one slice per layer, not one pooled vector no weights change here; hidden states are simply read out

The frames and instruction go through a frozen SmolVLM2. Instead of keeping only the final output, we keep the text-side hidden states from every transformer layer and stack them into a tensor \(\mathbf{C}\) of shape \(L \times S \times D\): one slice \(\mathbf{h}_\ell\) per layer, each with \(S\) tokens of \(D\) dimensions. Keeping the layer axis is deliberate. It lets the next stage write a different adapter for each layer instead of squeezing the whole video into one pooled vector and reusing it everywhere.

3. The Perceiver hypernetwork reads each layer slice

Layer slice h_ℓ S tokens × D dims one token per row Encoder resampler learned latent queries cross-attend to h_ℓ → fixed-size summary latent size Z = 512 Decoder resampler one output query per (target module m, rank r) M = 1 module, R = 16 ranks 16 rank latents for layer ℓ module m = MLP down_proj r = 1 r = 2 r = 3 r = 4 r = 5 r = 6 r = 7 r = 8 r = 9 r = 10 r = 11 r = 12 r = 13 r = 14 r = 15 r = 16 each latent is a vector of size Z = 512 full output O ∈ R^(L × M × R × Z) repeated for every layer ℓ = 0 … L − 1 still no weights, just latents

This is the only trained component. For each layer slice, an encoder resampler lets a set of learned latent queries (latent size 512) cross-attend to the \(S\) hidden states, producing a fixed-size summary no matter how many frames went in. A decoder resampler then asks that summary one question per target module and per LoRA rank direction. With one target module (the MLP down-projection) and rank 16, that is 16 rank latents for this layer. The same thing happens for every layer, so the output is a tensor of shape \(L \times M \times R \times Z\): still latents, not weights.

4. Latents become LoRA factors and attach to the frozen model

16 rank latents layer ℓ, from step 3 ⋮ r = 1 … 16 Shared projection head same head for all layers A_ℓ (16 × d_in) learned scale, starts at 1 B_ℓ (16 × d_out) learned scale, starts at 0 one row of each per rank latent B at zero ⇒ untrained adapter is a no-op Layer ℓ, MLP down_proj frozen weight W_ℓ (d_out × d_in) y = x W_ℓᵀ + s · (x A_ℓᵀ) B_ℓ a rank-16 update ΔW_ℓ = s · B_ℓᵀ A_ℓ on top of weights that never change specific to this video this is the adapter θ(v): {A_ℓ, B_ℓ} for every layer

A shared projection head turns each rank latent into one row of \(\mathbf{A}_\ell\) and one row of \(\mathbf{B}_\ell\), giving rank-16 factors for the layer. Learned multipliers scale them; the \(\mathbf{B}\) scale is initialized to zero, so before training the adapter is a null perturbation and the model behaves exactly like the base. The factors are added to the frozen down-projection of that layer as a standard LoRA update. The full set, \(\theta(v) = \{\mathbf{A}_\ell, \mathbf{B}_\ell\}_{\ell}\), is the adapter for this one video. Notice what happened to the token budget along the way:

f1f2…f12→A_0, B_0A_1, B_1…A_L−1, B_L−1→0 visual tokens in context

5. Ask the question with zero visual tokens

Base model, video in context context window 12 frames × hundreds of visual tokens each "Describe the video in as much useful visual detail…" Frozen SmolVLM2 "…holding a lighter and a piece of paper. He is blowing on the paper…" token-F1 0.32 Frames2LoRA, adapter attached context window 0 visual tokens (the frames were internalized in steps 1–4) "Describe the video in as much useful visual detail…" Frozen SmolVLM2 + θ(v) "…black tank top… tattoos… using their left hand to light it with a lighter." token-F1 0.56 same model weights, same prompt, same decoding; only the video's route into the model differs

Now the CaReBench caption prompt goes in: “Describe the video in as much useful visual detail as possible. Include the main activity, visible people or objects, scene context, appearance, and any important visual details that help explain what is happening.” Same frozen SmolVLM2, same prompt, same decoding in both rows. The base model, with all 12 frames in context, gets the room right but misreads the action: “He is holding a lighter and a piece of paper. He is blowing on the paper and then putting it in his mouth.” (token-F1 0.32 against the reference). Frames2LoRA, with nothing but the prompt in context, answers: “A person is smoking a cigarette in a room with a brick wall and a wooden door. The person is wearing a black tank top and has tattoos on their arms. They are holding the cigarette in their right hand and using their left hand to light it with a lighter.” (token-F1 0.56). It calls the pipe a cigarette, but it has the tank top, the tattoos, the lighter and the act of lighting.

6. Keep the adapter and ask again

θ(v) generated once Frozen SmolVLM2 + θ(v) Q1: What is the weather like? Q2: second question Q3: third question ⋮ Q15: fifteenth question VidCapBench: 15.23 questions per video each one: text prompt only, 0 visual tokens Average TTFT per question, 2.2B, internalization time included 7.06 s base, video in context 0.58 s Frames2LoRA, adapter reused 500M: 6.45 s → 0.55 s

The adapter does not expire after one answer. Every later question about the same video is a text-only prompt through the same adapted model, and the frames are never re-encoded. VidCapBench is the natural place to measure this, because each video comes with 15.23 questions on average. Averaged over all 1,523 queries, and charging Frames2LoRA for the one-time internalization, time to first token per question drops from 7.06 s to 0.58 s at 2.2B and from 6.45 s to 0.55 s at 500M. Amortized over the first 5 questions it is already 1.44 s per question at 2.2B; after 10 it is 0.80 s.

Under the hood

Symbol Meaning
\(v\), \(i\), \(p\), \(y\) video, internalization instruction, downstream text prompt, response
\(E\), \(F\) frozen SmolVLM2 used as video encoder and as answer model (same weights)
\(H_\phi\) the Perceiver hypernetwork; \(\phi\) are the only trained parameters
\(\mathbf{h}_\ell\), \(\mathbf{C}\) text-side hidden states after layer \(\ell\); their stack, \(L \times S \times D\)
\(L\), \(S\), \(D\) number of layers, fused sequence length, hidden dimension
\(M\), \(R\), \(Z\) target modules per layer (1: MLP down_proj), LoRA rank (16), latent size (512)
\(\mathbf{A}_{\ell,m} \in \mathbb{R}^{R \times d_{\mathrm{in}}}\), \(\mathbf{B}_{\ell,m} \in \mathbb{R}^{R \times d_{\mathrm{out}}}\) generated LoRA factors for layer \(\ell\), module \(m\)
\(\theta(v)\) the generated adapter: all \(\mathbf{A}\), \(\mathbf{B}\) factors for video \(v\)
\(s\) fixed LoRA scaling factor

The whole method is three lines. The encoder produces video-conditioned states, the hypernetwork maps them to an adapter, and the answer model conditions on the prompt and the adapter but never on the video tokens:

\[\mathbf{C} = E(v, i), \qquad \theta(v) = H_\phi(\mathbf{C}), \qquad p_\phi(y \mid p, v) = F\big(y \mid p;\, \theta(v)\big).\]

Inside a frozen linear layer with weight \(\mathbf{W} \in \mathbb{R}^{d_{\mathrm{out}} \times d_{\mathrm{in}}}\), the generated factors act as an ordinary LoRA update. In the row-vector convention the layer computes \(\mathbf{x}\mathbf{W}^\top\) and the adapter adds a rank-\(R\) term, which is the same as perturbing the weight by \(\Delta\mathbf{W} = s\,\mathbf{B}^\top\mathbf{A}\):

\[\mathbf{y} = \mathbf{x}\mathbf{W}^\top + s\,(\mathbf{x}\mathbf{A}_{\ell,m}^\top)\,\mathbf{B}_{\ell,m}.\]

Training is teacher-forced cross-entropy over response tokens. A frozen SmolVLM2 teacher that does see the frames writes captions and summaries offline; the student answer model has to reproduce them from the prompt and the adapter alone, and the gradient flows only into \(\phi\):

\[\mathcal{L}(\phi) = -\sum_t \log p_\phi\big(y_t \mid y_{<t},\, p;\, \theta(v)\big).\]
video v + instruction i Encoder E frozen SmolVLM2 C Hypernetwork H_φ per layer ℓ: encoder resampler → decoder resampler → shared projection head the only trained part θ(v) Answer model F same frozen SmolVLM2 + {A_ℓ, B_ℓ} on down_proj; no visual tokens text prompt p output y token by token Frozen teacher same SmolVLM2, sees the frames Cross-entropy loss teacher caption vs. y, teacher-forced training: gradients update φ only; E, F and the teacher stay frozen. At inference the teacher and the loss disappear.
The full loop. Only the hypernetwork receives gradients; the encoder, the answer model and the teacher are the same frozen SmolVLM2.

Two details in the hypernetwork matter more than they look. First, the Perceiver bottleneck is what makes frame count a free variable: the latent queries produce a fixed-size summary whether \(S\) covers 8 frames or 1,024, which is why a model trained only at 12 frames and 384 px can be run at 1,024 frames and 1024 px. Second, the zero-initialized \(\mathbf{B}\) scale means training starts from the base model’s behavior and learns a perturbation, rather than starting from a random adapter that has to be unlearned. Training data are spans from FineVideo, mixed 60/30/10 across single-scene, adjacent multi-scene and full-video spans, with audio excluded.

The finding I did not expect is that adapters compose in rank space. Split a video into two temporal halves, internalize each independently, and concatenate the two rank-16 adapters along the rank dimension. Nothing in training ever saw a composed adapter, yet on VDC the composed adapter keeps 93.1% of the single-video adapter’s mean token-F1 at 500M (0.206 vs 0.221) and 86.2% at 2.2B (0.211 vs 0.245), and it produces coherent video-level captions rather than text tied to one half.

first half→θ(v₁), rank 16⊕second half→θ(v₂), rank 16=composed adapter, rank 32

The rank directions are redundant but not interchangeable. Ranking each rank slice by the product of its factor norms and keeping only the top 8 gives 0.1264 token-F1 on ActivityNet Captions, against 0.1262 for the full rank-16 adapter, while the lowest-scoring single slice lands below the zero-adapter baseline. The ordering is the same in every one of 500 examples (direction R11 always scores highest), which suggests the hypernetwork has learned a fixed coordinate system for its output. Layer-wise removal on the 2.2B model adds that the updates whose removal hurts most sit in the later layers, close to the output logits.

What the numbers say

  • Quality holds. On all five captioning benchmarks (ActivityNet Captions, PLM-RDCap, PLM-RCap, VDC, CaReBench), at both 500M and 2.2B, Frames2LoRA is statistically non-inferior and equivalent to video-in-context inference under an LLM judge (Spearman 0.823 with human ratings), recovering 91.9% of the base judge score at 2.2B and 84.2% at 500M. Video QA was never trained on, yet 7 of 8 benchmark-scale pairings pass, and on NExT-QA the adapter beats the base at both scales.
  • Queries get cheap. Across a sweep of 8 to 1,024 frames and 224 to 1024 px, query TTFT falls by a geometric mean of 6.7x at 500M and 20.1x at 2.2B (maximum 79.1x), and answer-time input tokens fall by 150x and 302x on average, reaching 713x and 1,507x.
  • It survives where in-context inference does not. Trained at 12 frames, the model stays stable through 1,024 frames and 1024 px (average token-F1 change of -0.012 at 500M). At 1024 px and high frame counts direct inference degenerates into repetitive output and Frames2LoRA leads by +0.12 to +0.13 token-F1.

Try it