Frames2LoRA: Parametric Video Internalization for Vision-Language Models

Abstract

Formerly titled Video2LoRA (arXiv v1).

Processing video in vision-language models is expensive: each frame occupies hundreds of tokens, and inference cost scales with every frame and every repeated query. We introduce Frames2LoRA, a method for parametric video internalization. A perceiver hypernetwork reads the intermediate representations produced layer-by-layer as a frozen VLM encodes a video, and generates a Low-Rank Adaptation (LoRA) adapter in a single forward pass. Unlike standard LoRA fine-tuning, which requires iterative gradient updates, Frames2LoRA predicts these weights directly from the video.

Trained for SmolVLM2 500M and 2.2B on video summarization and captioning, Frames2LoRA enables the same frozen VLM to answer queries from the adapter alone, with zero visual tokens in its context at query time. Frames2LoRA is statistically non-inferior and equivalent to direct video-in-context inference across all five captioning benchmarks at both model scales, and across seven of eight video question answering benchmark-scale pairings. Although trained only on 12 frames at 384px, it remains stable up to 1,024 frames and 1024px, where direct video-in-context inference often degenerates. Across this sweep, it reduces answer-time visual-token load by up to 1,500x and query TTFT by 6-80x, while preserving video-faithful outputs. We also find that independently generated adapters for non-overlapping video segments can compose in rank space, suggesting a path toward chunked long-video internalization.

The problem

Video understanding in VLMs is built on a token-heavy abstraction: frames are encoded as visual tokens and concatenated into the model’s context window. Each frame at standard resolution contributes hundreds of tokens, so a clip of a few dozen frames produces tens of thousands of tokens before any text query is added. Past a capacity threshold the model does not degrade gracefully; it produces repetitive or incoherent text unrelated to the video.

Frame subsampling, visual token compression, long-context architectures, and streaming memory all reduce this burden without removing it. Visual tokens remain in context at query time, every query re-pays the encoding cost, and every approach eventually hits the same ceiling. We take a different route: rather than fitting more video into the context window, we remove it from the query entirely by storing the video in the model’s parameters before any query is issued.

Approach

Frames2LoRA converts a video into a video-specific LoRA adapter in a single forward pass. Both the video encoder and the answer model are frozen copies of the same SmolVLM2 backbone; only the hypernetwork is trained.

• Video encoder. A frozen SmolVLM2 encodes the sampled frames together with an internalization instruction. We keep the text-side hidden states from every transformer layer, giving a stack of layer-indexed video-conditioned states rather than a single pooled vector.
• Perceiver hypernetwork. For each layer slice, an encoder resampler attends from learned latent queries to the hidden states, and a decoder resampler emits one latent per target module and LoRA rank direction. A shared projection head maps each rank latent to the LoRA factors A and B. The factors are scaled by learned multipliers, with the B scale initialized to zero so the adapter starts as a null perturbation.
• Dynamic LoRA injection. The generated factors are added to the frozen linear layers using the standard LoRA update. Each video receives its own adapter; in our experiments the adapters target the MLP down-projection modules of the text decoder with rank 16.
• Training objective. The hypernetwork is trained with teacher-forced cross-entropy over response tokens. Targets are cached generations from a frozen SmolVLM2 teacher that saw the video frames; the student answer model sees only the text prompt plus the generated adapter. Training uses 12 uniformly sampled frames at 384px, on single-scene, adjacent multi-scene, and full-video spans from FineVideo mixed 60/30/10. Audio is excluded.

Example

Two real evaluation outputs from the project page, comparing the base SmolVLM2 prompted with the video in context against Frames2LoRA answering from the generated adapter alone (no video tokens). Token-F1 is against the benchmark reference.

Both answers are video-faithful; the adapter-only answer is a little more verbose, which is the pattern the paper reports for token-F1 on short-answer QA.

Results

All comparisons use the same videos, prompts, references, frame sampling, and decoding for the video-in-context baseline and Frames2LoRA. Quality is measured with token-level F1 and a Qwen3-30B LLM judge (Spearman 0.823 with human ratings on 200 examples), with paired bootstrap 95% confidence intervals and formal non-inferiority (NI, margin 0.15 on the rescaled judge score) and equivalence (Eq) tests.

Source: Table 1 of the paper (LLM-judge score, rescaled to [0, 1]). Eq / NI list the equivalence and non-inferiority outcomes, which are identical at both scales. Higher is better. The same table's token-F1 averages are 0.243 vs 0.242 (500M) and 0.250 vs 0.250 (2.2B).

Source: Table 4 of the paper (LLM-judge score). Frames2LoRA is trained only on captioning; QA is zero-shot. Higher is better.

• Captioning. Frames2LoRA passes both non-inferiority and equivalence on all 10 benchmark-scale combinations under the LLM judge and all 10 under token-F1. It recovers 91.9% of the base model’s judge score at 2.2B and 84.2% at 500M; on PLM-RCap at 500M it beats the base by +0.026 token-F1 (+14.7%).
• Video QA, zero-shot. 7 of 8 benchmark-scale pairings pass the judge tests; on NExT-QA it surpasses the base at both scales. The one failure is PLM-SGQA at 2.2B.
• Frame and resolution generalization. Sweeping 8 to 1,024 frames and 224 to 1024px on VDC background captioning, Frames2LoRA stays stable everywhere (average token-F1 change of -0.012 at 500M) and beats the base by +0.12 to +0.13 token-F1 at 1024px and high frame counts, where direct inference degenerates into repetitive output.
• Efficiency. Query TTFT drops by a geometric mean of 6.7x (500M) and 20.1x (2.2B), with maxima of 17.2x and 79.1x; answer-time input tokens fall by 150x and 302x on average, reaching 713x and 1,507x. On VidCapBench (100 videos, 1,523 queries), average TTFT including the one-time internalization falls from 6.45s to 0.55s at 500M and 7.06s to 0.58s at 2.2B.
• Chunk composition. Internalizing two temporal halves independently and concatenating the LoRA ranks retains 93.1% (500M) and 86.2% (2.2B) of the single-video adapter’s token-F1 on VDC, with no training for composition.
• Where it is weakest. Camera-style captions in VDC recover only 42.3% of the base judge score at 500M, rising to 82.0% at 2.2B.

Resources

• arXiv: 2606.04351 and PDF
• Project page with a qualitative-example explorer
• Code on GitHub
• SmolVLM2 checkpoints on Hugging Face (500M and 2.2B)
• Hugging Face Papers page
• Explainer post on this site
• Follow-up work extending the idea to audio-visual context: Omni2LoRA
• UMD CS coverage: Teaching AI to See Faster
• All publications

Quick start

From the repository README: install with uv, download a checkpoint, and run inference over a JSONL manifest with one row per video.

git clone https://github.com/frames2lora/Frames2LoRA.git
cd Frames2LoRA
uv sync

uv run huggingface-cli download MananSuri27/Frames2LoRA-SmolVLM-ckpts \
  --local-dir checkpoints/Frames2LoRA-SmolVLM-ckpts

# manifest.jsonl, one line per video:
# {"id":"sample-0001","video_path":"/path/to/video.mp4","prompt":"Describe what is happening in this video.","task_type":"caption"}

uv run python -m scripts.frames2lora.infer \
  --checkpoint checkpoints/Frames2LoRA-SmolVLM-ckpts/frames2lora-smolvlm2-500m-best-ce.pt \
  --manifest /path/to/manifest.jsonl \
  --output outputs/tiny_generations.jsonl