Abstract

Omnimodal language models (OLMs) enable unified audio-visual understanding, but processing long joint token sequences makes inference computationally prohibitive. While recent token compression methods attempt to alleviate this burden, compressing modalities in isolation often destroys the temporal cross-modal anchors necessary for coherent reasoning. We introduce Omni2LoRA, a two-stage framework for efficient parametric memory compression via coherence-preserving context distillation that bypasses the token bottleneck entirely. First, a Perceiver hypernetwork processes intermediate representations from a frozen OLM to encode the multimodal context into a full-rank Low-Rank Adaptation (LoRA) adapter in a single forward pass. To prevent the resulting parameter footprint from scaling linearly with recording length, we optimize a discrete rank allocation policy via Group Relative Policy Optimization (GRPO) that uses a modality-ablated counterfactual reward to explicitly penalize the loss of audio-visual coherence, forcing the model to allocate its fixed sub-linear rank budget to synergistic cross-modal anchors rather than isolated visual features.

Across three omnimodal backbones (Qwen2.5-Omni-3B/7B, InteractiveOmni-4B), Omni2LoRA operating at a 30% rank budget outperforms direct full-context inference and strong token-compression baselines (OmniZip, OMAC, O-MARC) on four audio-visual question answering benchmarks, improving average accuracy by 8-12% over the strongest baseline and remaining stable under compression ratios as tight as 75%, where token-pruning methods degrade sharply. By converting multimodal memory into a fixed-budget, reusable parameter state, our method drives answer-time multimodal-token load to zero, cutting per-query Time to First Token (TTFT) by up to 12x relative to full-context inference and amortizing to under 0.5s after a handful of queries, establishing a robust paradigm for long-context omnimodal memory compression.

The problem

Omni models reason jointly over what is seen and what is heard, which matters for real-world video where speech, ambient sound, scene transitions, and visual actions provide complementary evidence. But video frames and audio waveforms are both token-intensive, so their joint stream inflates context length, memory, and latency, and pushing a model past its capacity leads to lost cross-modal alignment or incoherent repetition.

Token compression methods prune or merge this stream, but compressing audio and video in isolation can remove exactly the evidence that joint reasoning depends on. A sound is only meaningful once grounded in the visible scene, and a visual event may stay ambiguous without its audio. Naively moving the recording into parameter space does not solve this either: an adapter generated for every temporal chunk grows linearly with recording length, and under a fixed budget the visually dominant features crowd out the brief acoustic anchors.

Approach

Omni2LoRA extends the parametric internalization idea of Frames2LoRA from silent video to synchronized audio-visual recordings, and adds a learned compression stage so the adapter footprint stays fixed regardless of recording length.

Overview of Omni2LoRA. Stage 1: a frozen omni encoder produces layer-wise hidden states for each temporal chunk; a Perceiver hypernetwork maps them to a full-rank bank of candidate LoRA slots, trained with cross-entropy against teacher captions. Stage 2: a scoring network selects a fixed-budget subset of slots by sequential without-replacement sampling and is optimized with GRPO, where the advantage is reshaped using joint, visual-only, and audio-only reference rollouts so that allocations which drop cross-modal evidence are penalized.
  • Stage 1, full-rank hypernetwork training. The recording is split into non-overlapping temporal chunks. For each chunk, the frozen OLM encoder produces layer-wise hidden states, and a hierarchical Perceiver hypernetwork maps them to candidate LoRA factors for every target module, layer, and rank direction. The hypernetwork is trained with teacher-forced cross-entropy against cached teacher captions on VALOR-1M, with the encoder and answer model frozen. Once converged, it is frozen and treated as a deterministic mapping from a recording to a bank of rank-one candidate updates, called slots.
  • Stage 2, memory-augmented compression distillation. A lightweight scoring network, conditioned on each chunk’s pooled audio-visual summary and a positional embedding of (layer, module, rank), assigns a logit to every slot. Allocations under a fixed total rank budget are sampled sequentially without replacement, which keeps clipped importance-ratio training well defined. The policy is trained on FineVideo with a PPO-style clipped, KL-regularized objective; only the scoring network is updated.
  • Coherence-aware advantage shaping. For each training query, the frozen answer model is scored under three full-token references: joint audio-visual, visual-only (audio masked), and audio-only (frames masked). The Audio-Visual Dependence Score is the gap between the joint reference and the best unimodal one, isolating queries that need both modalities. Each sampled allocation’s degradation relative to the joint reference is amplified by this dependence score and subtracted from the standard group-relative GRPO advantage, so allocations that score well by luck while dropping cross-modal rank directions are penalized.
  • Inference. The recording is processed once: the frozen hypernetwork builds the slot bank, the scoring network greedily keeps the top-budget slots, and the compact adapter is reused for every subsequent query. The frozen backbone answers with zero audio or visual tokens in its active context.
Audio-visual recording temporal chunks Frozen omni LM encodes once layer-wise states Perceiver hypernetwork Stage 1: trained with CE, then frozen Full-rank slot bank rank-1 updates per layer/module Scoring policy keeps 30% budget Stage 2: GRPO, only part trained coherence-aware reward: joint vs audio-only vs visual-only rollouts Frozen OLM + adapter 0 audio or video tokens in context Questions Answers Internalize once: recording → full-rank LoRA memory then compress to a fixed rank budget that does not grow with recording length
Stage 1 turns a recording into a full-rank bank of rank-one LoRA slots through a frozen omni model and a Perceiver hypernetwork; Stage 2 trains a scoring policy with GRPO to keep a fixed 30% rank budget, using joint, audio-only, and visual-only reference rollouts so that slots carrying cross-modal evidence survive, and the compact adapter answers every later question with zero audio or video tokens in context.

Results

Evaluation covers three backbones (Qwen2.5-Omni-3B, InteractiveOmni-4B, Qwen2.5-Omni-7B) on UGC-AVQA, WorldSense, OmniVideoBench, and DailyOmni for accuracy, and VidCapBench for efficiency. Baselines are direct full-context inference plus the token-compression methods OmniZip, OMAC, and O-MARC, all at the same 32-frame setting and 30% retained ratio.

Qwen2.5-Omni-7B, 32 frames Retained DailyOmni UGC-AVQA OmniVideo WorldSense Average
Full tokens 100% 56.3 54.1 34.6 43.6 47.2
OmniZip 30% 51.8 52.7 30.0 39.6 43.5
OMAC 30% 53.6 53.2 30.9 42.4 45.0
O-MARC 30% 60.4 64.6 35.2 44.0 51.1
Omni2LoRA 30% 63.6 68.0 36.6 45.8 53.2

Source: Table 1 of the paper. DailyOmni, UGC-AVQA (overall), and WorldSense report accuracy (%); OmniVideo reports average score. Omni2LoRA's gains over O-MARC on DailyOmni, UGC-AVQA, OmniVideo, WorldSense, and the average are significant at p < 0.05 (Wilcoxon signed-rank). Higher is better.

Four-benchmark average Full tokens OmniZip OMAC O-MARC Omni2LoRA
Qwen2.5-Omni-3B 44.1 41.0 42.8 45.8 47.3
InteractiveOmni-4B 46.0 42.6 44.1 45.8 47.6
Qwen2.5-Omni-7B 47.2 43.5 45.0 51.1 53.2

Source: Table 1 of the paper, average column for each backbone. Compression baselines and Omni2LoRA use a 30% retained ratio. Higher is better.

  • Cross-modal coherence. Gains are largest on UGC-AVQA, which strictly requires joint acoustic and visual evidence: 68.0% overall on the 7B backbone versus 64.6 for O-MARC and 54.1 for full tokens, with improvements on all four of its categories (event progression 68.2, scene or temporal transition 70.4, cross-scene alignment 63.5, fine-grained contrast 70.2).
  • Allocation ablation. The uncompressed full-rank adapter reaches near parity with direct audio-visual inference (for example 50.3 vs 48.9 on UGC-AVQA at 3B). Under a 30% budget, uniform allocation and Frobenius-norm-scored allocation degrade sharply (44.9 and 47.9 on UGC-AVQA at 3B, versus 61.7 for the learned O2L-GRPO policy).
  • Compression ratio and long recordings. On UGC-AVQA, Omni2LoRA holds 60.7 accuracy at 75% compression while OmniZip falls to 47.1, OMAC to 49.1, and O-MARC to 56.3. Sweeping 8 to 1,024 frames, the direct baseline collapses to 22.0 at 1,024 frames from context exhaustion, while Omni2LoRA improves monotonically to 46.2.
  • Efficiency. On VidCapBench with the 7B backbone, single-question TTFT averages 0.49s including internalization, versus 6.03s for full-context inference and 3.45s for O-MARC. Amortized over repeated questions, latency falls to 0.82s per query (7B) and 0.72s (3B) after five queries and plateaus near 0.43s, while O-MARC stays at 3.6s to 4.2s.

Evaluation benchmarks

Benchmark Videos Queries Metric
UGC-AVQA 206 1,648 Accuracy (%)
WorldSense 1,662 3,172 Accuracy (%)
OmniVideoBench 628 1,000 Average score
DailyOmni 684 1,197 Accuracy (%)
VidCapBench 643 9,494 TTFT (s)

Source: Table 4 of the paper. UGC-AVQA figures are for the difficulty-filtered benchmark split; VidCapBench counts refer to the automatically-assessable subset. Stage 1 trains on VALOR-1M (about 1M clips) and Stage 2 on FineVideo (43,751 videos, about 219K queries).

Resources