Highlight-guided Attention Steering

Don't drop the video.
Steer the attention.

HAS keeps the full video stream available to a frozen multimodal LLM, turns a prompt-conditioned highlight distribution into a token-level steering vector, and softly biases selected visual cross-attention heads during decoding.

Rui Chu · Yingjie LaoTufts UniversityFrozen backboneECCV 2026
One video, two ways to summarize
Toggle the mechanism. The underlying video never changes.
continuous highlight prior h
highlight priormodel attentionevidence removed by hard selection
All frames remain accessible; emphasis changes continuously.
01 / Background

The bottleneck is not finding importance. It is what happens next.

Many summarization pipelines turn importance into a discrete subset. HAS instead treats the highlight signal as an internal attention prior, so low-scored context can remain available.

Information bottleneck lab
Reduce the selection budget and watch a bridging moment disappear on the left.
5 / 12
Score → hard select → summarizediscrete
Bridge context: object handoff between two high-score momentsunavailable
Full video → soft steering → summarizecontinuous
Bridge context: still present even when assigned low highlightavailable
Visual rule used throughout this page: low highlight does not mean deleted. Under HAS, low-scored frames can be deemphasized, but they remain part of the video evidence.
02 / Method I

Build a prompt-conditioned temporal prior.

An off-the-shelf highlight generator produces raw temporal scores. HAS aligns them to the video length, min-max normalizes them, then vectorizes the result for attention steering.

Highlight Prior Lab
The same video, different prompt, different temporal emphasis.
raw scores ĥ
Raw output of H(F, q). Press “Build prior” to step through alignment and normalization.
ĥ ← H(F, q)generator
ĥ ← Interp(ĥ, T)time align
h ← MinMaxNorm(ĥ)[0, 1]
V ← Vectorize(h)token prior
Active Algorithm 1

This visualization intentionally does not insert a Gaussian/TV smoothing optimizer: those variants are not part of the active camera-ready algorithm.

03 / Method II

Lift one frame score into visual-token space.

The MLLM attends over visual tokens, not scalar frame scores. HAS copies each frame’s highlight value across its P visual tokens and moves the prior into log-space.

Choose a frame
Click any frame in the strip.
t = 7
Frame-to-token microscope
Every patch/tubelet token from frame t receives the same temporal prior hₜ.
Repeat(h, P)
V = log( Repeat(h, P) + ε )RTP
04 / Method III

Steer selected cross-attention heads, not the whole model.

At decoding time, HAS adds a gated, head-specific logit bias to pre-softmax visual attention. Drag the controls and watch the same attention matrix redistribute mass across time.

Original attention A(ℓ,m)baseline
Steering vectorg β V
token-aligned prior
HAS attention Ã(ℓ,m)steered
0.95
0.80
Inference update
Ãi,:(ℓ,m) = Ai,:(ℓ,m) + gℓ,m βℓ,m V

At β = 0 or g = 0, the right heatmap returns to the baseline attention pattern.

Policy calibration
mina,β LNLL(a,β) + λs Σ gℓ,m

The backbone MLLM stays frozen. Gate and strength parameters are calibrated once on held-out data, then fixed for forward-only test-time steering.

05 / Method recap

One pipeline. No missing transformation.

Click any stage to jump back to the corresponding interactive explanation.

→→→→→→→→
“The video stays intact. What changes is how much attention the model allocates to each moment.”
01
Continuous evidence

Keep the full temporal stream instead of converting importance into a hard deletion decision.

02
Token-aligned control

Turn frame-level highlight values into a bias that matches the MLLM’s visual token space.

03
Sparse intervention

Steer only selected visual cross-attention heads while leaving the backbone parameters frozen.