Traffic signs from 0.4% of the input 90.4% top-1 ›
How little does a model need to see?
LookThere trains a vision transformer to process only the patches a task needs. A shallow selector views a low-resolution copy of the image and chooses which high-resolution patches to keep. A deep extractor then processes only those patches. Both are trained together with reinforcement learning.
Vision transformers typically treat every image token as equally important, yet for most tasks in computer vision only a fraction are needed. Adaptive computation methods accelerate inference by choosing which tokens to process, but existing methods struggle at extreme sparsity and require heuristics that may not generalize like token diversity and attention scores. We address these limitations with LookThere, achieving a new pareto frontier in performance-compute trade-offs through an end-to-end reinforcement learning framework that jointly trains a shallow input selector and a deep representation extractor. The selector learns where to look and the extractor learns what to see, together saving computation by selecting only what is worth processing for a given task without relying on auxiliary signals. We show that LookThere only selects the task-specific input, excelling at sparse recognition in high-resolution settings (traffic signs, billiards), and maintaining accuracy with as little as 0.2% of the input. It generalizes across tasks and models, including global recognition (ImageNet classification), local recognition (ADE20K segmentation), zero-shot classification (by distillation), and regression (counting). Across all settings, LookThere surpasses state-of-the-art selection to provide a general and scalable framework for specialized and efficient adaptive computation.
Interesting is not the same as important. Attention highlights what is salient, not what the task needs.
Each image shows the 10% of patches ranked highest by DINOv2 attention and by LookThere. Self-supervised models such as DINOv2 learn what is visually interesting without ever seeing a task. On the bird, DINOv2 attends to the surrounding foliage. On the street, it attends to cars and lamps rather than the sign. LookThere is trained on the task itself, so it selects the patches that the task depends on.
Birdsnap
DINOv2 attentiontop 10%
LookTheretop 10%
Traffic signs
DINOv2 attentiontop 10%
LookTheretop 10%
Select first, then compute. The selector chooses patches before the extractor runs, so dropped patches are never processed.
input
selector
policy
extractor
Two modelsThe selector is a 3-layer transformer that views the image at 154 px and predicts a keep probability for every patch. The extractor is a ViT-B/14 that processes only the kept patches at full resolution.
Coarse to fineThe policy samples keep-or-drop decisions for regions on a 6×6 grid, then on an 18×18 grid, then for individual patches on the 37×37 grid. When a region is dropped, the patches inside it are likely to be dropped too, so large irrelevant areas are removed together.
LookThereLookWhere
Any backboneImageNet top-1 accuracy at about 90% sparsity, with both models initialized from each backbone.
Traffic signsTop-1 accuracy (%) vs sparsity (%) · 1008 px
LookThereLookWhereDense
BilliardsTop-1 accuracy (%) vs sparsity (%) · 1008 px
LookThereLookWhereDense
Birdsnap, 518 px: accuracy against inference GFLOPs
The task determines where it looks. One table, three questions and three different selections.
The Billiard Balls benchmark asks three questions about the same image. A LookThere model is trained for each question, and each one learns to select only the balls its question depends on. LookWhere distills its selector from DINOv2 attention, so it selects the same patches whatever the question.
5 of 5,184 patches
1 / 4
One selector for any budget. Choose how many patches to keep at inference time.
LookThere can also be conditioned on a target number of patches. During training, it samples a budget for each image and learns an embedding for each budget level. A Lagrange multiplier for each level keeps the selection close to its target. Below, LookWhere keeps the same number of patches as LookThere. On these ImageNet images, LookThere places its patches on the object, while LookWhere places many of them along the image border. Across ImageNet, conditioned LookThere is competitive with LookWhere and outperforms it at higher sparsity.
LookThere65 of 1,369 patches
LookWhere65 of 1,369 patches
Sparse inference in a few lines. Code, configs and checkpoints for the paper's runs.
Load a checkpoint, sample a mask with the selector and run the extractor on the kept patches.
1Warm upThe extractor first learns the task from all patches, so it can give the selector useful feedback.
2Fine-tune with RLThe selector samples 8 masks per image. Each mask is rewarded for a low task loss and for sparsity. GRPO compares the rewards within each group and makes the better masks more likely.
3EvaluateMasks are sampled at the final training temperature, or chosen deterministically by keeping every patch with a probability above 0.5.
inference.pytrain.sh
from modeling import Extractor, Selector, mask_to_padded_indices
ckpt = torch.load("billiards/max-left-right/sparsity90/model.pt")
selector = Selector(cfg["img_size"], cfg["grid_stages"])
extractor = Extractor(cfg["img_size"], num_classes)
with torch.no_grad():
logits, prefix = selector(image) # (1, num_patches)
keep, _, _ = selector.sample(logits, cfg["grid_stages"], temp)
idx, pad = mask_to_padded_indices(keep)
out = extractor(image, prefix, idx, pad) # kept patches only
pred = extractor.head(out["cls"]).argmax(-1)
Citation. If LookThere is useful in your work, please cite the paper.
@inproceedings{rammohan2026lookthere,
title = {LookThere! Sparse Vision by Reinforced Selection},
author = {Rammohan, Sreehari and Yassin, Yousef and Fuller, Anthony
and Wen, Junfeng and Vondrick, Carl and Shelhamer, Evan},
booktitle = {Advances in Neural Information Processing Systems},
year = {2026}
}
AF is primarily supported by an NSERC PGS-D scholarship. ES is supported by a Canada CIFAR AI Chair and the Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grant (RGPIN-2025-06878). YY is primarily supported by an NSERC CGRS-M scholarship. JW acknowledges the support of NSERC, RGPIN-2024-05357. SR and CV are partially supported by the National Science Foundation's Center for Smart Streetscapes (CS3) and the National Science Foundation's AI Institute for Artificial and Natural Intelligence.