Visual Grounding

How to read this page: This article maps the topic from beginner to expert across six levels � Remembering, Understanding, Applying, Analyzing, Evaluating, and Creating. Scan the headings to see the full scope, then read from wherever your knowledge starts to feel uncertain. Learn more about how BloomWiki works ?

Visual grounding is the AI capability to connect language to specific visual regions — locating the objects, regions, or relationships described by text within an image or video. While image classification says "there is a dog," visual grounding says "the dog is in the bottom-left corner, sitting on a red mat." This capability underpins a family of tasks: referring expression comprehension (find what a phrase describes), visual question answering (answer questions about specific image regions), grounded captioning (generate text anchored to specific regions), and phrase grounding (link words to image regions). Visual grounding is essential for robots, assistive AI for the blind, and multimodal reasoning systems.

Remembering

• Visual grounding — Localizing image regions corresponding to natural language descriptions.
• Referring Expression Comprehension (REC) — Given an image and a referring expression ("the woman in the red dress on the left"), locate the described object.
• Phrase grounding — Linking each phrase in a caption to the corresponding image region.
• Visual Question Answering (VQA) — Answering questions about image content; often requires grounding attention to relevant regions.
• Region proposal — Generating candidate bounding boxes in an image; the first stage of many grounding pipelines.
• DETR (Detection Transformer) — An end-to-end object detection transformer; adapted for grounding in MDETR and others.
• MDETR — Modulated DETR: a joint vision-language model for end-to-end grounding.
• Grounding DINO — Open-vocabulary object detection and grounding using DINO + grounding pre-training.
• GLIP (Grounded Language-Image Pre-training) — Unifies detection and grounding pre-training for open-vocabulary detection.
• RefCOCO — A widely used referring expression dataset (120,000+ expressions for COCO images).
• Bounding box — A rectangle locating an object in an image; the primary output format for grounding.
• Region features — Visual features extracted from specific image regions (RoI pooling, RoI align) for multi-modal reasoning.
• Open-vocabulary detection — Detecting objects described by arbitrary text, not just a fixed set of training categories.
• SAM (Segment Anything Model) — Meta's foundation model for segmenting any object from a point, box, or text prompt.

Understanding

Visual grounding requires simultaneously understanding language semantics and visual scene structure, then aligning them. This is fundamentally harder than either task alone — it requires knowing what "the woman on the left in the red dress" means (language), identifying all relevant visual regions (vision), and matching the description to the correct region (grounding).

Two-stage vs. end-to-end: Early grounding systems used two stages:

1. generate region proposals (Selective Search, RPN),
2. rank proposals by language-visual similarity. Modern end-to-end systems (MDETR, Grounding DINO) jointly process image and text, generating grounded outputs in one forward pass. End-to-end approaches outperform two-stage but require more data and training.

Grounding DINO: The current standard for open-vocabulary grounding. Combines DINO (a strong visual backbone) with language-conditioned attention. Given an image and any text query, it outputs bounding boxes for described objects. Crucially, it generalizes to objects not seen during training — "open vocabulary" — making it vastly more flexible than fixed-category detectors.

SAM + language: SAM can segment any object from a bounding box or point prompt. Combining Grounding DINO (detect → bounding box) with SAM (box → precise segmentation mask) gives a powerful open-vocabulary segmentation pipeline. LangSAM makes this combination accessible in a few lines of code.

Multimodal LLMs and grounding: GPT-4V, LLaVA, and Qwen-VL can discuss image regions conversationally. However, precise bounding box output requires specialized models. The field is rapidly moving toward unified models that can both ground (output boxes) and reason (generate text about grounded regions) in a single framework (e.g., Qwen2-VL, InternVL2).

Applying

Open-vocabulary grounding with Grounding DINO + SAM: <syntaxhighlight lang="python"> from PIL import Image import torch import numpy as np

1. Method 1: Grounding DINO for bounding box grounding

from groundingdino.util.inference import load_model, load_image, predict, annotate

model = load_model("groundingdino/config/GroundingDINO_SwinT_OGC.py",

                  "weights/groundingdino_swint_ogc.pth")

image_source, image = load_image("street_scene.jpg")

1. Ground a natural language description

boxes, logits, phrases = predict(

   model=model,
   image=image,
   caption="the woman in the red dress . the yellow car on the left",
   box_threshold=0.35,
   text_threshold=0.25

) annotated = annotate(image_source=image_source, boxes=boxes, logits=logits, phrases=phrases) Image.fromarray(annotated).save("grounded_output.jpg") print(f"Found {len(boxes)} objects: {phrases}")

1. Method 2: LangSAM (Grounding DINO + SAM combined)

from lang_sam import LangSAM

lang_sam = LangSAM() image = Image.open("garden.jpg").convert("RGB")

1. Get segmentation masks for any text description

masks, boxes, phrases, logits = lang_sam.predict(image, "red flowers")

1. masks: list of boolean numpy arrays — precise pixel-level masks

for i, (mask, phrase) in enumerate(zip(masks, phrases)):

   masked_image = np.array(image.copy())
   masked_image[~mask] = 0  # Keep only the grounded region
   Image.fromarray(masked_image).save(f"grounded_mask_{i}.png")

</syntaxhighlight>

Visual grounding systems

Open-vocabulary detection → Grounding DINO, GLIP, OWL-ViT (Google)

Segmentation from text → LangSAM, SEEM, X-Decoder

Referring expression → MDETR, TransVG, SeqTR

Multimodal reasoning → Qwen2-VL, InternVL2, LLaVA-1.6 (grounded output)

Video grounding → TubeDETR, MOMA, CLIP4Clip for temporal grounding

Analyzing

Visual Grounding Benchmarks (RefCOCO val)

Model	val Accuracy	testA (people)	testB (objects)	Speed
TransVG	81.0%	82.7%	78.4%	Fast
MDETR	86.8%	89.6%	81.4%	Moderate
Grounding DINO	90.6%	92.5%	86.7%	Moderate
Qwen2-VL (7B)	91.4%	93.2%	87.8%	Moderate
GPT-4o (with tools)	~89%	~91%	~85%	Slow (API)

Failure modes: Attribute confusion — "the large red ball" may ground to a small red ball or a large blue ball. Relational grounding failures — "the ball to the left of the box" requires spatial reasoning that models handle inconsistently. Overcrowded scenes — models struggle when many similar objects are present. Out-of-vocabulary objects — even open-vocabulary models fail on rare, unusual objects.

Evaluating

Visual grounding evaluation:

1. RefCOCO/RefCOCO+/RefCOCOg: standard referring expression benchmarks; report [email protected] IoU (box overlaps ground truth by >50%).
2. Flickr30k Entities: phrase grounding benchmark.
3. Open vocabulary: COCO novel categories test (COCO-base trained, COCO-novel tested).
4. Accuracy by referent type: people vs. objects vs. scenes — performance varies significantly.
5. Robustness: test with paraphrased expressions, negations, spatial relations.

Creating

Building a grounding-enabled visual search system:

1. Ingest image library; extract Grounding DINO features.
2. Text query → Grounding DINO → bounding boxes + confidence scores.
3. Apply SAM to convert boxes to precise segmentation masks.
4. Post-processing: NMS to remove duplicate detections; threshold on confidence.
5. Interface: display image with highlighted regions and confidence scores.
6. Applications: e-commerce visual product search, accessibility (describe what's at position X), medical image ROI identification, surveillance (find person in red jacket), robotics (pick the blue cup).