Atomic Visual Entailment (AVE)

Code for the master's thesis Atomic Visual Entailment, LIACS, Leiden University.

Visual entailment asks whether an image supports, contradicts, or is neutral towards a sentence. Hypotheses often bundle several claims about objects, attributes, actions and relations into one sentence, which makes a single judgement hard to trust. AVE decomposes each hypothesis into self-contained atomic facts, compares three ways of reasoning over them with frozen vision-language models, combines the resulting predictions into one label, and localises the visual evidence behind that label without region-level supervision.

Model outputs from every stage of the pipeline are archived at doi.org/10.5281/zenodo.22051313.

Results

All numbers are SNLI-VE accuracy with frozen vision-language models and no visual-entailment training.

Method	Dev	Test
Best single VLM, full hypothesis	0.749	0.754
Best single atomic configuration	0.762	0.765
AVE-MV (majority voting, K=12)	0.765	0.769
AVE-LS (learned selection)	0.799	0.803
Oracle over the K=12 pool	0.941	0.941

Learned selection beats majority voting by 3.4 points and closes the gap to the oracle from 17.2 to 13.8 points on test. The two disagree on a large number of instances, and the split is not close to even: AVE-LS recovers 1,271 test cases majority voting gets wrong while losing 661, a 65.8% win rate on disagreements (McNemar, p = 2.3 x 10^-44).

Against published work on SNLI-VE, AVE-LS leads the zero-shot and hybrid methods by a wide margin — CLIP Res50x16 reaches 0.682, IMMO 0.657, IdealGPT 0.553 — and lands among supervised models trained on the task, above UNITER-Large (0.794) and level with VILLA-Large (0.800) and CLIP-ViL (0.802).

Where the gain comes from. Neutral is the hardest label for every method here. AVE-LS raises neutral recall from 0.551 to 0.682 on test, the largest single improvement in the system, and entailment recall from 0.832 to 0.857. It pays for this with contradiction recall, down from 0.923 to 0.871. Contradiction still ends up the most reliable label and neutral the least, but the selector is clearly trading the easy class for the hard one.

Grounding, evaluated against Flickr30k Entities with no region-level supervision anywhere in the pipeline:

Predicted label	Coverage	Recall@1	Recall@3	Mean IoU@3
Entailment	86.9%	0.823	0.896	0.843
Contradiction	77.3%	0.852	0.904	0.847
Overall	82.1%	0.837	0.900	0.845

Coverage is lower for contradiction because contradiction evidence often concerns something absent or an attribute mismatch, rather than a named object a detector can find.

Two negative results, both reported in full in the thesis:

Refinement hurts. Revisiting a prediction with extra textual evidence lowers accuracy in both variants tried — QA-assisted to 0.695, caption-assisted to 0.618, against 0.762 for the initial atomic prediction. The judge never sees the image, so it depends entirely on intermediate text, and when that text misses a visual detail the judge overturns a correct prediction. This is why refinement sits in ablations/ rather than in the pipeline.

Predictions do use the image. Replacing the image with a blank one collapses accuracy from 0.72–0.76 to 0.33–0.40, near chance on the three balanced labels. The models are not solving SNLI-VE from hypothesis text alone.

Stage	What it does
Decomposition	hypothesis to atomic facts
Prediction	VLM judgements over the full hypothesis and the atomic facts
Selection	combine candidate predictions into one label
Grounding	localise the evidence for the selected prediction
Evaluation	accuracy, ablations, grounding recall

Contents

• Installation
• Dataset
• Repository layout
• Run
  • Decomposition
  • Prediction
  • Selection
  • Grounding
  • Results
  • Refinement ablation

Installation

git clone https://github.com/Nalla-V/atomic-visual-entailment.git
cd atomic-visual-entailment
cp env.example.sh env.sh

Edit env.sh with your own paths. It is not tracked by git, so no filepaths need to be edited anywhere else.

Variable	Meaning
DATA_ROOT	directory holding Input/ and Output/
REPO_ROOT	where this repository is checked out
ENV_DECOMPOSE	conda environment name for the decomposition stage
ENV_PREDICT	conda environment name for the prediction stage
ENV_SELECTION	conda environment name for the selection stage
ENV_GROUNDING	conda environment name for the grounding stage
ENV_REFINEMENT	conda environment name for Florence-2 captioning
HF_TOKEN	HuggingFace token, needed only for gated models
HF_HOME	model cache directory

Each stage has its own environment, because the vision-language models and the text models need different library versions.

conda create -n decompose_env python=3.10 -c conda-forge -y
conda activate decompose_env
pip install -r envs/decompose.txt

conda create -n predict_env python=3.10 -c conda-forge -y
conda activate predict_env
pip install -r envs/predict.txt

conda create -n selection_env python=3.10 -c conda-forge -y
conda activate selection_env
pip install -r envs/selection.txt

conda create -n grounding_env python=3.10 -c conda-forge -y
conda activate grounding_env
pip install -r envs/grounding.txt

conda create -n refinement_env python=3.10 -c conda-forge -y
conda activate refinement_env
pip install -r envs/refinement.txt

The selection stage runs on CPU only, so envs/selection.txt has no CUDA dependencies.

The environments are not interchangeable. Grounding DINO, Florence-2 and Qwen3 each need a different transformers major version, and the mismatches are not always loud: Grounding DINO on the wrong version shifts every predicted box by a few pixels without raising an error. The pins are in envs/.

Dataset

Experiments use SNLI-VE, built on Flickr30k images. Grounding is evaluated against Flickr30k Entities. Please refer to their pages for how to download the data; the Flickr30k images require a signed request form and cannot be redistributed here.

Dataset	Source
SNLI-VE	https://github.com/necla-ml/SNLI-VE
Flickr30k images	https://shannon.cs.illinois.edu/DenotationGraph/
Flickr30k Entities	https://github.com/BryanPlummer/flickr30k_entities

Arrange them under $DATA_ROOT:

$DATA_ROOT/
├── Input/
│   ├── snli_ve_train.jsonl
│   ├── snli_ve_dev.jsonl
│   ├── snli_ve_test.jsonl
│   ├── flickr30k_images/        (needed from the prediction stage on)
│   └── flickr30k_entities/      (grounding evaluation only)
└── Output/                      (created automatically)

The archived outputs let the selection, grounding and results stages be reproduced without a GPU: download the prediction archives, extract them into $DATA_ROOT, and run those stages directly.

Repository layout

src/
├── config.py                    paths, model registry, shared settings
├── decomposition/
│   ├── decompose.py             hypothesis to atomic facts
│   └── prompts.py               decomposition prompt
├── prediction/
│   ├── predict.py               entry point, resume, output writing
│   ├── models.py                VLM registry and per-family adapters
│   ├── common.py                parsing, label scoring, confidence diagnostics
│   ├── prompts.py               prediction prompts
│   ├── baseline.py              full-hypothesis prediction
│   ├── joint.py                 joint atomic prediction
│   ├── selfdecompose.py         self-decomposition prediction
│   └── independent.py           independent atomic prediction
├── selection/
│   ├── voting.py                majority voting over the candidate pool
│   ├── train.py                 learned selector training and model selection
│   ├── evaluate.py              learned selector evaluation on dev and test
│   ├── analysis.py              ablations, importance, and the figures
│   ├── features.py              meta-feature extraction
│   ├── classifiers.py           classifier configurations
│   ├── candidates.py            the K=12 candidate pool
│   └── common.py                metrics and shared helpers
├── grounding/
│   ├── phrases.py               reasoning to groundable phrases
│   ├── detect.py                Grounding DINO boxes and figures
│   ├── summary.py               Flickr30k Entities evaluation
│   ├── prompts.py               phrase extraction prompt
│   └── common.py                phrase cleaning helpers
├── results/
│   ├── vlm_comparison.py        VLM comparison on full-hypothesis prediction
│   ├── candidate_analysis.py    all candidate configurations, by method and prompt
│   ├── ave_ls_summary.py        AVE-LS against majority voting
│   ├── hypothesis_bias.py       blank-image comparison
│   └── ablation_summary.py      refinement ablation
└── (ablations/refinement/)      the refinement ablation runs

assets/                          few-shot examples and the blank images
jobs/                            SLURM jobs, one per stage
envs/                            pip requirements, one per stage
env.example.sh                   template for env.sh

Run

Decomposition

Splits each hypothesis into atomic facts.

mkdir -p Logs
sbatch jobs/job_decompose.sh MODEL SPLIT [LIMIT]

sbatch jobs/job_decompose.sh qwen3 dev 10        # quick check
sbatch jobs/job_decompose.sh llama dev           # full dev split
sbatch --partition=gpu-a100-80g --mem=80G --gres=gpu:a100:1 --time=3-00:00:00 \
       jobs/job_decompose.sh qwen32 train        # main run

Logs/ must exist before submitting. Without SLURM:

source env.sh
conda activate $ENV_DECOMPOSE
python -m src.decomposition.decompose --model qwen32 --split train

Flag	Values	Default
--model	qwen32, qwen3, llama	qwen32
--split	train, dev, test	train
--limit	integer	all records

Model	Hub ID	GPU	Role
qwen32	Qwen/Qwen2.5-32B-Instruct	80 GB	reported results
qwen3	Qwen/Qwen3-8B	24 GB	decomposer comparison
llama	meta-llama/Meta-Llama-3.1-8B-Instruct	24 GB	decomposer comparison

Llama is gated: set HF_TOKEN and accept the licence on its model page.

Prediction

Runs the VLMs over each image. Both prompt styles run together from a single model load.

sbatch jobs/job_predict.sh VLM SPLIT METHOD [LIMIT]

sbatch jobs/job_predict.sh internvl dev all 10   # quick check, all methods
sbatch jobs/job_predict.sh internvl test all     # full test split
sbatch jobs/job_predict.sh qwen3 test joint      # one method
sbatch --partition=gpu-a100-80g --mem=80G --gres=gpu:a100:1 \
       jobs/job_predict.sh qwen3vl_32b dev all   # 32B model

Without SLURM:

source env.sh
conda activate $ENV_PREDICT
python -m src.prediction.predict --vlm internvl --split test --method all

Flag	Values	Default
--vlm	see table below	required
--split	train, dev, test	required
--method	baseline, joint, selfdecompose, independent, all	required
--limit	integer	all records
--decomposer	decomposer whose atoms to read	qwen32

VLM	Hub ID	Precision	Methods
qwen3	Qwen/Qwen3-VL-8B-Instruct	bfloat16	all four
internvl	OpenGVLab/InternVL3-8B-hf	bfloat16	all four
qwen3vl_32b	Qwen/Qwen3-VL-32B-Instruct	bfloat16	baseline
llava	llava-hf/llava-onevision-qwen2-7b-ov-hf	float16	baseline
idefics2	HuggingFaceM4/idefics2-8b	float16	baseline
qwen2vl_2b	Qwen/Qwen2-VL-2B-Instruct	bfloat16	baseline
internvl3_1b	OpenGVLab/InternVL3-1B-hf	bfloat16	baseline

qwen3 and internvl form the AVE prediction pool. The other five appear only in the VLM comparison, so all resolves to baseline for them.

Method	What the VLM sees
baseline	the full hypothesis
joint	the atomic facts together
selfdecompose	the hypothesis, decomposed by the VLM itself
independent	one atomic fact at a time

Hypothesis bias

Replaces the image with a blank one, to test whether predictions depend on the image or on the hypothesis text alone.

sbatch jobs/job_predict_bias.sh VLM SPLIT IMAGE [LIMIT]

sbatch jobs/job_predict_bias.sh internvl test black
sbatch jobs/job_predict_bias.sh internvl test white

IMAGE is black or white, taken from assets/. Covers the two pipeline VLMs and the full-hypothesis and joint atomic methods.

Selection

Combines the candidate predictions into one label. CPU only.

Majority voting

sbatch jobs/job_voting.sh

Intra-model voting per VLM (K=6), inter-model voting over the full pool (K=12), and the oracle upper bound.

Learned selection

Trains a classifier to pick the final label from the candidate pool.

sbatch jobs/job_train_selector.sh [OUTPUT_NAME]

Flag	Meaning	Default
--output-name	output folder under Output/	AVE_train_learned_selector_v3
--validation-size	held-out fraction of the training pool	0.20
--random-state	seed for the split and the classifiers	42

Feature variant	Candidate pool
baseline_only	full-hypothesis only, K=4
simple_atomic	atomic methods, simple prompt, K=4
structured_atomic	atomic methods, structured prompt, K=4
full_12_methods	2 VLMs x 3 methods x 2 prompts, K=12

Evaluation loads the saved model and applies it to a split:

sbatch jobs/job_evaluate_selector.sh dev [TRAIN_RUN_NAME]
sbatch jobs/job_evaluate_selector.sh test [TRAIN_RUN_NAME]

The post-training analyses take considerably longer than training itself, so they are separate:

sbatch jobs/job_analysis.sh [OUTPUT_NAME]

Use --skip data_efficiency importance objective to redraw the figures from the saved CSVs without recomputing. Training is seeded, so a rerun reproduces the same model.

Grounding

Links the selected prediction to visible image regions. Three steps, in order. Only entailment and contradiction predictions are grounded; the evaluation stage marks which rows qualify.

Phrase extraction

Turns the selected candidate's reasoning into short visual phrases.

sbatch jobs/job_grounding_phrases.sh SPLIT [LIMIT]

Stops at --target-per-label, 2500 by default, so grounding runs on a sample rather than the whole split.

Detection

sbatch jobs/job_grounding_detect.sh SPLIT [LIMIT]

Grounding DINO localises each phrase and draws the boxes. Pass --no-images to skip the figures.

Setting	Value
Model	IDEA-Research/grounding-dino-base
Box threshold	0.35
Text threshold	0.25
Boxes kept	3

Evaluation

sbatch jobs/job_grounding_summary.sh SPLIT

Scores the boxes against Flickr30k Entities. Reads annotations.zip directly, so the archive does not need unpacking.

Without SLURM:

source env.sh
conda activate $ENV_GROUNDING
python -m src.grounding.phrases --split test
python -m src.grounding.detect --split test
conda activate $ENV_SELECTION
python -m src.grounding.summary --split test

Note the environment changes between detection and evaluation.

Results

Tables and figures built from the earlier stages' outputs. CPU only.

sbatch jobs/job_results.sh SCRIPT [ARGS...]

Script	What it reports
vlm_comparison	each VLM's best full-hypothesis configuration
candidate_analysis	every configuration of VLM, method, prompt and label derivation
ave_ls_summary	AVE-LS against majority voting, including the paired test
hypothesis_bias	accuracy with the real image against a blank one
ablation_summary	the refinement ablation

sbatch jobs/job_results.sh vlm_comparison --split dev
sbatch jobs/job_results.sh candidate_analysis
sbatch jobs/job_results.sh ave_ls_summary

vlm_comparison takes --split; the others read both splits themselves. hypothesis_bias and ablation_summary need the corresponding ablation runs to have been done first.

Refinement ablation

Two ways of revisiting a prediction with extra evidence. Both start from the same initial prediction and feed src/results/ablation_summary.py. The scripts live in ablations/refinement/, since refinement is not part of AVE.

QA-assisted

sbatch jobs/job_question_generation.sh SPLIT
sbatch jobs/job_qa_answer_generation.sh SPLIT
sbatch jobs/job_qa_refinement.sh SPLIT

In order: questions are generated, answered against the image, and used to keep or overturn the initial prediction.

Caption-assisted

sbatch jobs/job_caption_generation.sh [LIMIT]
sbatch jobs/job_caption_refinement.sh [LIMIT]

Florence-2 captions every image once, then the judge works from the caption instead of the image. Captioning takes several hours; LIMIT is for testing.