EgoSAT: A Comprehensive Benchmark of Egocentric Streaming Interaction Understanding

Introduction

Recent advances in wearable cameras, edge computing, and vision-language models are making first-person AI assistants increasingly practical. In real deployments, such assistants must continuously understand streaming video, respond to the user's current environment, remember relevant past events, and anticipate plausible future interactions.

Existing benchmarks often evaluate video question answering, online narration, and activity anticipation as separate problems. EgoSAT instead uses a unified streaming formulation for egocentric interaction understanding. At each query time, a model can only use video that has already been observed, and it is evaluated on retrospective, present, and prospective reasoning in the same benchmark. EgoSAT also studies whether models can recognize answerability under partial observability and calibrate their confidence when an answer may be uncertain.

Three distinct reasoning modes

Dataset

EgoSAT is built from Ego4D and provides annotations, multiple-choice candidates, effective querysets, metadata, SFT manifests, and optional ROI cache files for egocentric streaming interaction understanding. Raw Ego4D videos are not redistributed with EgoSAT; users should obtain Ego4D access separately under the corresponding terms.

Dataset Statistics

Answerability-aware Construction

Branchiness. Branchiness captures uncertainty from multiple plausible and semantically diverse future continuations after the same observed interaction prefix.

Surprise. Surprise captures uncertainty from abrupt visual and semantic shifts between the recent observed context and the imminent future event.

Evaluation Pipeline

Data Preparation

EgoSAT separates benchmark annotations from raw videos. Download the EgoSAT annotation release and ROI cache release from Hugging Face, then place local Ego4D videos under your private video root before running video inference.

EgoSAT-data/
  egosat/
    gt/
    mcq_shuffled/
    effective_querysets/
    metadata/
  sft/

EgoSAT-ROI-Cache/
  roi_cache/
  metadata/

Ego4D/
  videos/

Model Preparation

To evaluate a custom model on EgoSAT, implement a model adapter that connects the model to the EgoSAT runner and helper pipeline. An adapter should load video prefixes available before each query time, receive task-specific prompts and querysets, return structured answers, and optionally expose multiple-choice confidence scores for diagnostics.

Evaluation Command

The official scorer uses raw per-sample prediction JSON files and the released effective querysets. Normalized JSONL files are provided for inspection and downstream convenience, but they are not the official scoring source.

python evaluation/evaluate_main_table.py \
  --pred-root /path/to/predictions \
  --gt-root /path/to/EgoSAT-data/egosat/gt \
  --mcq-root /path/to/EgoSAT-data/egosat/mcq_shuffled \
  --effective-queryset-root /path/to/EgoSAT-data/egosat/effective_querysets

Experimental Results

Performance of closed-source proprietary models, offline open-weight VLMs, and streaming VLMs on EgoSAT

Citation

If you find EgoSAT useful for your research, please cite:

@inproceedings{lei2026egosat,
  title={EgoSAT: A Comprehensive Benchmark of Egocentric Streaming Interaction Understanding},
  author={Lei, Yijia and Li, Jinzhao and Zhang, Yichi and Hua, Jiacheng and Li, Yin and Liu, Miao},
  booktitle={European Conference on Computer Vision},
  year={2026}
}

License

Code in this repository is released under the MIT License. EgoSAT annotations, querysets, SFT manifests, and ROI cache are released for research use under the usage terms described in the Hugging Face dataset cards. Raw Ego4D videos are not redistributed and remain governed by the Ego4D license and terms of use.