# Egocentric Vision

> Source: https://aiwiki.ai/wiki/egocentric_vision
> Updated: 2026-08-01
> Fact-checked: 2026-08-01
> Categories: AI Research, Computer Vision
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "Egocentric Vision." aiwiki.ai, 1 Aug 2026. https://aiwiki.ai/wiki/egocentric_vision
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

**Egocentric vision**, also called first-person vision, is the branch of [computer vision](https://aiwiki.ai/wiki/computer_vision) concerned with video and sensor data recorded from a camera worn on the head or body of the person whose activity is being studied. The framing dates to at least 2012, when Takeo Kanade and Martial Hebert set out first-person vision as a distinct problem in the Proceedings of the IEEE.[12] The field's modern form was set by three data releases: EPIC-KITCHENS in 2018, which put 55 hours of unscripted kitchen activity from 32 participants on the table[7]; Ego4D in 2021, which raised the scale to 3,670 hours from 931 camera wearers across 74 locations in 9 countries[1]; and Ego-Exo4D in 2023, which recorded 740 participants performing skilled activities from a head-mounted camera and several external cameras at the same time.[6] Egocentric vision supplies the public training data and evaluation that research on [AI glasses](https://aiwiki.ai/wiki/ai_glasses) is built on, and the community meets annually at the Joint Egocentric Vision (EgoVis) workshop, whose third edition was held on 3 June 2026 at [CVPR](https://aiwiki.ai/wiki/cvpr) in Denver.[15]

The field is a software one. [Waveguide display](https://aiwiki.ai/wiki/waveguide_display) optics and [neural wristband](https://aiwiki.ai/wiki/meta_neural_band) input are separate literatures; the 2024 survey by Plizzari and colleagues states that it treats egocentric vision "independent of any specific hardware."[11]

## What makes the first-person view a distinct problem

The camera is attached to a moving head, and that single fact propagates through every task. Plizzari et al. describe egocentric video as "characterised by sharp head rotations and predominantly forward motion," which produces rapid view changes and "short and noisy feature tracks"; worse, the dominant 3D rotation from natural head motion reduces parallax and causes triangulation errors.[11] The same survey notes that motion blur and unusual viewpoints "introduce overwhelming challenges" for 3D reconstruction of dynamic content, and it traces a line of egocentric-specific [SLAM](https://aiwiki.ai/wiki/slam) work built in response, including egomotion estimation aligned to the wearer's head motion and non-geometric formulations that map a person's habitual locations instead of a metric scene.[11]

The wearer's hands occupy a large share of the frame and are usually the point of the shot. The survey argues that egocentric video "provides a more fine-grained view into object interactions in which hands are central," and that methods for it differ substantially from human-object interaction detection using fixed cameras.[11] Ego4D's documentation makes the same point in operational terms: its Hands and Objects benchmark tracks "when, where and how an object is changed during its interaction," which the project calls "only possible through a first person Viewpoint."[2]

Two things are missing from the frame that third-person video supplies for free. The first is the wearer's own body, which is why Ego-Exo4D turns 3D body pose into a benchmark task of its own, to be solved "using only first-person raw video and/or egocentric camera pose."[15] The second is context: objects leave the field of view and stay gone, so Ego4D's visual-query task asks a system to localize an object "the last time it was seen in the video" rather than in the current frame.[3]

The reverse of the usual convention also holds. In third-person video, camera motion is a nuisance to be stabilized away; in egocentric video, the camera trajectory is a record of where the wearer went. Ego4D's forecasting benchmark makes this explicit by asking models to predict future ground-plane trajectories of the camera wearer, with ground truth obtained "using structure from motion" on the video itself, and by stating that forecasting "requires comprehending the camera wearer's intention."[4]

How much of this difficulty is really about the viewpoint is contested. A study presented at [ICCV](https://aiwiki.ai/wiki/iccv) 2025 tracked objects in synchronized first-person and third-person recordings of the same activities and found the answer "is not simply 'yes' or 'no'." Generic trackers did degrade on first-person footage, but the authors name the under-representation of the human-object activity domain in tracking training sets as one contributing factor rather than the viewpoint itself. Foundation-model trackers showed no first-person penalty when the target was visible, and objects held in the hands were easier to track from the first-person view, where they occupy more pixels.[21]

## The datasets

Scale figures for these datasets are widely misquoted, partly because several of them exist in more than one official version. The numbers below come from each paper's abstract or the dataset's own site.

| Dataset | Year | Venue | Scale | Camera wearers | Paper |
| --- | --- | --- | --- | --- | --- |
| EPIC-KITCHENS | 2018 | ECCV 2018 | 55 h, 11.5M frames, 39.6K action segments, 454.3K object boxes | 32, in 4 cities, 10 nationalities | arXiv:1804.02748[7] |
| EPIC-KITCHENS-100 | 2020 | IJCV | 100 h, 20M frames, 90K actions, 700 videos, 45 environments | not stated in the abstract | arXiv:2006.13256[8] |
| Ego4D | 2021 | CVPR 2022 | 3,670 h of daily-life activity, 74 locations, 9 countries | 931 (paper), 923 (site) | arXiv:2110.07058[1][2] |
| Ego-Exo4D | 2023 | CVPR 2024 | 1,286 h combined ego and exo, 123 scene contexts, takes of 1 to 42 minutes | 740, in 13 cities | arXiv:2311.18259[6] |
| HoloAssist | 2023 | ICCV 2023 | 166 h, 7 synchronized streams per session | 350 instructor-performer pairs | arXiv:2309.17024[13] |
| HD-EPIC | 2025 | CVPR 2025 | 41 h in 9 kitchens, 69 recipes, 59K fine-grained actions, 26K VQA questions | not stated in the abstract | arXiv:2502.04144[10] |

EPIC-KITCHENS established the collection method that later projects copied. Participants were not given a script: the authors "simply asked each participant to start recording every time they entered their kitchen," and the participants narrated their own footage afterwards, so the labels reflect stated intention rather than an annotator's guess.[7] The 2020 extension to 100 hours also introduced a "test of time" evaluation, checking whether models trained on 2018 footage still work on data collected two years later.[8] The project site adds that EPIC-KITCHENS-100 covers 45 kitchens in 4 cities with 20,000 unique narrations over 97 verb and 300 noun classes.[9]

Ego4D was assembled by [Meta AI](https://aiwiki.ai/wiki/meta_ai) (then Facebook AI) with what its site calls "an international consortium of 13 universities," comprising 88 researchers, using seven different off-the-shelf head-mounted camera models.[2] Its hours figure appears as three different numbers across first-party sources, which is the single most common error in citations to it: the paper's title says 3,000 hours, its abstract and the project site say 3,670, and the extended journal version in IEEE Transactions on Pattern Analysis and Machine Intelligence is titled "Around the World in 3,600 Hours."[1][2][5] Participant counts differ too, at 931 in the abstract and 923 on the site.[1][2] The corpus is not an open download: obtaining the data or annotations requires reviewing and accepting a license agreement first.[2]

Ego-Exo4D narrowed the activity range to skills with a notion of doing them well (cooking, bike repair, health procedures, music, dance, soccer, basketball, climbing) and recorded each take from an egocentric camera and several exocentric cameras simultaneously, with multichannel audio, eye gaze, 3D point clouds, camera poses, inertial data, and expert commentary from coaches and teachers.[6] Here too the released figures differ from the raw collection: the EgoVis workshop describes the effort as 839 camera wearers and 1,422 hours, while the paper reports the released 740 participants and 1,286 hours.[6][15]

The three terms are worth keeping apart. The **dataset** is the video and annotations. The **benchmark suite** is the set of task definitions, splits, and metrics released with it. The **challenge** is the annual competition run on a subset of those tasks, and it changes. The EPIC-KITCHENS-100 paper listed six challenges in 2020, including action anticipation[8]; the EPIC-KITCHENS set listed for the 2026 EgoVis workshop has since added semi-supervised video object segmentation and two EPIC-SOUNDS audio tracks.[15]

## Tasks and benchmarks

Ego4D organized its five benchmarks by temporal perspective: the past, the present, and the future.[1]

| Benchmark | What it measures | Why the first-person view makes it hard |
| --- | --- | --- |
| Episodic memory | Localize where in the wearer's own past video the answer to a query can be seen. Three query types: natural language (NLQ), visual (VQ), and moment (MQ).[3] | The search space is hours of unedited video with no shot structure, and the target may have left the frame long ago. |
| Hands and objects | Point-of-no-return temporal localization, state-change object detection, and object state-change classification.[2] | The point of no return is a single keyframe, and the hands causing the change are usually in front of it. |
| Forecasting | Locomotion prediction, hand-movement prediction, short-term object-interaction anticipation, and long-term action anticipation.[4] | The label is the wearer's future behaviour, so the task is inference about intention, not classification of what is visible. |
| Audio-visual diarization | Localize and track speakers in the visual field, detect speech activity, and transcribe what was said.[2] | Speakers move in and out of a narrow, shaking field of view, and identities in the corpus are anonymized to meet the consortium's guidelines.[2] |
| Social interactions | Detect when a person in view is looking at the camera wearer, and when they are talking to them.[2] | The signal is another person's attention directed at someone the camera cannot see. |

Episodic memory attracted the most follow-on work, and its annotation design is worth noting because it constrains what the benchmark can measure: the natural-language queries were written from 13 question templates before being rewritten for variety, and the moment queries draw on a taxonomy of 110 activities mined from the narrations.[3] Ego-Exo4D added four families of its own: fine-grained keystep recognition, proficiency estimation, cross-view translation between the ego and exo views, and 3D hand and body pose.[6] EPIC-KITCHENS contributes the recognition-side tasks, including action recognition, action detection, unsupervised domain adaptation, and cross-modal retrieval.[8][15]

## Where the numbers stand

Results here are reported per challenge track, so a score means nothing without its metric and split. Below are the winning entries the CVPR 2026 EgoVis organizers list for the previous edition of each challenge, plus two published benchmark findings.

| Task | Result | Source |
| --- | --- | --- |
| Ego4D short-term object-interaction anticipation | Top-5 overall mAP 7.21 (previous edition's winning entry) | EgoVis CVPR 2026 challenge listing[15] |
| Ego4D episodic memory, Goal Step track | 35.18 r@1 at IoU 0.3 (previous edition's winning entry) | EgoVis CVPR 2026 challenge listing[15] |
| Ego-Exo4D ego body pose | MPJPE 14.36 (EgoCast, listed as current state of the art) | EgoVis CVPR 2026 challenge listing[15] |
| EPIC-KITCHENS-100 action recognition | 48.1% top-1, 77.4% top-5 (previous edition's winning entry) | EgoVis CVPR 2026 challenge listing[15] |
| EPIC-KITCHENS-100 action detection | Action average mAP 31.97 (previous edition's winning entry) | EgoVis CVPR 2026 challenge listing[15] |
| EgoSchema multiple-choice QA | Below 33% for models with several billion parameters at publication, against 20% random and about 76% for humans | Mangalam et al., arXiv:2308.09126[14] |
| HD-EPIC VQA, 26,000 questions | 38.5% for Gemini Pro | Perrett et al., arXiv:2502.04144[10] |

[EgoSchema](https://aiwiki.ai/wiki/egoschema), built from Ego4D clips, is the clearest illustration of the gap. It poses over 5,000 human-curated multiple-choice questions over three-minute clips, and its authors introduced "temporal certificate sets" to argue that its questions require intrinsic temporal understanding 5.7 times longer than the next-closest dataset.[14] Single-digit and low-double-digit scores on anticipation and episodic-memory tracks are the norm rather than outliers.

## Project Aria as a research instrument

Ego4D was recorded on seven different off-the-shelf camera models, deployed so that models would not overfit to a single capture device.[2] [Project Aria](https://aiwiki.ai/wiki/project_aria) is Meta Reality Labs Research's purpose-built alternative. Its paper describes the device as "an egocentric, multi-modal data recording and streaming device with the goal to foster and accelerate research in this area."[16] It is not sold as a product.

Academic use runs through the Aria Research Kit, which Meta says "is available to select partners through a rolling application process." The kit bundles Aria Gen 2 glasses with a Companion App, the open-source Project Aria Tools, the Aria Studio desktop tool, a Client SDK, and cloud machine-perception services that Meta lists as including 6DoF trajectory and 3D eye-gaze estimation. No price is published.[17] Meta reported in October 2025 that nearly 300 academic laboratories in 27 countries had used over 1,000 first-generation Aria devices, and that it was targeting a broad rollout of Aria Gen 2 to qualified applicants in the second quarter of 2026.[18] Aria supplied the egocentric capture for [Ego-Exo4D](https://aiwiki.ai/wiki/ego_exo4d), so the instrument and the dataset programme are the same effort.[6] Anonymization is built into it: Meta's EgoBlur system detects and Gaussian-blurs bystander faces and vehicle license plates before Aria footage is retained for research.[19]

## What actually reaches products

The shipping counterpart to this research is the assistant on [Meta Ray-Ban AI glasses](https://aiwiki.ai/wiki/meta_ray_ban_ai). Meta introduced Live AI to Early Access Program members in the United States and Canada on 16 December 2024, describing it as a mode in which "Meta AI can see what you see continuously and converse with you more naturally than ever before," where the wearer "can ask questions without saying 'Hey Meta,' reference things you discussed earlier in the session, and interrupt anytime to ask follow-up questions or change topics." The same post says that "eventually live AI will, at the right moment, give useful suggestions even before you ask," which is the proactive-assistance problem stated as a product roadmap.[25]

The research-to-product link is not publicly documented. Meta has not published which models or datasets power Live AI, and no first-party source connects a specific Ego4D or Ego-Exo4D benchmark to a shipping feature. What can be observed is which capabilities appear and which do not. Continuous [visual question answering](https://aiwiki.ai/wiki/visual_question_answering) over the current scene, live translation, and music identification shipped in the same December 2024 update.[25] Episodic memory, the flagship Ego4D benchmark and the capability the "where did I leave my keys?" example has stood for since 2021, did not: no general retrieval over a wearer's own recorded past is available on these devices as of 2 August 2026. The NeurIPS 2025 WAGIBench paper opens with that exact query as its motivating example and then measures how far current systems are from answering it.[22]

## Open problems as of 2026

**Long-horizon memory.** S-EMBER, a 2026 benchmark of 3,141 videos and 388 hours captured on Ray-Ban Meta glasses with 9,448 question-answer pairs, reports what its authors call a localization paradox: semantic reasoning improves as models get larger, but temporal grounding precision "remains a stagnant architectural bottleneck that does not benefit from brute-force increases in model size, resolution, or frame density."[23] That is an architectural finding rather than a data-scale one, which is why more hours of video are not obviously the fix.

**Evaluation that reflects real use.** WAGIBench collected 29 hours of multimodal data from 348 participants across 3,477 recordings to measure whether a wearable agent can infer a user's goal from context. Humans scored 93% on its multiple-choice split against 84% for the best vision-language model, and in the generative setting the models produced relevant goals only 55% of the time, which the authors describe as "far from practical usefulness."[22] The Plizzari survey makes the related methodological point that a real device cannot run one deep model per task, because the tasks are co-dependent, and that open-set evaluation is needed so models can handle categories that were not in the label set.[11]

**Privacy, in both directions.** Bystander anonymization has a deployed baseline in EgoBlur.[19] The wearer is the less-studied case. EgoPrivacy, presented at ICML 2025, defines seven tasks across demographic, individual, and situational privacy and shows that foundation models can recover a wearer's identity, scene, gender, and race at 70% to 80% accuracy zero-shot, with a retrieval-augmented ego-to-exo attack raising the demographic figures further.[20] The broader consumer-facing debate is covered under [smart glasses privacy](https://aiwiki.ai/wiki/smart_glasses_privacy).

**On-device efficiency.** Continuous capture is expensive: the Plizzari survey notes that wearable glasses battery life "is often only a few hours" and that constant video capture shortens it further.[11] A 2026 system called EPIC attacks this with algorithm-hardware co-design, using gaze, pose, and inertial signals to infer intent and discard uninformative input, reporting a 27.5 times reduction in memory footprint and a 24.3 times reduction in energy against a full-video baseline while preserving accuracy on egocentric video understanding tasks.[24] Whether such gating is compatible with episodic memory, which by definition cannot know in advance what will later be asked about, is unresolved.

## Use beyond wearable devices

Much of the influence of these [datasets](https://aiwiki.ai/wiki/datasets) has run past the leaderboards. Ego4D ships dense free-form textual narrations alongside the video,[2] which makes it usable as paired video-and-text pretraining material for [vision-language models](https://aiwiki.ai/wiki/vision_language_model) and other [multimodal models](https://aiwiki.ai/wiki/multimodal_model) trained with [self-supervised](https://aiwiki.ai/wiki/self-supervised_learning) objectives, not only as a [benchmark](https://aiwiki.ai/wiki/benchmark) suite; first-person footage of people manipulating objects doubles as demonstration data for [embodied AI](https://aiwiki.ai/wiki/embodied_ai) and robot learning. Ego-Exo4D's paired views were released to connect first-person and third-person understanding, which the 2024 survey names as the direction the field should converge on.[6][11] Work in the area is presented mainly at CVPR, ICCV, and [NeurIPS](https://aiwiki.ai/wiki/neurips).

## See also

- [Ego4D](https://aiwiki.ai/wiki/ego4d)
- [Ego-Exo4D](https://aiwiki.ai/wiki/ego_exo4d)
- [Project Aria](https://aiwiki.ai/wiki/project_aria)
- [AI glasses](https://aiwiki.ai/wiki/ai_glasses)
- [EgoSchema](https://aiwiki.ai/wiki/egoschema)
- [Smart glasses and privacy](https://aiwiki.ai/wiki/smart_glasses_privacy)

## References

1. Grauman, K., et al. "Ego4D: Around the World in 3,000 Hours of Egocentric Video." arXiv:2110.07058 (v3, 2022-03-11); CVPR 2022, doi:10.1109/CVPR52688.2022.01842. https://arxiv.org/abs/2110.07058
2. "Egocentric 4D Perception (EGO4D)." Ego4D consortium project site. https://ego4d-data.org/
3. "Episodic Memory." Ego4D benchmark documentation. https://ego4d-data.org/docs/benchmarks/episodic-memory/
4. "Forecasting." Ego4D benchmark documentation. https://ego4d-data.org/docs/benchmarks/forecasting/
5. Grauman, K., et al. "Ego4D: Around the World in 3,600 Hours of Egocentric Video." IEEE Transactions on Pattern Analysis and Machine Intelligence 47(11): 9468-9509, 2025. doi:10.1109/TPAMI.2024.3381075. https://doi.org/10.1109/TPAMI.2024.3381075
6. Grauman, K., et al. "Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives." arXiv:2311.18259 (v4, 2024-09-25); CVPR 2024, doi:10.1109/CVPR52733.2024.01834. https://arxiv.org/abs/2311.18259
7. Damen, D., et al. "Scaling Egocentric Vision: The EPIC-KITCHENS Dataset." arXiv:1804.02748 (v2, 2018-07-31); ECCV 2018, doi:10.1007/978-3-030-01225-0_44. https://arxiv.org/abs/1804.02748
8. Damen, D., et al. "Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100." International Journal of Computer Vision, 2021-10-20. doi:10.1007/s11263-021-01531-2. https://doi.org/10.1007/s11263-021-01531-2
9. "EPIC-KITCHENS Dataset." University of Bristol project site, 2026 edition. https://epic-kitchens.github.io/2026
10. Perrett, T., et al. "HD-EPIC: A Highly-Detailed Egocentric Video Dataset." arXiv:2502.04144, 2025-02-06; CVPR 2025. https://arxiv.org/abs/2502.04144
11. Plizzari, C., Goletto, G., Furnari, A., Bansal, S., Ragusa, F., Farinella, G. M., Damen, D., Tommasi, T. "An Outlook into the Future of Egocentric Vision." International Journal of Computer Vision, 2024-05-28. doi:10.1007/s11263-024-02095-7. Preprint: https://arxiv.org/abs/2308.07123
12. Kanade, T., and Hebert, M. "First-Person Vision." Proceedings of the IEEE 100(8): 2442-2453, August 2012. doi:10.1109/JPROC.2012.2200554. https://doi.org/10.1109/JPROC.2012.2200554
13. Wang, X., et al. "HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real World." arXiv:2309.17024, 2023-09-29; ICCV 2023. https://arxiv.org/abs/2309.17024
14. Mangalam, K., Akshulakov, R., Malik, J. "EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding." arXiv:2308.09126, 2023-08-17. https://arxiv.org/abs/2308.09126
15. "Third Joint Egocentric Vision (EgoVis) Workshop, in conjunction with CVPR 2026." 2026-06-03, Denver. https://egovis.github.io/cvpr26/
16. Engel, J., et al. "Project Aria: A New Tool for Egocentric Multi-Modal AI Research." arXiv:2308.13561 (v3, 2023-10-01). https://arxiv.org/abs/2308.13561
17. "Aria Research Kit." Project Aria, Meta. https://www.projectaria.com/research-kit/
18. "Aria Gen 2 Applications Are Now Open." Meta, 2025-10-19. https://www.meta.com/blog/aria-gen-2-updates/
19. Raina, N., et al. "EgoBlur: Responsible Innovation in Aria." arXiv:2308.13093, 2023-08-24. https://arxiv.org/abs/2308.13093
20. "EgoPrivacy: What Your First-Person Camera Says About You?" arXiv:2506.12258, 2025-06-13; ICML 2025. https://arxiv.org/abs/2506.12258
21. "Is Tracking really more challenging in First Person Egocentric Vision?" arXiv:2507.16015, 2025-07-21; ICCV 2025. https://arxiv.org/abs/2507.16015
22. "Benchmarking Egocentric Multimodal Goal Inference for Assistive Wearable Agents" (WAGIBench). arXiv:2510.22443, 2025-10-25; NeurIPS 2025 spotlight. https://arxiv.org/abs/2510.22443
23. "S-EMBER: A Large-Scale Benchmark for Streaming Egocentric Memory Retrieval." arXiv:2607.02689, 2026-07-02. https://arxiv.org/abs/2607.02689
24. "EPIC: A System Framework for Efficient Egocentric Perception on Embodied AR Glasses." arXiv:2606.15859, 2026-06-14. https://arxiv.org/abs/2606.15859
25. "Ray-Ban Meta Glasses Add Live AI, Live Translation, and Shazam Support." Meta, 2024-12-16. https://www.meta.com/blog/ray-ban-meta-v11-software-update-live-ai-translation-shazam/