Today we are announcing Ego4D. This long-term project of Facebook AI aims to address research challenges surrounding egocentric perception. It is an AI-based project that allows it to interact and understand the world from the first-person perspective. AI usually learns from third-person photos and videos, but the next generation of AI will have to learn from videos that depict the world from the center. This perspective could open up new opportunities for immersive experiences. Devices such as AR glasses and VR headsets will become just as useful as smartphones in everyday life.
For this project, we gathered a group of 13 labs and universities from nine countries. They collected more than 2 ,200 hour of first-person video, with over 700 people going about their day. This data set X is X larger than any other, in terms of hours of footage. This significantly increases the amount of egocentric information that is publicly available to researchers.
We also developed five benchmark challenges for developing smarter, more useful AI assistants including:
- Episodic memory: What happened when? (e.g. “Where did you leave your keys
- Forecasting? What are my chances of doing next?” You can also say, “Wait! You’ve already added salt Hand and object manipulation: What are you doing?” (e.g. “Teach Me How to Play the Drums
- Audiovisual Diarization: Who said What When?”)
- . (e.g. “What was the main subject during class
- Social interaction with whom?” (e.g., “Help me better hear the person talking to me at this noisy restaurant”)
Learn more about our AI research and developments.