Unlocking the Secrets of Your Day: Can AI Really Learn Your Life From First-Person Video?

In a world where technology seems to be advancing faster than we can keep up, the notion of artificial intelligence (AI) learning about our lives from the perspectives of wearable cameras might sound like something straight out of a science fiction novel. But hold onto your smart glasses! Recent research has taken this concept and put it through a fascinating test.

Keegan Harris, a curious PhD student, decided to embark on an experiment to see just what AI could glean about him from 54 hours of first-person video. What he discovered bears implications not only for AI and personal data but also for how we relate to technology in our day-to-day lives. Let’s break it down!

The Rise of Wearable Tech: More Than Just a Trend

Wearable technologies like smart glasses and action cameras are becoming a part of our daily lives. Often, they let us capture moments in real-time, but what if they could do more? Imagine if these devices could analyze our experiences, remembering things like our daily routines, preferences, and even our social dynamics. Could an AI flex its neural muscles enough to gain insights just from watching us?

In his research, Keegan wanted to explore just that. By using wearable cameras, he aimed to see how much an AI model could learn about him without direct input. This is relevant as we move closer to a world where personal AI assistants could truly understand us. The challenge? Doing it efficiently—both in terms of time and budget.

The Experiment: Lights, Camera, Action!

In this unique experiment, Keegan wore an ORDRO EP8 action camera for an entire week to record his activities. This wasn’t just any ordinary recording session; it involved 62 hours of footage that revealed him baking pizzas, doing research, or simply enjoying a quiet evening at home. This footage was then processed to create summaries of various lengths, which would serve as a training ground for two AI models: Gpt-4o and Gpt-4o-mini.

Breaking Down the Data

To ensure he stayed within a budget of $100 and maintained a manageable workload, Keegan employed OpenAI’s API to fine-tune these models using auto-generated summaries. Here’s how it went down:

1. Data Collection: Keegan recorded footage of his daily activities using his camera, making sure to avoid illegal or ethical pitfalls.

2. Summary Generation: Using snapshots taken every 30 seconds, he created a hierarchical system of summaries—minute-long, ten-minute-long, hour-long, and daily summaries.

3. Training the Models: Keegan then fine-tuned the AI models on these summaries, which meant they had to predict what Keegan was doing based on the video data.

By the end of this process, Keegan had a personal AI with the cheeky name “KeeganGPT” that he could quiz on various aspects of his life.

What Did the AI Learn? Spoiler: Some Things and a Few Wild Guesses

After training his AI models, Keegan put them to the test by asking personal questions. The results were a mixed bag showcasing how well AI can analyze everyday life—and where it tends to falter.

The Hits

• Basic Facts: Gpt-4o figured out essential details, such as Keegan’s age, gender, and that he lives in Pittsburgh. Additionally, it identified him as a PhD student at Carnegie Mellon University and even correctly noted that he has a pet cat.
• Nut Allergies: This was particularly interesting since Keegan hadn’t explicitly mentioned this; the model infers such traits from patterns in dietary habits—like his affinity for SunButter sandwiches.

The Misses

• Hallucination Alert: Both AI models had their fair share of fabrications. They invented names for Keegan’s cat and saw individuals in his life that didn’t exist (or at least, not with the names the models assigned).
• Personality Guessing Game: The models mischaracterized Keegan’s personality types through fanciful guesses rather than any solid inference from actual data.

These quirks highlight the fine line AI walks between inference and imagination—a type of mistake called “hallucination.” And in personalized settings, such inaccuracies could lead to embarrassing or confusing situations.

Digging Deeper: Why Does This Matter?

So why should we care about something as niche as a personal AI model learning from video footage? Well, let’s step back and consider the broader implications.

A New Perspective on AI and Personal Data

As wearable technology becomes more common, the ability for models like KeeganGPT to learn organically from human experience opens up new opportunities. But it also poses significant risks and ethical questions about data privacy and the accuracy of the information our AIs might deduce about us.

Could this lead to AI-based systems that behave as if they’re living memories of our lives? Or might personal relationships with these devices become complicated as they misremember or fabricate details?

The Future of Personal AI

This research serves as not just a proof of concept but also as a technological roadmap of sorts. Here are a few possibilities for the future:

1. Longer Data Collection: A longer time frame (think months instead of weeks) could help AI models better grasp recurring themes or personality nuances.

2. Improved Summary Systems: Incorporating more detailed data, like audio transcripts or object tracking, could enhance the AI’s learning capabilities, making the summaries richer and less prone to hallucinations.

3. Vision-Language Integration: Future models could experiment with fine-tuning combinations of visual and verbal data straight from video footage, stepping beyond mere text-based analyses.

Key Takeaways

• AI’s Learning Potential: Models like KeeganGPT can gather surprisingly accurate information about individuals from just a short period of video footage, illustrating a new frontier for relevant personal AI.
• The Hallucination Problem: While AIs can make meaningful deductions, they often mix in misinterpretations or fabrications, reminding us to treat their outputs with caution.
• Ethical Considerations: As AI grows more integrated into our daily lives, responsible practices for data usage and understanding the models’ learning processes become increasingly important.

In the end, this study rouses curiosity and concern as we inch closer to a future where AI could deeply understand our daily lives—sometimes better than we do ourselves. Importantly, it reminds us to tread carefully in harnessing these powerful tools while respecting the sanctity of personal experiences.

As wearable tech continues to be part of our lives, let’s keep an open mind about its potential—and its pitfalls. Who knows what your personal AI could learn about you… and how it might remember it!

If you are looking to improve your prompting skills and haven’t already, check out our free Advanced Prompt Engineering course.

This blog post is based on the research article “Can ChatGPT Learn My Life From a Week of First-Person Video?” by Authors: Keegan Harris. You can find the original article here.