BLOG / All Posts

Personal AI

January 9, 2025|AI & Emerging Tech

How far are we from Zero and Hatchet in real life? The short answer is somewhere between today and five years depending on how fully you want that to be realized. Maybe it's easier to talk about what has to happen.

Devices

The natural home for an always-on, always-there AI Twin is AR glasses as discussed in the last note. But that's not the only place, nor will it be the first—phones will be. "Hey AI, what am I looking at?" type demos are already flying around YouTube (see the links below). Similarly, room speakers are another approach: won't it be great when Alexa and Siri aren't lousy anymore?

Multi-modality

This is already partially achieved. Throughout 2024, the premier labs have been adding the ability to look at images, listen to speech, and even live video. It's still a little bit "one thing at a time" as opposed to living in the moment like they'd have to do in your glasses, but it's getting there.

Reasoning

As Zero dismissively says, today Large Language Models (e.g. ChatGPT) really do blurt out the first thing that comes to mind. If you prompt it "Tell me a story about a robot." It will guess the most probable word that would come next (e.g. "Once") then feed itself a new prompt "Tell me a story about a robot. Once" then guess the next most likely world (e.g. "upon") and so on. There are many details on it, but that's the core idea. It doesn't think about what comes next or where it's going, it just blurts out the most likely next word (give or take) until it decides it's reached the end. This is also known as System One thinking. The sort of thing that we do without "thinking about it" like driving a car on familiar roads or catching a ball. But we want more from our AI. We want it to think about the answer, at least some of the time. This is called System Two thinking.

This is the common goal. All the major AI labs are working to get there. OpenAI (o3) and Google (Gemini 2.0) have just released versions that do a form of self-talk reasoning, where they generate multiple answers in the manner described above, but then recycle them through another pass, effectively acting as an editor. It's massively brute-force (aka expensive), but it's producing some great results.

Other labs (e.g. Facebook) are chasing different solutions, and for my money, it's the fact that different approaches are being pursued with big bets that makes it likely to come to pass sooner rather than later. As soon as one approach seems to be taking off, most of the other labs will copy it.

Memory

This is a complicated topic. Most major models now support some type of memory across queries, under different names, but this is more like remembering past chat sessions. It's a start.

What a Twin would need is full multi-modal input, localized to a single user, and the ability to consolidate that from raw input down to something that won't overload the datacenter ("AI Dreaming"). This is an area of active research, and just starting to be presented to the public. Google's Project Astra, for instance, remembers the most recent ten minutes of audio, video, and chat, then a background process will extract the most important bits using a combination of its own judgment and user requests ("remember this for me"). Getting this to work properly, at scale, will be one of the hardest pieces of the puzzle.

Conversation

It's ironic that a bunch of nerds are responsible for teaching our planet's next intelligent species how to have conversations, but fate loves irony. Getting the AIs to learn how to identify voices, ignore background noise, and correctly draw conclusions from others (sirens, bangs, creaky floorboards, etc.) is a monumental task requiring great judgment. It's well suited to the approaches of AI, rather than rule-based approaches.

Personalization

The basics of personalization are well understood. The process called fine-tuning takes pre-trained models which are fed the internet and trained to predict the next most likely token (word) and turns them into helpful assistants like ChatGPT, Claude, and Grok. It gives them a personality, of helpfulness which is hardly the only thing a model could learn by eating the internet.

Applying that Personalization to a Twin won't be hard, though the economics of having a personalized model per user are a challenge. Likely the bulk of the model is common, and the Twin sits as a small wrapper model on top. Or perhaps a few dozen pre-built models cover the major personality types and are then wrapped by a smaller layer for tone. Variations of these approaches have been used in other domains already.

Putting it all together

They already are! Google has a project Astra which is targeting exactly this problem (https://www.youtube.com/watch?v=ctWfv4WUp2I), using an Android phone as initial the interface. It's very much a work in progress, but you can feel all the pieces coming together. Beyond Google, just about every AI researcher worth his or her salt dreams of having the equivalent of Tony Stark's Jarvis. And all the rest of us want our Zeros. This is coming. I bet that by the end of 2026, there will be the first commercial solutions. That's two years away. Then they keep getting smarter. Buckle up.

Further reading and watching

  • Andrej Karpathy is one of the great teachers of AI and put together an excellent general introduction in late 2023. The entire video is worth watching, but this section in particular talks about how all the major labs are chasing the "System Two" thinking challenge here.

Andreeson on Twins

Google Astra

OpenAI demo'd their capabilities for conversational LLMs (including translation) in May of 2024.

More recently, Khoj was announced providing an initial framework for a personal AI based on local compute (instead of trusting a cloud provider).

This recent X post covers some of the experimentation underway on memory frameworks. The point isn’t that this is “the answer”, but rather that there is furious experimentation underway, and someone, somewhere will figure it out and it will be copied. Let a thousand flowers bloom.