I really love this (internal) benchmark that @OpenAI's memory team optimizes for in their new dreaming blog (linked below) They measure for "Staying correct over time" This is truly a hard problem for memory systems. but why? (this post is not AI written so pls read, thanks) Imagine a pile of books in front of you. Let's say you can look through 5 pages max before writing something new to the book. Someone tells you their name. You look through the titles of the book, then the table of contents, and then look at the 5 pages. Did I already know the name? Don't do anything. Did I know the wrong name before? Update information. Is this a new name? Create new. Do this a hundred times over, and you'll have information drift - You could not see everything when writing something new, so now you have redundant information, stale information, and your users will be complaining about memory. + If you do not have forgetfulness built into the system, it's not even the updates, it's the results themselves being stale on retrieval time. This happens to every AI product btw, I was recently complaining about Poke bringing up incorrect and stale information, and (i think) it's some mixture of these two problems stacking over. at @supermemory, we solve this by post-training a model that's specialized at "learning" about incoming context, along with a sampler that's also specialized for ensuring there's no redundancy :)