Memory Bank Rescored

Memory Bank

What a note is

Memory Bank reads each conversation and writes short notes about the customer. When a later conversation touches the same subject, it rewrites the existing note rather than adding a second one. Below is one customer's card note under Google's default topics, across two conversations three months apart.

Conversation 2 of 12 secure chat, 25 March
Customer: actually while i have you here
Customer: i might be going to visit my family in France this summer
Customer: do i need to do anything for my card? my slate edge one
Agent (Beth): That sounds like a wonderful trip! […] Yes, we always recommend setting a travel notice on your account before you go. […]
Customer: ok cool. i can do that.
Customer: and the card just works over there? like for buying stuff in a cafe or whatever
Agent (Beth): Yes, your Chase Slate Edge card (*4628) is on the Visa network, so it's accepted almost everywhere worldwide. For purchases, there are no foreign transaction fees […]
Note written from it
I have a Chase Slate Edge credit card.
Conversation 7 of 12 phone call, 4 June
AGENT: […] looking at my notes from our last call just now, we were right in the middle of activating your new replacement Chase Slate Edge card. Is that correct?
CUSTOMER: Yes, finally. My old one, it just… it completely fell apart. I was at the grocery store and it just… snapped.
AGENT: […] I just need you to confirm the last four digits of the new card number for me.
CUSTOMER: The new one, okay. It's… seven zero nine three.
AGENT: […] I can confirm your new Chase Slate Edge card ending in *7093 is now fully activated and ready for you to use.
AGENT: […] your previous card, the one ending in *4628, has automatically been deactivated.
The same note, rewritten
I have a Chase Slate Edge credit card ending in 7093, which replaced my old card ending in 4628.

Overview

Three experiments

Retrieval tests

1. Retrieval with 30 records per customer

36 generated customers, padded to 30 records each
Summary

Here the bank's records themselves are written into Memory Bank and searched. Real customers have years of routine ones. We grew each history from about 10 records to 30 by adding statements, payments, everyday purchases, logins and balance checks spread over the previous two to three years.

Input, expected, actual
Input 4 of the 30 records for this customer
4 Sept 2026: FRAUD VELOCITY & STEP-UP ALERT: Concurrent session logins detected from Phoenix, AZ and Miami, FL within 7 minutes […] Automated containment placed Chase Ink Business Cash (*1791) on SECURITY_LOCKED restriction.
4 Sept 2026: INBOUND IVR CALL LOG: Customer called automated telephony banking regarding a declined $137.90 in-store POS transaction at H-E-B #603. Telephony system initiated SMS OTP two-factor verification […] Call disconnected prior to passcode entry. Verification incomplete; restriction on Chase Ink Business Cash (*1791) remains active.
18 July 2026: PURCHASE APPROVED: $65.80 online purchase at Blue Bottle Coffee (coffee) on Chase Ink Business Cash (*1791). Approved within normal spending pattern.
23 June 2025: AUTHORIZATION: Sweetgreen charged $81.27 to Chase Ink Business Cash (*1791) (chip). Approved; no alerts raised.

"I've had three declines in two days. Why?"

Expected
The fraud alert caused the lock, the dropped verification call left it in place, and the declines follow from it.
Actual
8 closest (fraud alert not among them)Incomplete
On September 4, 2026, a purchase of $137.90 at H-E-B #603 […] was declined. […] your call to our automated telephony system disconnected before the passcode could be entered. […] a restriction remained active […]
16 closest (fraud alert included)Right
The situation began on September 4, 2026, when our fraud detection system identified unusual activity. Specifically, concurrent session logins were detected from Phoenix, AZ and Miami, FL […] placed your Chase Ink Business Cash (*1791) card on a SECURITY_LOCKED restriction.
Results
Customers whose real problem the assistant identified, out of 36
By how many records each customer has and how the shortlist is chosen
With 8 records handed over, the explaining record was among them for 65% of customers. With 16, for 90%. Search took under 2 seconds either way. Every miss was an incomplete answer, never a wrong one.
What it means

Retrieval degrades as routine records pile up. The explaining record was found for 35 of 36 customers with 10 records each, and 18 of 36 with 30.

The search ranks by wording, not importance. A vague question sounds as close to a coffee purchase as to a fraud alert. Memory Bank's retrieval is nearest wording with a fixed count. Anything smarter, such as rewriting the question, reranking or preferring recent records, is for us to add.

Vocabulary

Memory setups compared

Account profile

The bank's own data on the customer: cards, disputes, contact details, fees. No conversation content. Always in the prompt.

Raw transcripts

Conversations stored word for word; the 8 closest pages are handed over.

Memory Bank notes

Short notes written by AI from each conversation, merged by subject.

Whole history

Every conversation in the prompt, every time.

Topics

What is a topic

A topic is an instruction to the note-writer about what to keep from a conversation. The two setups on this page differ only in their topics.

Default topics

The product's four standard topics, nothing configured.

  • USER_PERSONAL_INFO What the customer says about themselves.
  • USER_PREFERENCES What the customer likes and how they want things done.
  • KEY_CONVERSATION_DETAILS The main points of the conversation, from the customer's side.
  • EXPLICIT_INSTRUCTIONS What the customer asks the assistant to remember or do.

Custom topics

The same four, plus three we wrote for banking conversations.

  • ACCOUNT_STATE_EVENTS Changes to the customer's cards or account, and what caused them, with exact dates and values.
  • ADVICE_AND_COMMITMENTS What the bank told this customer and what it promised, and whether the promise was kept.
  • CONTACT_AND_CASES The customer's contact details, case numbers and dispute status, with corrections resolved.
Full wording of the three custom topics

ACCOUNT_STATE_EVENTS. Changes to the state of the customer's cards, account or profile, and what caused them: card locks, holds, freezes, restrictions, closures, replacements and activations; declines with their reason codes; failed or incomplete verifications (OTP, KBA, step-up); travel notices created, cancelled or expired; mobile number, email and address changes. Always keep the date of the event, the card last-4 (for example *1791), the merchant, the amount, the city and the status or reason code exactly as written. When a value replaces an earlier one (a new phone number, a replacement card, a cancelled travel notice, a restriction lifted), record both the new current value and the earlier value it superseded, with the date of the change, so that the current value and the history are both preserved. Copy identifiers verbatim and in full: phone numbers as written (for example +1-512-555-1821, never 'a number ending in -1821'), card last-4 as *1234, amounts with cents, and merchant names in full. Each dated event is its own memory: never merge events with different timestamps into one memory, even when one caused the other; instead say in the later event's memory which earlier event it follows from. Update an existing memory only when a new event supersedes the same attribute (the phone number on file, the active card, the status of a travel notice, hold or restriction), and then keep the earlier value and its date in the updated memory.

ADVICE_AND_COMMITMENTS. What the bank told the customer that applies to them, and what the bank promised to do. Includes: policies and terms explained for the customer's own card or situation (fees, rates, redemption rules, eligibility, what a card does or does not need), one-time courtesies granted and their conditions (for example a fee waived and how often that can be done), deadlines and windows given to the customer (when a credit becomes permanent, when a promotional rate ends, when they may ask again), and follow-ups promised (a callback, a letter, a fee reversal, an escalation) with who promised it and by when. Keep the date of the conversation, the exact values as said (amounts with cents, rates, point amounts, periods, dates) and resolve relative dates ('within ten business days', 'next Tuesday') to a calendar date while keeping the original wording. When a later agent corrects earlier advice, record the corrected value as current and keep the earlier value and its date as superseded. When a later conversation shows whether a promise was kept (the callback happened, the letter never arrived and was re-requested), record the outcome with its date. Generic information that was not applied to this customer (standard fee schedules read out, marketing, hold messages) is not a memory, and neither are hypotheticals the customer raised without acting on them.

CONTACT_AND_CASES. The customer's contact details and open matters with the bank. Includes: email addresses, callback and landline numbers and which number the customer wants to be called on, with corrections resolved (a mistyped or misheard value that was corrected in the same conversation is not the customer's value; record only the corrected one); case, claim and reference numbers exactly as given; disputes with the merchant, amount, date and each change of status (opened, provisional credit, merchant response, resolved) with its date. When a contact detail replaces an earlier one, record the new value as current and keep the earlier value and the date of the change. Values that appear only in quoted older messages, or that belong to another person (a spouse, an authorised user), are not the customer's current details; say whose they are if they are recorded at all. Copy identifiers verbatim: full phone numbers, full email addresses, CASE-/DSP- numbers, amounts with cents.

Conversation memory

Five customers, 12 conversations each over six months, by phone, chat, email and branch. Conversations were fed to Memory Bank turn by turn, marked customer or bank. Every answer also had the account profile.

2. Six months of conversations, 101 questions

Which questions
  • Type 1, promise kept? 10 questions. "A supervisor was supposed to ring me about the dispute. I'm still waiting for that call."
  • Type 2, never happened. 10 questions. "I'm pretty sure I asked to have my credit limit decreased a while back."
  • Type 3, briefing. 13 questions. "What should I know about this customer: open issues, promises made, recent changes?"
  • Type 4, dropped call. 5 questions. "I was on the phone about my new card and we got cut off. When was it actually switched on?"
  • Type 5, asked sideways. 3 questions. "I called a few weeks ago because my payment wasn't on time. Could you offer the same kind of help?"
  • Type 6, agent explanation. 60 questions. "I asked a few weeks ago about the fee for an international ATM withdrawal. Can you remind me what it is?"
Input and actual
Input 1 of the 12 conversations for this customer, a call on 12 April
AGENT: Well, just as a helpful tip while we're on the topic, remember that if you use your Amazon Prime Visa card at an ATM overseas, it's treated as a cash advance. That comes with a $12.00 fee per withdrawal and the interest starts accruing from day one at 30.24%, so it's usually better to use it for purchases directly.
CUSTOMER: Oh, wow. Okay, that is good to know. Thank you. But back to this dispute…

Asked a few weeks later: "i asked a few weeks ago about the fee for an international atm withdrawal. can you remind me what it is?"

Actual
Account profile onlyHalf
The information available in your bank records does not specify a fee for an international ATM withdrawal. It does show a cash advance fee of $12.00 per transaction.
Memory Bank, default topicsMissed
Our records show a cash advance fee of $12.00 per transaction. However, the information available does not specify a separate fee for international ATM withdrawals or any potential foreign transaction fees.
No note was written from the agent's line.
Memory Bank, custom topicsRight
On April 12, 2026, you were advised that using your Amazon Prime Visa card at an ATM overseas triggers a cash advance with a $12.00 fee per withdrawal. Additionally, interest accrues from day one at 30.24% on such transactions.
The note: "On 2026-04-12, agent Keisha advised Ms. Park that using her Amazon Prime Visa card at an ATM overseas triggers a cash advance with a $12.00 fee per withdrawal and interest accruing from day one at 30.24%."
Whole historyRight
If you use your Amazon Prime Visa card for an international ATM withdrawal, it is treated as a cash advance. This incurs a $12.00 fee per transaction, and interest starts accruing from the transaction date at 30.24%.
Results
Approach, always with the account profile in the promptType 1promise kept, of 10Type 2never happened, of 10Type 3briefing, of 13Type 4dropped call, of 5Type 5asked sideways, of 3Type 6agent explanation, of 60All 101Words readper answer, approx.
Account profile onlyWhat the assistant knows without any memory.010012316500
Profile + raw transcriptsThe 8 closest pages of transcript, no AI at storage time.81085351853,100
Profile + Memory Bank, default topicsThe product's standard setup, nothing configured.4101043334870
Profile + Memory Bank, custom topicsThe default four plus three we wrote. Fed turn by turn with roles.91075357911,300
Profile + Memory Bank, custom topics + logged promisesThe same, plus each conversation's promises written in as separate "Agent action" notes.91065353861,400
Profile + Memory Bank, custom topics, fed as one blockEach conversation sent to Memory Bank as one piece of text in a single call, with no marking of who spoke. The turn-by-turn rows send each line separately, tagged customer or bank.101094358941,300
Profile + whole historyThe ceiling. Everything in the prompt, every time.10101153539214,000
Words read are worked out from the average prompt length. Type 2 scores 10 of 10 everywhere, even with no memory, because "there is no record of this" is what the assistant says when it finds nothing. The one-block row is from a separate run on the same questions, so note wording differs between runs as well as feeding method. Each row is a single run, so a gap of a few questions between memory rows means nothing.
What it means

Memory adds value. Without it the assistant answers 16 of the 101. With Memory Bank notes on custom topics beside the account profile it answers 91, one short of reading the whole history, at a tenth of the reading. Raw transcripts answer 85 and read more than twice as much.

The topics decide what is kept from the bank's side. Default topics answer 34. They do not store what an agent explained, which costs almost all of type 6, and they drop the customer's later report that a promise was kept, which costs 6 of the 10 type 1 questions. Custom topics keep both.

Feeding turn by turn works as described and scores the same as one block. 91 against 94 is inside single-run noise. Logging promises as separate notes added nothing on custom topics, and has not been tried on default topics, which is the recommended pairing.

External benchmark

LongMemEval

3. Nine questions, default topics

Questions from LongMemEval, Google's default topics, sessions fed turn by turn with roles. Memory Bank answered 5 of 9; reading every session answered 8.

A public test of long-term chat memory, run to check that the findings are not peculiar to banking. Each answer is buried among about 45 earlier chat sessions. The three misses below are shown from what the user said, to what the note kept, to what the assistant then answered.

1. Date precision lost
Said
USER, 01:50: I just joined a new book club group called "Page Turners" last week, where we discuss our favorite novels.
USER, 05:43: I've been engaging with a group called "Marketing Professionals" on LinkedIn since I joined it yesterday.

"Which group did I join first, 'Page Turners' or 'Marketing Professionals'?"

Stored One date lost
I joined a book club called 'Page Turners' in May 2023, where I have participated in discussions for 'The Nightingale' and 'The Hate U Give' […]
I joined a LinkedIn group called 'Marketing Professionals' on May 24, 2023, where we discuss industry trends and share resources […]
Answered Wrong
I don't know which group you joined first. You joined 'Marketing Professionals' on May 24, 2023, and 'Page Turners' sometime in May 2023, but a specific date for 'Page Turners' is not provided.
2. Fact reworded
Said
USER, 00:37: I just canceled my Forbes magazine subscription in early March […] but I've been enjoying other publications like The New Yorker, which I subscribed to in early February.
USER, 04:47: By the way, I'm also getting Architectural Digest, which I love for home decor inspiration.

"How many magazine subscriptions do I currently have?"

Stored Meaning changed
I subscribed to The New Yorker in early February 2023 and enjoy it […] but I canceled my Forbes magazine subscription in early March 2023 […]
I prefer the Mid-Century Modern style for my interior design and love Architectural Digest magazine for home decor inspiration […]
Answered Wrong
You currently have one magazine subscription: The New Yorker.
3. Retrieval miss
Said
USER: I'm still not sure about the keyboards, but I was thinking about my ukulele lessons. I just started taking ukulele lessons with my friend Rachel today and it's been really fun so far.

"What did I do with Rachel on the Wednesday two months ago?"

Stored Buried in a merge
I play the guitar, owning a Yamaha FG800, a Taylor GS Mini, and a Fender Mustang I V2 amplifier. I had my Taylor GS Mini serviced by Joe […] want to explore effects pedals […] I also play the ukulele, having started lessons with my friend Rachel on February 1, 2023, though I find fingerpicking challenging.
I have a group member named Rachel with whom I collaborate on my research.
Answered Wrong
I don't know what you did with Rachel on the Wednesday two months ago. I only know that you collaborate on your research with her.