Memory and personalization make AI more likely to tell you what you want to hear

AI companies have touted context retention (memory) and the availability of personal details (personalization) as mechanisms for improving AI model interaction. Both have value to help keep models from losing the thread of a conversation. But they raise the potential for sycophancy, where models will say what they predict you …

[Yet Another Anonymous coward]

A new el'reg unit

"with up to 25x higher sycophancy rates than in-context baselines."

What's that in Mandelsons ?

[elsergiovolador]

Re: A new el'reg unit

1 Mandelson = 1 Starmer + 1 Palantir?

[Bebu sa Ware]

Re: A new el'reg unit

> 1 Mandelson = 1 Starmer + 1 Palantir?

You mean a Mandelson is to a Palantir and Starmer like a guinea (= £1/1/- or 21s) was to £ and shilling ?

Not clear which would be the larger unit of synchophancy of the Palantir and Starmer and the rate of conversion between the two.

[Paul Herber]

Re: A new el'reg unit

Starmer and Mandelson - Burnham both.

[elsergiovolador]

Re: A new el'reg unit

I think it's because I missed Epstein coefficient.

[Paul Herber]

Re: A new el'reg unit

The Epstein Coefficient trumps everything.

[nobody who matters]

"Memory and personalization make AI more likely to tell you what you want to hear"

Tell us something we didn't already know.

[Pascal Monett]

"tell you what you want to hear"

How much longer is it going to take to end this senseless, useless waste of resources before we get back to actually working ?

[ecofeco]

Re: "tell you what you want to hear"

A lifetime. Or some great catastrophe. Not even joking.

[steelpillow]

Devil's advocacy

Rather than ranting at some AI for refusing to support some view which I hold dear, one of my regular techniques is to play Devil's advocate and challenge that view myself, and see if the AI will argue to uphold it or agree it's wrong. Really helps sidestep those endless "that's not what I meant" loops. Can't do that if it already knows who I am and what my views are.

[find users who cut cat tail]

Perfect

C-suite trying to replace staff with LLMs will be delighted to hear this.

[danielchalef]

The paper is flawed

Daniel from Zep here (one of the systems tested). The paper is flawed: its eval prompt orders the model to answer "solely" from retrieved memories — a recall-benchmark prompt aimed at reasoning tasks — while the baseline it's compared against gets no such instruction. Much of the measured "sycophancy" is just the model obeying that instruction, not a property of memory. Walked through with Writer's own numbers: https://blog.getzep.com/sycophancy-is-a-design-choice/

[ecofeco]

Turbo bias?

What could go wrong?