Is Your AI Memory Working Against You? How to Know and How to Fix It