Guess the MMORPG business model (originally pioneered by casinos and their ilk) has escaped containment. Figure out what secret no-no hurts and squeeze that for money.
In unrelated news, Claude said something weird to me this morning before it gave me the first response of the day. "If you or someone you know is having a difficult time, free support is available." with a little stylized birdy icon sitting on a human hand. Should I be worried? 😅 Guess Anthropic is building a lawsuit shield now.
What struck me most was that first slide four psychiatric labels not as a diagnostic tool, but as a targeting map. There's something deeply unsettling about a system that looks at a person's anxiety or impulsivity and asks not how do we help, but how do we use this?
I find Mila's point about scale is what really stayed with me. A human consultant exploiting vulnerabilities is already wrong. But an algorithm doing it to hundreds of thousands of people, invisibly and automatically that's a different category of harm entirely.
The PCL framework shows that personalization doesn't have to mean manipulation, I feel the difference comes down to intent, are you adjusting to serve the person, or to pressure them?
Thank you both for bringing this into the open. These are exactly the conversations leaders need to be having before they sign the next vendor contract.
The same points struck me as well, Diamantino! And sadly it’s something I foresee happening more and more, which is why I love how @Mila Agius outlined it and brought it to light so clearly.
Excellent work, as usual, Mila! You can teach ethics frameworks, but you can’t make someone choose integrity. Here you show that no amount of pressure will move you off how you want to show up as an expert. I won’t even start on how they designed that system - it’s horrifying. Thanks for this collab!
This is so important to understand, not because of the labels or even the tactics, but because of how easy it would be for smart, well-intentioned people to rationalize it. When revenue is the primary metric, and AI removes the friction of scale, the ethical conversation can quietly fade into the background. No one sets out to build a weapon. They set out to build something that works.
What struck me most is that this was not a technology failure. It was a leadership and incentive failure. AI did not invent exploitation; it simply made it efficient and scalable. That distinction matters because it shifts the responsibility back where it belongs. The issue is not whether the model performs. The issue is what it has been designed to optimize.
As leaders, we have to ask ourselves what we are rewarding before we reward performance. Once a system is trained, automated, and embedded into infrastructure, intention becomes part of the architecture. Architecture is much harder to unwind than a campaign or a messaging tweak. If we don't define our guardrails at the beginning, scale will magnify whatever values we encoded, whether we meant to or not.
Personalization can absolutely reduce cognitive load and help people make clearer decisions. I believe that deeply. But if the system is designed to take advantage of someone’s emotional patterns rather than support their agency, then we need to question what we are building. I appreciate the courage it takes to surface this kind of story, because these are the conversations we need to have before systems become too big, too profitable, and too normalized to question.
Guess the MMORPG business model (originally pioneered by casinos and their ilk) has escaped containment. Figure out what secret no-no hurts and squeeze that for money.
In unrelated news, Claude said something weird to me this morning before it gave me the first response of the day. "If you or someone you know is having a difficult time, free support is available." with a little stylized birdy icon sitting on a human hand. Should I be worried? 😅 Guess Anthropic is building a lawsuit shield now.
really?? wow!!
Yeah, it was entertaining enough that I screenshot it for entertainment later.
What struck me most was that first slide four psychiatric labels not as a diagnostic tool, but as a targeting map. There's something deeply unsettling about a system that looks at a person's anxiety or impulsivity and asks not how do we help, but how do we use this?
I find Mila's point about scale is what really stayed with me. A human consultant exploiting vulnerabilities is already wrong. But an algorithm doing it to hundreds of thousands of people, invisibly and automatically that's a different category of harm entirely.
The PCL framework shows that personalization doesn't have to mean manipulation, I feel the difference comes down to intent, are you adjusting to serve the person, or to pressure them?
Thank you both for bringing this into the open. These are exactly the conversations leaders need to be having before they sign the next vendor contract.
The same points struck me as well, Diamantino! And sadly it’s something I foresee happening more and more, which is why I love how @Mila Agius outlined it and brought it to light so clearly.
Excellent work, as usual, Mila! You can teach ethics frameworks, but you can’t make someone choose integrity. Here you show that no amount of pressure will move you off how you want to show up as an expert. I won’t even start on how they designed that system - it’s horrifying. Thanks for this collab!
Yes, Anna! Exactly!!!
Disturbing but crucial, ethical AI isn’t optional when systems can exploit human vulnerabilities at scale.
This is so important to understand, not because of the labels or even the tactics, but because of how easy it would be for smart, well-intentioned people to rationalize it. When revenue is the primary metric, and AI removes the friction of scale, the ethical conversation can quietly fade into the background. No one sets out to build a weapon. They set out to build something that works.
What struck me most is that this was not a technology failure. It was a leadership and incentive failure. AI did not invent exploitation; it simply made it efficient and scalable. That distinction matters because it shifts the responsibility back where it belongs. The issue is not whether the model performs. The issue is what it has been designed to optimize.
As leaders, we have to ask ourselves what we are rewarding before we reward performance. Once a system is trained, automated, and embedded into infrastructure, intention becomes part of the architecture. Architecture is much harder to unwind than a campaign or a messaging tweak. If we don't define our guardrails at the beginning, scale will magnify whatever values we encoded, whether we meant to or not.
Personalization can absolutely reduce cognitive load and help people make clearer decisions. I believe that deeply. But if the system is designed to take advantage of someone’s emotional patterns rather than support their agency, then we need to question what we are building. I appreciate the courage it takes to surface this kind of story, because these are the conversations we need to have before systems become too big, too profitable, and too normalized to question.
Personalization becomes unethical the moment it targets human vulnerabilities rather than serving needs
What a fascinating read - thanks for introducing me to Mila's work Joel 👏