Google has announced the release of its new AI model family, PaliGemma 2, which boasts the ability to analyze images and detect emotions. While this capability may seem impressive, experts are raising red flags over the potential consequences of making such technology openly available.

The PaliGemma 2 family of models can generate detailed captions for images, going beyond simple object identification to describe actions, emotions, and the overall narrative of the scene. However, emotion recognition doesn't work out of the box, and the model needs to be fine-tuned for this purpose. Despite this, experts are concerned about the implications of an openly available emotion detector.

Sandra Wachter, a professor in data ethics and AI at the Oxford Internet Institute, expressed her reservations about the technology, stating that it's "problematic to assume that we can 'read' people's emotions." She likened it to asking a Magic 8 Ball for advice, highlighting the uncertainty and potential inaccuracy of emotion detection.

For years, startups and tech giants have attempted to build AI that can detect emotions, but the science behind it is shaky at best. The majority of emotion detectors rely on the work of Paul Ekman, a psychologist who theorized that humans share six fundamental emotions. However, subsequent studies have cast doubt on Ekman's hypothesis, demonstrating that people from different backgrounds express emotions in complex and varied ways.

Mike Cook, a research fellow at Queen Mary University specializing in AI, emphasized that emotion detection is not possible in the general case, as people experience emotions in complex ways. He noted that while it's possible to detect some generic signifiers in certain cases, it's not something that can be fully "solved."

The unreliability and bias of emotion-detecting systems are well-documented. A 2020 MIT study showed that face-analyzing models can develop unintended preferences for certain expressions, like smiling. More recent work suggests that emotional analysis models assign more negative emotions to Black people's faces than white people's faces.

Google claims to have conducted "extensive testing" to evaluate demographic biases in PaliGemma 2, finding "low levels of toxicity and profanity" compared to industry benchmarks. However, the company did not provide the full list of benchmarks used or indicate which types of tests were performed. The only disclosed benchmark is FairFace, a set of tens of thousands of people's headshots, which some researchers have criticized as a bias metric.

Heidy Khlaaf, chief AI scientist at the AI Now Institute, emphasized that interpreting emotions is a subjective matter that extends beyond visual aids and is heavily embedded within a personal and cultural context. She noted that research has shown that we cannot infer emotions from facial features alone.

Emotion detection systems have raised concerns among regulators, who have sought to limit their use in high-risk contexts. The AI Act, a major piece of AI legislation in the EU, prohibits schools and employers from deploying emotion detectors, but not law enforcement agencies.

The biggest apprehension around open models like PaliGemma 2 is that they will be abused or misused, leading to real-world harm. Khlaaf warned that if this capability is built on pseudoscientific presumptions, it could have significant implications in how it is used to further discriminate against marginalized groups.

When asked about the dangers of publicly releasing PaliGemma 2, a Google spokesperson emphasized the company's tests for "representational harms" as they relate to visual question answering and captioning. However, Wachter remains unconvinced, stating that responsible innovation requires considering the consequences from the first day of development and throughout the lifecycle of a product.

As the technology continues to evolve, it's essential to weigh the potential benefits of emotion detection against the risks of misuse and bias. With PaliGemma 2, Google has sparked a critical conversation about the responsible development and deployment of AI capabilities.