The Deeper Problem With Google’s Racially Diverse Nazis

Is there a right way for Google’s generative AI to create fake images of Nazis? Apparently so, according to the company. Gemini, Google’s answer to ChatGPT, was shown last week to generate an absurd range of racially and gender-diverse German soldiers styled in Wehrmacht garb. It was, understandably, ridiculed for not generating any images of Nazis who were actually white. Prodded further, it seemed to actively resist generating images of white people altogether. The company ultimately apologized for “inaccuracies in some historical image generation depictions” and paused Gemini’s ability to generate images featuring people.

The situation was played for laughs on the cover of the New York Post and elsewhere, and Google, which did not respond to a request for comment, said it was endeavoring to fix the problem. Google Senior Vice President Prabhakar Raghavan explained in a blog post that the company had intentionally designed its software to produce more diverse representations of people, which backfired. He added, “I can’t promise that Gemini won’t occasionally generate embarrassing, inaccurate or offensive results—but I can promise that we will continue to take action whenever we identify an issue,” which is really the whole situation in a nutshell.

Google—and other generative-AI creators—are trapped in a bind. Generative AI is hyped not because it produces truthful or historically accurate representations: It’s hyped because it allows the general public to instantly produce fantastical images that match a given prompt. Bad actors will always be able to abuse these systems. (See also: AI-generated images of SpongeBob SquarePants flying a plane toward the World Trade Center.) Google may try to inject Gemini with what I would call “synthetic inclusion,” a technological sheen of diversity, but neither the bot nor the data it’s trained on will ever comprehensively mirror reality. Instead, it translates a set of priorities established by product developers into code that engages users—and it does not view them all equally.

[Read: AI’s present matters more than its imagined future]

This is an old problem, one that Safiya Noble diagnosed in her book Algorithms of Oppression. Noble was one of the first to comprehensively describe how modern programs such as those that target online advertisements can “disenfranchise, marginalize, and misrepresent” people on a mass scale. Google products are frequently implicated. In what’s now become a textbook example of algorithmic bias, in 2015, a Black software developer named Jacky Alciné posted a screenshot on Twitter showing that Google Photos’ image-recognition service labeled him and his friends as “gorillas.” That fundamental problem—that the technology can perpetuate racist tropes and biases—was never solved, but rather papered over. Last year—well after that initial incident—a New York Times investigation found that Google Photos still did not allow users “to visually search for primates for fear of making an offensive mistake and labeling a person as an animal.” This appears to still be the case.

“Racially diverse Nazis” and racist mislabeling of Black men as gorillas are two sides of the same coin. In each example, a product is rolled out to a huge user base, only for that user base—rather than Google’s staff—to discover that it contains some racist flaw. The glitches are the legacy of tech companies that are determined to present solutions to problems that people didn’t know existed: the inability to render a visual representation of whatever you can imagine, or to search through thousands of your digital photos for one specific concept.

Inclusion in these systems is a mirage. It does not inherently mean more fairness, accuracy, or justice. In the case of generative AI, the miscues and racist outputs are typically attributed to bad training data, and specifically the lack of diverse data sets that result in the systems reproducing stereotypical or discriminatory content. Meanwhile, people who criticize AI for being too “woke” and want these systems to have the capacity to spit out racist, anti-Semitic, and transphobic content—along with those who don’t trust tech companies to make good decisions about what to allow—complain that any limits on these technologies effectively “lobotomize” the tech. That notion furthers the anthropomorphization of a technology in a way that gives far too much credit to what’s occurring under the hood. These systems do not have a “mind,” a self, or even a sense of right and wrong. Placing safety protocols on AI is “lobotomizing” it in the same way that putting emissions standards or seat belts in a car is stunting its capacity to be human.

[Read: Don’t talk to people like they’re chatbots]

All of this raises the question of what the best-use case for something like Gemini is in the first place. Are we really lacking in sufficient historically accurate depictions of Nazis? Not yet, although these generative-AI products are positioned more and more as gatekeepers to knowledge; we might soon see a world where a service like Gemini both constrains access to and pollutes information. And the definition of AI is expansive; it can in many ways be understood as a mechanism of extraction and surveillance.

We should expect Google—and any generative-AI company—to do better. Yet resolving issues with an image generator that creates oddly diverse Nazis would rely on temporary solutions to a deeper problem: Algorithms inevitably perpetuate one kind of bias or another. When we look to these systems for accurate representation, we are ultimately asking for a pleasing illusion, an excuse to ignore the machinery that crushes our reality into small parts and reconstitutes it into strange shapes.