Témoignages. Dans l’enfer des “nettoyeurs” des réseaux sociaux

Alors que les géants du numérique tentent de renforcer le contrôle sur leurs plateformes, les “modérateurs de contenu” sont exposés à d’innombrables posts violents ou haineux dans le cadre leur travail. Le quotidien japonais “Asahi Shimbun” est allé à leur rencontre.

Publié le 27 juin 2024 à 05h00 Shiori Tabuchi, Azusa Ushio

Ces vidéos prolifèrent sur la Toile. Violences, menaces, actes sexuels… Pourtant, ils n’ont que deux ou trois minutes pour décider de les supprimer ou non.

Nous sommes dans un immeuble, dans une ville d’Asie du Sud-Est. Dans une salle, assis en silence devant leur ordinateur, casque sur les oreilles, des modérateurs de contenu, surnommés “nettoyeurs des réseaux sociaux”, suppriment les publications Internet jugées inappropriées.

Parmi eux, un Japonais, employé par un sous-traitant d’un géant du numérique qui exploite un site de partage de vidéos, a accepté de répondre à nos questions, à condition de ne divulguer ni son nom, ni son âge :

“On m’interdit de parler en détail du contenu de mon travail.”

Il travaille en trois-huit avec des équipes constituées par langue pour un salaire mensuel d’environ 200 000 yens [1 200 euros]. Soumis à une stricte confidentialité, il n’a pas le droit d’apporter son smartphone dans la salle, ni même un simple stylo.

Lorsqu’il arrive à son poste, il allume ses deux écrans. Sur l’un d’eux, une vidéo passe en vitesse rapide. L’autre affiche les nombreuses règles de modération à appliquer, un document qui semble faire un millier de pages. Lorsqu’il repère un contenu proscrit, il classe la vidéo dans une catégorie, par exemple “violence”, “porno”, “harcèlement” ou “haine”. Et cherche la règle qu’elle enfreint et copie cette dernière dans le champ des commentaires. “La chose essentielle est de la trouver aussi vite que possible”, explique-t-il.

Lorsqu’il a fini de vérifier une vidéo, la suivante apparaît. Outre les contenus signalés par des utilisateurs, “il y a probablement des publications détectées automatiquement par l’intelligence artificielle (IA), mais je ne sais pas comment elles sont choisies”.

Jeu du chat et de la souris

Si une vidéo montre une personne battue jusqu’au sang ou contient des menaces du genre “Je vais le tuer”, il la supprime immédiatement. En cas de doute, il envoie la vidéo à un service spécialisé. Sur les quelque 80 vidéos qu’il visionne par jour, il en supprime environ trois. Il y en a également une dizaine qu’il trouve difficiles à juger. Il ignore combien il y a de services au total, et qui prend les décisions en définitive. “Je procède de manière mécanique”, confie-t-il.

Il se souvient d’un pic d’activité après l’assassinat par balle de l’ancien Premier ministre Shinzo Abe [en juillet 2022]. Des images de la scène ont été publiées à de nombreuses reprises. “J’effaçais les vidéos non floutées pratiquement les unes après les autres.”

Les règles de modération sont nombreuses et détaillées, et les changements sont annoncés chaque semaine lors de réunions matinales. Est également fournie une base de données rassemblant les mots tabous. À la fin de chaque journée de travail, les modérateurs passent un test visant à évaluer leur connaissance des dernières règles : ceux qui obtiennent un faible score voient leur salaire réduit.

Les vidéos supprimées sont fréquemment republiées, et certains contenus passent entre les mailles du filet. Notre modérateur est conscient des critiques :

“Nous faisons de notre mieux, mais c’est comme le jeu du chat et de la souris. Nous ne pouvons pas effacer toutes les vidéos. Celles qui ne sont pas signalées restent.”

Le géant du numérique qui assure ce service de modération soutenait autrefois qu’il ne faisait que fournir un “lieu” d’expression et n’était pas responsable des contenus publiés. Mais la prolifération des publications nuisibles l’a contraint à réagir et à renforcer sa surveillance.

Le règlement sur les services numériques (Digital Services Act, DSA), adopté par l’Union européenne (UE), oblige aujourd’hui les grandes plateformes Internet à supprimer les publications nuisibles, notamment les contenus discriminatoires et les fausses informations. Si beaucoup sont supprimées automatiquement par l’IA, certaines nécessitent une intervention humaine. Selon les rapports que la Commission européenne a demandé aux géants du numériques de présenter en octobre dernier, Facebook a supprimé en Europe près de 47 millions de contenus contrevenant à la réglementation au cours des cinq mois qui ont suivi la fin avril 2023. Et 2,83 millions d’entre eux, soit 6 %, ont été supprimés par des modérateurs.

“Soldats des réseaux”

Facebook emploie environ 15 000 modérateurs et X environ 2 300. TikTok en compte environ 40 000, chargés notamment de contrôler les vidéos populaires qui dépassent un certain nombre de vues et de supprimer celles qui posent problème.

“Les modérateurs sont les soldats qui œuvrent dans l’ombre des réseaux sociaux”, estime Kauna Malgwi, 30 ans, qui vit aujourd’hui à Abuja, la capitale du Nigeria. Il y a cinq ans, alors qu’elle était une mère célibataire en situation précaire, elle est partie étudier au Kenya. Elle y a accepté ce qui était présenté comme un “poste d’interprète dans un ‘service clientèle’ utilisant le haoussa”, l’une des langues qui comptent le plus grand nombre de locuteurs en Afrique de l’Ouest. En réalité, elle s’est retrouvée modératrice pour Meta, qui exploite Facebook et Instagram. En parallèle à ses études de troisième cycle, pendant environ quatre ans, jusqu’en mars 2023, elle a travaillé neuf heures par jour, cinq jours par semaine, pour la succursale kenyane d’un sous-traitant du géant du numérique américain.

Expérience traumatisante

La première vidéo qu’elle a visionnée montrait un homme chutant du 15e étage d’un immeuble. Devant l’effroyable spectacle du corps s’écrasant au sol, elle a sauté de sa chaise. Elle devait remplir un questionnaire pyramidal énonçant les motifs de suppression du haut vers le bas. Après avoir répondu par la négative à la première question – “Voit-on des corps nus ?” –, elle a coché les cases “Voit-on des viscères ?” et “Voit-on du sang ?”.

Agressions sexuelles sur des enfants en bas âge, exécutions par des groupes extrémistes, suicides par balle… Chaque jour, elle examinait un millier de vidéos, détectées par l’IA ou signalées par des utilisateurs, et avait un maximum de cinquante-cinq secondes par vidéo pour décider de leur suppression ou non.

Elle supprimait également des textes à caractère raciste et d’autres messages de haine contenant des mots spécifiques.

“Il n’y avait pas que les textes. Par exemple, un dessin représentant un Asiatique et un singe côte à côte avec la légende ‘deux frères’ devait être supprimé.”

Elle a même supprimé des contenus publiés en Asie du Sud-Est, à plusieurs milliers de kilomètres de là.

Elle gagnait 60 000 shillings kényans (environ 400 euros) par mois, ce qui correspond au revenu mensuel moyen au Kenya. Mais elle souffrait à la fois d’insomnie et de trouble panique, ce qui l’a conduite plusieurs fois à l’hôpital.

Les accords de confidentialité ne lui ont même pas permis de se confier à sa famille. Ses collègues, les seuls avec lesquels elle pouvait partager ses sentiments, fumaient du cannabis pendant leurs pauses pour échapper à la réalité. Certains ont même avoué envisager le suicide. “C’est certes un travail important de protéger les nombreux utilisateurs de ces institutions que sont devenus les réseaux sociaux, mais quand même…” Aujourd’hui encore, il lui arrive de pleurer en repensant aux images qu’elle a vues.

Deluge of ‘pink slime’ websites threaten to drown out truth with fake news in US election

US sites pushing misinformation are proliferating, aiming to look like reliable sources as local newspapers close down

Eric Berger Thu 20 Jun 2024 12.00 CEST

Political groups on the right and left are using fake news websites designed to look like reliable sources of information to fill the void left by the demise of local newspapers, raising fears of the impact that they might have during the United States’ bitterly fought 2024 election.

Some media experts are concerned that the so-called pink slime websites, often funded domestically, could prove at least as harmful to political discourse and voters’ faith in media and democracy as foreign disinformation efforts in the 2016 and 2020 presidential elections.

According to a recent report from NewsGuard, a company that aims to counter misinformation by studying and rating news websites, the websites are so prolific that “the odds are now better than 50-50 that if you see a news website purporting to cover local news, it’s fake.”

NewsGuard estimates that there are a staggering 1,265 such fake local news websites in the US – 4% more than the websites of 1,213 daily newspapers left operating in the country.

“Actors on both sides of the political spectrum” feel “that what they are doing isn’t bad because all media is really biased against their side or that that they know actors on the other side are using these tactics and so they feel they need to,” said Matt Skibinski, general manager of NewsGuard, which determined that such sites now outnumber legitimate local news organizations. “It’s definitely contributed to partisanship and the erosion of trust in media; it’s also a symptom of those things.”

Pink slime websites, named after a meat byproduct, started at least as early as 2004 when Brian Timpone, a former television reporter who described himself as a “biased guy” and a Republican, started funding websites featuring names of cities, towns and regions like the Philly Leader and the South Alabama Times.

Timpone’s company, Metric Media, now operates more than 1,000 such websites and his private equity company receives funding from conservative political action committees, according to NewsGuard.

The Leader recently ran a story with the headline, “Rep Evans votes to count illegal aliens towards seats in Congress.”

In actuality, Representative Dwight Evans, a Democrat, did not vote to start counting undocumented immigrants in the 2030 census but rather against legislation that would have changed the way the country has conducted apportionment since 1790.

That sort of story is “standard practice for these outlets”, according to Tim Franklin, who leads Northwestern University’s Local News Initiative, which researches the industry.

“They will take something that maybe has just a morsel of truth to it and then twist it with their own partisan or ideological spin,” Franklin said. “They also tend to do it on issues like immigration or hot-button topics that they think will elicit an emotional response.”

A story published this month on the NW Arkansas News site had a headline on the front page that reported that the unemployment rate in 2021 in Madison county was 5.1% – even though there is much more recent data available. In April 2024, the local unemployment rate was 2.5%.

“Another tactic that we have seen across many of this category of sites is taking a news story that happened at some point and presenting it as if it just happened now, in a way that is misleading,” Skibinski said.

The left has also created websites designed to look like legitimate news organizations but actually shaped by Democratic supporters.

The liberal Courier Newsroom network operates websites in Arizona, Florida, Iowa, Michigan and Nevada, among other states, that – like the conservative pink slime sites – have innocuous sounding names like the Copper Courier and Up North News. The Courier has runs stories like “Gov Ducey Is Now the Most Unpopular Governor in America,” referring to Doug Ducy, the former Republican Arizona governor.

“In contrast, coverage of Democrats, including US President Joe Biden, Democratic Arizona Gov Katie Hobbs, and US Sen Mark Kelly of Arizona, is nearly always laudatory,” NewsGuard stated in a report about Courier coverage.

Tara McGowan, a Democratic strategist who founded the Courier Newsroom has received funding from liberal donors like Reid Hoffman and George Soros, as well as groups associated with political action committees, according to NewsGuard.

“There are pink slime operations on both the right and the left. To me, the key is disclosure and transparency about ownership,” said Franklin.

In a statement, a spokesperson for the Courier said comparisons between its operations and rightwing pink slime groups were unfair and criticized NewsGuard’s methodology in comparing the two.

“Courier publishes award-winning, factual local news by talented journalists who live in the communities we cover, and our reporting is often cited by legacy media outlets. This is in stark contrast to the pink slime networks that pretend to have a local presence but crank out low-quality fake news with no bylines and no accountability. Courier is proudly transparent about our pro-democracy values, and we carry on the respected American tradition of advocacy journalism,” the spokesperson said.

While both the left and the right have invested in the pink slime websites, there are differences in the owners’ approaches, according to Skibinski.

The right-wing networks have created more sites “that are probably getting less attention per site, and on the left, there is a smaller number of sites, but they are more strategic about getting attention to those sites on Facebook and elsewhere”, Skibinski said. “I don’t know that we can quantify whether one is more impactful than the other.”

Artificial intelligence could also help site operators quickly generate stories and create fake images.

“The technology underlying artificial intelligence is now becoming more accessible to malign actors,” said Kathleen Hall Jamieson, a University of Pennsylvania communications professor and director of the Annenberg Public Policy Center, which publishes Factcheck.org. “The capacity to create false images is very high, but also there is a capacity to detect the images that is emerging very rapidly. The question is, will it emerge rapidly with enough capacity?”

Still, it’s not clear whether these websites are effective. Stanford University reported in a 2023 study that engagement with pink slime websites was “relatively low” and little evidence that living “in a news desert made people more likely to consume pink slime”.

The Philly Leader and the NW Arkansas News both only have links to Facebook accounts on their websites and have less than 450 followers on each. Meanwhile, the Copper Courier and Up North News have accounts on all the major platforms and a total of about 150,000 followers on Facebook.

Franklin said he thinks that a lot of people don’t actually click links on social media posts to visit the website.

“The goal of some of these operators is not to get traffic directly to their site, but it’s to go viral on social media,” he said.

Republican lawmakers and leaders of the conservative news sites the Daily Wire and the Federalist have also filed a lawsuit and launched investigations accusing NewsGuard of helping the federal government censor right-leaning media. The defense department hired the company strictly to counter “disinformation efforts by Russian, Chinese and Iranian government-linked operations targeting Americans and our allies”, Gordon Crovitz, the former Wall Street Journal publisher who co-founded NewsGuard, told the Hill in response to a House oversight committee investigation. “We look forward to clarifying the misunderstanding by the committee about our work for the Defense Department.”

To counter the flood of misinformation, social media companies must take a more active role in monitoring such content, according to Franklin and Skibinski.

“The biggest solution to this kind of site would be for the social media platforms to take more responsibility in terms of showing context to the user about sources that could be their own context. It could be data from third parties, like what we do,” said Skibinski.

Franklin would like to see a national media literacy campaign. States around the country have passed laws requiring such education in schools.

Franklin also hopes that legitimate local news could rebound. The MacArthur Foundation and other donors last year pledged $500m to help local outlets.

“I actually have more optimism now than I had a few years ago,” Franklin said. “We’re in the midst of historic changes in how people consume news and how it’s produced and how it’s distributed and how it’s paid for, but I think there’s still demand for local news, and that’s kind of where it all starts.”

Scientists Are Getting Eerily Good at Using WiFi to 'See' People Through Walls in Detail

The signals from WiFi can be used to map a human body, according to a new paper.

January 17, 2023, 7:50pm

Researchers at Carnegie Mellon University developed a method for detecting the three dimensional shape and movements of human bodies in a room, using only WiFi routers.

To do this, they used DensePose, a system for mapping all of the pixels on the surface of a human body in a photo. DensePose was developed by London-based researchers and Facebook’s AI researchers. From there, according to their recently-uploaded preprint paper published on arXiv, they developed a deep neural network that maps WiFi signals’ phase and amplitude sent and received by routers to coordinates on human bodies.

Researchers have been working on “seeing” people without using cameras or expensive LiDAR hardware for years. In 2013, a team of researchers at MIT found a way to use cell phone signals to see through walls; in 2018, another MIT team used WiFi to detect people in another room and translate their movements to walking stick-figures.

The Carnegie Mellon researchers wrote that they believe WiFi signals “can serve as a ubiquitous substitute” for normal RGB cameras, when it comes to “sensing” people in a room. Using WiFi, they wrote, overcomes obstacles like poor lighting and occlusion that regular camera lenses face.

Interestingly, they position this advancement as progress in privacy rights; “In addition, they protect individuals’ privacy and the required equipment can be bought at a reasonable price,” they wrote. “In fact, most households in developed countries already have WiFi at home, and this technology may be scaled to monitor the well-being of elder people or just identify suspicious behaviors at home.”

They don’t mention what “suspicious behaviors” might include, if this technology ever hits the mainstream market. But considering companies like Amazon are trying to put Ring camera drones inside our houses, it’s easy to imagine how widespread WiFi-enabled human-detection could be a force for good—or yet another exploitation of all of our privacy.

DensePose From WiFi

Jiaqi Geng, Dong Huang, Fernando De la Torre 31 Dec 2022

Abstract

Advances in computer vision and machine learning techniques have
led to significant development in 2D and 3D human pose estimation
from RGB cameras, LiDAR, and radars. However, human pose esti-
mation from images is adversely affected by occlusion and lighting,
which are common in many scenarios of interest. Radar and LiDAR
technologies, on the other hand, need specialized hardware that is
expensive and power-intensive. Furthermore, placing these sensors
in non-public areas raises significant privacy concerns.

To address these limitations, recent research has explored the use
of WiFi antennas (1D sensors) for body segmentation and key-point
body detection. This paper further expands on the use of the WiFi
signal in combination with deep learning architectures, commonly
used in computer vision, to estimate dense human pose correspon-
dence. We developed a deep neural network that maps the phase
and amplitude of WiFi signals to UV coordinates within 24 human
regions. The results of the study reveal that our model can estimate
the dense pose of multiple subjects, with comparable performance
to image-based approaches, by utilizing WiFi signals as the only
input. This paves the way for low-cost, broadly accessible, and
privacy-preserving algorithms for human sensing.

Densepose

Official website of densepose

McDonald's AI Drive-Thru debacle is a warning to us all

By Joe Foley published 5 hours ago

Did it not test this?

We've mentioned before the risks for brands jumping on the AI bandwagon too quickly. And that extends beyond using AI image generators to any kind of customer-facing application, as McDonald's may have learned from its AI Drive Thru fiasco.

AI technology is advancing rapidly but remains in a state of relative infancy, and in many cases it just isn't good enough yet to implement without causing significant friction. The world's biggest fastfood brand has sensibly decided not to extend the contract on an AI voice recognition service and has told franchisees to remove the tech, but did it not think it should at least test it before it became the subject of viral videos?

Developed by IBM, McDonald's AI ordering system was implemented in over 100 McDonald's locations in the US starting back 2021. It was supposed to use voice recognition to process orders, but customers reported frequent frustrations, including quite spectacular order mixups, from bacon being added to ice cream to orders being hugely inflated.

In one video shared on TikTok with the caption "Fighting with McDonald's robot", the AI interpreted a woman's request for vanilla ice cream and a bottle of water to be an order for a caramel sundae and multiple sachets of ketchup and butter. In another, a customer inadvertently ordered 2,510 McNuggets Meals. That left a human attendant to have to reinput the order, rendering the AI a pointless frustration.

As reported by the trade publication Restaurant Business, McDonald's is removing the tech but remains determined to push forward with voice recognition technology to avoid having to employ humans to do the job of taking orders. The company said in a statement: "While there have been successes to date, we feel there is an opportunity to explore voice ordering solutions more broadly.

"After a thoughtful review, McDonald's has decided to end our current partnership with IBM on AOT (automated order taking) and the technology will be shut off in all restaurants currently testing it no later than 26 July, 2024."

This is far from the first case we've seen of experiments with AI resulting in a customer backlash. Lego is one of several brands to have apologised after using AI imagery. We've also seen branding agencies warn against AI washing, which is a tendency for companies to overstate their AI capabilities in order to make themselves look like part of the zeitgeist.

Here lies the internet, murdered by generative AI

Corruption everywhere, even in YouTube's kids content

Erik Hoel Feb 27, 2024

Art for The Intrinsic Perspective is by Alexander Naughton

The amount of AI-generated content is beginning to overwhelm the internet. Or maybe a better term is pollute. Pollute its searches, its pages, its feeds, everywhere you look. I’ve been predicting that generative AI would have pernicious effects on our culture since 2019, but now everyone can feel it. Back then I called it the coming “semantic apocalypse.” Well, the semantic apocalypse is here, and you’re being affected by it, even if you don’t know it. A minor personal example: last year I published a nonfiction book, The World Behind the World, and now on Amazon I find this.

What, exactly, are these “workbooks” for my book? AI pollution. Synthetic trash heaps floating in the online ocean. The authors aren’t real people, some asshole just fed the manuscript into an AI and didn’t check when it spit out nonsensical summaries. But it doesn’t matter, does it? A poor sod will click on the $9.99 purchase one day, and that’s all that’s needed for this scam to be profitable since the process is now entirely automatable and costs only a few cents. Pretty much all published authors are affected by similar scams, or will be soon.

Now that generative AI has dropped the cost of producing bullshit to near zero, we see clearly the future of the internet: a garbage dump. Google search? They often lead with fake AI-generated images amid the real things. Post on Twitter? Get replies from bots selling porn. But that’s just the obvious stuff. Look closely at the replies to any trending tweet and you’ll find dozens of AI-written summaries in response, cheery Wikipedia-style repeats of the original post, all just to farm engagement. AI models on Instagram accumulate hundreds of thousands of subscribers and people openly shill their services for creating them. AI musicians fill up YouTube and Spotify. Scientific papers are being AI-generated. AI images mix into historical research. This isn’t mentioning the personal impact too: from now on, every single woman who is a public figure will have to deal with the fact that deepfake porn of her is likely to be made. That’s insane.

And rather than this being pure skullduggery, people and institutions are willing to embrace low-quality AI-generated content, trying to shift the Overton window to make things like this acceptable:

source

That’s not hardball capitalism. That’s polluting our culture for your own minor profit. It’s not morally legitimate for the exact same reasons that polluting a river for a competitive edge is not legitimate. Yet name-brand media outlets are embracing generative AI just like SEO-spammers are, for the same reasons.

E.g., investigative work at Futurism caught Sports Illustrated red-handed using AI-generated articles written by fake writers. Meet Drew Ortiz.

He doesn’t exist. That face is an AI-generated portrait, which was previously listed for sale on a website. As Futurism describes:

Ortiz isn't the only AI-generated author published by Sports Illustrated, according to a person involved with the creation of the content…

"At the bottom [of the page] there would be a photo of a person and some fake description of them like, 'oh, John lives in Houston, Texas. He loves yard games and hanging out with his dog, Sam.' Stuff like that," they continued. "It's just crazy."

This isn’t what everyone feared, which is AI replacing humans by being better—it’s replacing them because AI is so much cheaper. Sports Illustrated was not producing human-quality level content with these methods, but it was still profitable.

The AI authors' writing often sounds like it was written by an alien; one Ortiz article, for instance, warns that volleyball "can be a little tricky to get into, especially without an actual ball to practice with."

Sports Illustrated, in a classy move, deleted all the evidence. Drew was replace by Sora Tanaka, bearing a face also listed for sale on the same website with the description of a “joyful asian young-adult female with long brown hair and brown eyes.”

Given that even prestigious outlets like The Guardian refuse to put any clear limits on their use of AI, if you notice odd turns of phrase or low-quality articles, the likelihood that they’re written by an AI, or with AI-assistance, is now high.

Sadly, the people affected the most by generative AI are the ones who can’t defend themselves. Because they don’t even know what AI is. Yet we’ve abandoned them to swim in polluted information currents. I’m talking, unfortunately, about toddlers. Because let me introduce you to…

the hell that is AI-generated children’s YouTube content.

YouTube for kids is quickly becoming a stream of synthetic content. Much of it now consists of wooden digital characters interacting in short nonsensical clips without continuity or purpose. Toddlers are forced to sit and watch this runoff because no one is paying attention. And the toddlers themselves can’t discern that characters come and go and that the plots don’t make sense and that it’s all just incoherent dream-slop. The titles don’t match the actual content, and titles that are all the parents likely check, because they grew up in a culture where if a YouTube video said BABY LEARNING VIDEOS and had a million views it was likely okay. Now, some of the nonsense AI-generated videos aimed at toddlers have tens of millions of views.

Here’s a behind-the-scenes video on a single channel that made 1.2 million dollars via AI-generated “educational content” aimed at toddlers.

As the video says:

These kids, when they watch these kind of videos, they watch them over and over and over again.

They aren’t confessing. They’re bragging. And the particular channel they focus on isn’t even the worst offender—at least that channel’s content mostly matches the subheadings and titles, even if the videos are jerky, strange, off-putting, repetitious, clearly inhuman. Other channels, which are also obviously AI-generated, get worse and worse. Here’s a “kid’s education” channel that is AI-generated (took about one minute to find) with 11.7 million subscribers.

They don’t use proper English, and after quickly going through some shapes like the initial video title promises (albeit doing it in a way that makes you feel like you’re going insane) the rest of the video devolves into randomly-generated rote tasks, eerie interactions, more incorrect grammar, and uncanny musical interludes of songs that serve no purpose but to pad the time. It is the creation of an alien mind.

Here’s an example of the next frontier: completely start-to-finish AI-generated music videos for toddlers. Below is a how-to video for these new techniques. The result? Nightmarish parrots with twisted double-beaks and four mutated eyes singing artificial howls from beyond. Click and behold (or don’t, if you want to sleep tonight).

All around the nation there are toddlers plunked down in front of iPads being subjected to synthetic runoff, deprived of human contact even in the media they consume. There’s no other word but dystopian. Might not actual human-generated cultural content normally contain cognitive micro-nutrients (like cohesive plots and sentences, detailed complexity, reasons for transitions, an overall gestalt, etc) that the human mind actually needs? We’re conducting this experiment live. For the first time in history developing brains are being fed choppy low-grade and cheaply-produced synthetic data created en masse by generative AI, instead of being fed with real human culture. No one knows the effects, and no one appears to care. Especially not the companies, because…

OpenAI has happily allowed pollution.

Why blame them, specifically? Well, first of all, their massive impact—e.g., most of the kids videos are built from scripts generated by ChatGPT. And more generally, what AI capabilities are considered okay to deploy has long been a standard set by OpenAI. Despite their supposed safety focus, OpenAI failed to foresee that its creations would thoroughly pollute the internet across all platforms and services. You can see this failure in how they assessed potential negative outcomes in the announcement of GPT-2 on their blog, back in 2019. While they did warn that these models could have serious longterm consequences for the information ecosystem, the specifics they were concerned with were things like:

Generate misleading news articles

Impersonate others online

Automate the production of abusive or faked content to post on social media

Automate the production of spam/phishing content

This may sound kind of in line with what’s happened, but if you read further, it becomes clear that what they meant by “faked content” was mainly malicious actors promoting misinformation, or the same shadowy malicious actors using AI to phish for passwords, etc.

These turned out to be only minor concerns compared to AI’s cultural pollution. OpenAI kept talking about “actors” when they should have been talking about “users.” Because it turns out, all AI-generated content is fake! Or it’s all kind of fake. AI-written websites, now sprouting up like an unstoppable invasive species, don’t necessarily have an intent to mislead; it’s just that AI content is low-effort banalities generated for pennies, so you can SEO spam and do all sorts of manipulative games around search to attract eyeballs and ad revenue.

That is, the OpenAI team didn’t stop to think that regular users just generating mounds of AI-generated content on the internet would have very similar negative effects to as if there were a lot of malicious use by intentional bad actors. Because there’s no clear distinction! The fact that OpenAI was both honestly worried about negative effects, and at the same time didn’t predict the enshittification of the internet they spearheaded, should make us extremely worried they will continue to miss the negative downstream effects of their increasingly intelligent models. They failed to foresee the floating mounds of clickbait garbage, the synthetic info-trash cities, all to collect clicks and eyeballs—even from innocent children who don’t know any better. And they won’t do anything to stop it, because…

AI pollution is a tragedy of the commons.

This term, "tragedy of the commons,” originated in the rising environmentalism of the 20th century, and would lead to many of the regulations that keep our cities free of smog and our rivers clean. Garrett Hardin, an ecologist and biologist, coined it in an article in [Science](https://math.uchicago.edu/~shmuel/Modeling/Hardin, Tragedy of the Commons.pdf) in 1968. The article is still instructively relevant. Hardin wrote:

An implicit and almost universal assumption of discussions published in professional and semipopular scientific journals is that the problem under discussion has a technical solution…

He goes on to discuss several problems for which there are no technical solutions, since rational actors will drive the system toward destruction via competition:

The tragedy of the commons develops in this way. Picture a pasture open to all. It is to be expected that each herdsman will try to keep as many cattle as possible on the commons. Such an arrangement may work reasonably satisfactorily for centuries because tribal wars, poaching, and disease keep the numbers of both man and beast well below the carrying capacity of the land. Finally, however, comes the day of reckoning, that is, the day when the long-desired goal of social stability becomes a reality. At this point, the inherent logic of the commons remorselessly generates tragedy.

One central example of Hardin’s became instrumental to the environmental movement.

… the tragedy of the commons reappears in problems of pollution. Here it is not a question of taking something out of the commons, but of putting something in—sewage, or chemical, radioactive, and heat wastes into water; noxious and dangerous fumes into the air; and distracting and unpleasant advertising signs into the line of sight. The calculations of utility are much the same as before. The rational man finds that his share of the cost of the wastes he discharges into the commons is less than the cost of purifying his wastes before releasing them. Since this is true for everyone, we are locked into a system of "fouling our own nest," so long as we behave only as independent, rational, free-enterprisers.

We are currently fouling our own nests. Since the internet economy runs on eyeballs and clicks the new ability of anyone, anywhere, to easily generate infinite low-quality content via AI is now remorselessly generating tragedy.

The solution, as Hardin noted, isn’t technical. You can’t detect AI outputs reliably anyway (another initial promise that OpenAI abandoned). The companies won’t self regulate, given their massive financial incentives. We need the equivalent of a Clean Air Act: a Clean Internet Act. We can’t just sit by and let human culture end up buried.

Luckily we’re on the cusp of all that incredibly futuristic technology promised by AI. Any day now, our GDP will start to rocket forward. In fact, soon we’ll cure all disease, even aging itself, and have robot butlers and Universal Basic Income and high-definition personalized entertainment. Who cares if toddlers had to watch inhuman runoff for a few billion years of viewing-time to make the future happen? It was all worth it. Right? Let’s wait a little bit longer. If we wait just a little longer utopia will surely come.

Underage Workers Are Training AI

Companies that provide Big Tech with AI data-labeling services are inadvertently hiring young teens to work on their platforms, often exposing them to traumatic content.

Like most kids his age, 15-year-old Hassan spent a lot of time online. Before the pandemic, he liked playing football with local kids in his hometown of Burewala in the Punjab region of Pakistan. But Covid lockdowns made him something of a recluse, attached to his mobile phone. “I just got out of my room when I had to eat something,” says Hassan, now 18, who asked to be identified under a pseudonym because he was afraid of legal action. But unlike most teenagers, he wasn’t scrolling TikTok or gaming. From his childhood bedroom, the high schooler was working in the global artificial intelligence supply chain, uploading and labeling data to train algorithms for some of the world’s largest AI companies.

The raw data used to train machine-learning algorithms is first labeled by humans, and human verification is also needed to evaluate their accuracy. This data-labeling ranges from the simple—identifying images of street lamps, say, or comparing similar ecommerce products—to the deeply complex, such as content moderation, where workers classify harmful content within data scraped from all corners of the internet. These tasks are often outsourced to gig workers, via online crowdsourcing platforms such as Toloka, which was where Hassan started his career.

A friend put him on to the site, which promised work anytime, from anywhere. He found that an hour’s labor would earn him around $1 to $2, he says, more than the national minimum wage, which was about $0.26 at the time. His mother is a homemaker, and his dad is a mechanical laborer. “You can say I belong to a poor family,” he says. When the pandemic hit, he needed work more than ever. Confined to his home, online and restless, he did some digging, and found that Toloka was just the tip of the iceberg.

“AI is presented as a magical box that can do everything,” says Saiph Savage, director of Northeastern University’s Civic AI Lab. “People just simply don’t know that there are human workers behind the scenes.”

At least some of those human workers are children. Platforms require that workers be over 18, but Hassan simply entered a relative’s details and used a corresponding payment method to bypass the checks—and he wasn’t alone in doing so. WIRED spoke to three other workers in Pakistan and Kenya who said they had also joined platforms as minors, and found evidence that the practice is widespread.

“When I was still in secondary school, so many teens discussed online jobs and how they joined using their parents' ID,” says one worker who joined Appen at 16 in Kenya, who asked to remain anonymous. After school, he and his friends would log on to complete annotation tasks late into the night, often for eight hours or more.

Appen declined to give an attributable comment.

“If we suspect a user has violated the User Agreement, Toloka will perform an identity check and request a photo ID and a photo of the user holding the ID,” Geo Dzhikaev, head of Toloka operations, says.

Driven by a global rush into AI, the global data labeling and collection industry is expected to grow to over $17.1 billion by 2030, according to Grand View Research, a market research and consulting company. Crowdsourcing platforms such as Toloka, Appen, Clickworker, Teemwork.AI, and OneForma connect millions of remote gig workers in the global south to tech companies located in Silicon Valley. Platforms post micro-tasks from their tech clients, which have included Amazon, Microsoft Azure, Salesforce, Google, Nvidia, Boeing, and Adobe. Many platforms also partner with Microsoft’s own data services platform, the Universal Human Relevance System (UHRS).

These workers are predominantly based in East Africa, Venezuela, Pakistan, India, and the Philippines—though there are even workers in refugee camps, who label, evaluate, and generate data. Workers are paid per task, with remuneration ranging from a cent to a few dollars—although the upper end is considered something of a rare gem, workers say. “The nature of the work often feels like digital servitude—but it's a necessity for earning a livelihood,” says Hassan, who also now works for Clickworker and Appen.

Sometimes, workers are asked to upload audio, images, and videos, which contribute to the data sets used to train AI. Workers typically don’t know exactly how their submissions will be processed, but these can be pretty personal: On Clickworker’s worker jobs tab, one task states: “Show us you baby/child! Help to teach AI by taking 5 photos of your baby/child!” for €2 ($2.15). The next says: “Let your minor (aged 13-17) take part in an interesting selfie project!”

Some tasks involve content moderation—helping AI distinguish between innocent content and that which contains violence, hate speech, or adult imagery. Hassan shared screen recordings of tasks available the day he spoke with WIRED. One UHRS task asked him to identify “fuck,” “c**t,” “dick,” and “bitch” from a body of text. For Toloka, he was shown pages upon pages of partially naked bodies, including sexualized images, lingerie ads, an exposed sculpture, and even a nude body from a Renaissance-style painting. The task? Decipher the adult from the benign, to help the algorithm distinguish between salacious and permissible torsos.

Hassan recalls moderating content while under 18 on UHRS that, he says, continues to weigh on his mental health. He says the content was explicit: accounts of rape incidents, lifted from articles quoting court records; hate speech from social media posts; descriptions of murders from articles; sexualized images of minors; naked images of adult women; adult videos of women and girls from YouTube and TikTok.

Many of the remote workers in Pakistan are underage, Hassan says. He conducted a survey of 96 respondents on a Telegram group chat with almost 10,000 UHRS workers, on behalf of WIRED. About a fifth said they were under 18.

Awais, 20, from Lahore, who spoke on condition that his first name not be published, began working for UHRS via Clickworker at 16, after he promised his girlfriend a birthday trip to the turquoise lakes and snow-capped mountains of Pakistan’s northern region. His parents couldn’t help him with the money, so he turned to data work, joining using a friend’s ID card. “It was easy,” he says.

He worked on the site daily, primarily completing Microsoft’s “Generic Scenario Testing Extension” task. This involved testing homepage and search engine accuracy. In other words, did selecting “car deals” on the MSN homepage bring up photos of cars? Did searching “cat” on Bing show feline images? He was earning $1 to $3 each day, but he found the work both monotonous and infuriating. At times he found himself working 10 hours for $1, because he had to do unpaid training to access certain tasks. Even when he passed the training, there might be no task to complete; or if he breached the time limit, they would suspend his account, he says. Then seemingly out of nowhere, he got banned from performing his most lucrative task—something workers say happens regularly. Bans can occur for a host of reasons, such as giving incorrect answers, answering too fast, or giving answers that deviate from the average pattern of other workers. He’d earned $70 in total. It was almost enough to take his high school sweetheart on the trip, so Awais logged off for good.

Clickworker did not respond to requests for comment. Microsoft declined to comment.

“In some instances, once a user finishes the training, the quota of responses has already been met for that project and the task is no longer available,” Dzhikaev said. “However, should other similar tasks become available, they will be able to participate without further training.”

Researchers say they’ve found evidence of underage workers in the AI industry elsewhere in the world. Julian Posada, assistant professor of American Studies at Yale University, who studies human labor and data production in the AI industry, says that he’s met workers in Venezuela who joined platforms as minors.

Bypassing age checks can be pretty simple. The most lenient platforms, like Clickworker and Toloka, simply ask workers to state they are over 18; the most secure, such as Remotasks, employ face recognition technology to match workers to their photo ID. But even that is fallible, says Posada, citing one worker who says he simply held the phone to his grandmother’s face to pass the checks. The sharing of a single account within family units is another way minors access the work, says Posada. He found that in some Venezuelan homes, when parents cook or run errands, children log on to complete tasks. He says that one family of six he met, with children as young as 13, all claimed to share one account. They ran their home like a factory, Posada says, so that two family members were at the computers working on data labeling at any given point. “Their backs would hurt because they have been sitting for so long. So they would take a break, and then the kids would fill in,” he says.

The physical distances between the workers training AI and the tech giants at the other end of the supply chain—“the deterritorialization of the internet,” Posada calls it—creates a situation where whole workforces are essentially invisible, governed by a different set of rules, or by none.

The lack of worker oversight can even prevent clients from knowing if workers are keeping their income. One Clickworker user in India, who requested anonymity to avoid being banned from the site, told WIRED he “employs” 17 UHRS workers in one office, providing them with a computer, mobile, and internet, in exchange for half their income. While his workers are aged between 18 and 20, due to Clickworker’s lack of age certification requirements, he knows of teenagers using the platform.

In the more shadowy corners of the crowdsourcing industry, the use of child workers is overt.

Captcha (Completely Automated Public Turing test to tell Computers and Humans Apart) solving services, where crowdsourcing platforms pay humans to solve captchas, are a less understood part in the AI ecosystem. Captchas are designed to distinguish a bot from a human—the most notable example being Google’s reCaptcha, which asks users to identify objects in images to enter a website. The exact purpose of services that pay people to solve them remains a mystery to academics, says Posada. “But what I can confirm is that many companies, including Google's reCaptcha, use these services to train AI models,” he says. “Thus, these workers indirectly contribute to AI advancements.”

Google did not respond to a request for comment in time for publication.

There are at least 152 active services, mostly based in China, with more than half a million people working in the underground reCaptcha market, according to a 2019 study by researchers from Zhejiang University in Hangzhou.

“Stable job for everyone. Everywhere,” one service, Kolotibablo, states on its website. The company has a promotional website dedicated to showcasing its worker testimonials, which includes images of young children from across the world. In one, a smiling Indonesian boy shows his 11th birthday cake to the camera. “I am very happy to be able to increase my savings for the future,” writes another, no older than 7 or 8. A 14-year-old girl in a long Hello Kitty dress shares a photo of her workstation: a laptop on a pink, Barbie-themed desk.

Not every worker WIRED interviewed felt frustrated with the platforms. At 17, most of Younis Hamdeen’s friends were waiting tables. But the Pakistani teen opted to join UHRS via Appen instead, using the platform for three or four hours a day, alongside high school, earning up to $100 a month. Comparing products listed on Amazon was the most profitable task he encountered. “I love working for this platform,” Hamdeen, now 18, says, because he is paid in US dollars—which is rare in Pakistan—and so benefits from favorable exchange rates.

But the fact that the pay for this work is incredibly low compared to the wages of in-house employees of the tech companies, and that the benefits of the work flow one way—from the global south to the global north, leads to uncomfortable parallels. “We do have to consider the type of colonialism that is being promoted with this type of work,” says the Civic AI Lab’s Savage.

Hassan recently got accepted to a bachelor’s program in medical lab technology. The apps remain his sole income, working an 8 am to 6 pm shift, followed by 2 am to 6 am. However, his earnings have fallen to just $100 per month, as demand for tasks has outstripped supply, as more workers have joined since the pandemic.

He laments that UHRS tasks can pay as little as 1 cent. Even on higher-paid jobs, such as occasional social media tasks on Appen, the amount of time he needs to spend doing unpaid research means he needs to work five or six hours to complete an hour of real-time work, all to earn $2, he says.

“It’s digital slavery,” says Hassan.

Amazon buys nuclear-powered data centre from Talen

US-based Talen Energy Corporation has sold its Cumulus data centre campus in Pennsylvania to Amazon subsidiary Amazon Web Services (AWS) for $650m. This includes a long-term agreement to provide power from Talen's Susquehanna NPP. The 2,500 MWe adjacent Susquehanna Steam Electric Station currently supplies power to the data centre.

13 March 2024

The $650m will be paid in stages – $350m on closing and $300m to be released on the attainment of development milestones later this year. Talen will also receive additional revenue from AWS related to sales of Susquehanna's energy to the grid.

“We believe this is a transformative transaction with long term benefits,” said Talen President & CEO Mark “Mac” McFarland, in a call with investors and media. As power demand continues to rise worldwide, “data centres are at the heart of that growth,” he noted.

Texas-based Talen is the majority owner and operator of the Susquehanna plant with 90% owned and operated by Talen subsidiary Susquehanna Nuclear. Allegheny Electric owns the other 10%. The plant’s two General Electric boiling water reactors began operation in 1983 and are licensed to operate until 2042 and 2044. In 2022, Talen filed for Chapter 11 bankruptcy as part of a financial restructuring, exiting bankruptcy in 2023. The transaction with AWS is expected to boost to its cash flow. After paying off debts, interest and other costs, Talen expects net proceeds of $361m from the deal.

The Cumulus campus is directly connected to the NPP. The data centre's four substations have a total potential 960 MW of redundant capacity. This includes 200 MW currently associated with the Nautilus cryptocurrency facility, in which Talen will retain its 75% interest. A further 240 MW of redundant capacity for data centres is expected to be ready this year. The campus has a "robust and redundant" fibre network.

According to Talen Energy’s investor presentation, it will supply fixed-price nuclear power to AWS’s new data centre as it is built. AWS has minimum contractual power commitments increasing in 120 MW increments over several years. AWS has a one-time option to cap commitments at 480 MW and two 10-year extension options tied to nuclear licence renewals.

"Power demand is growing for the first time in years, and AI and data centres are at the heart of that growth," McFarland said. "Data from the International Energy Agency suggests that energy demand from data centres, AI and cryptocurrencies could more than double over the next three years."

He added that the transaction will benefit the wider community by creating jobs and catalysing economic development as well as strengthening the Susquehanna plant itself as a major employer and significant taxpayer.

Maybe You Missed It, but the Internet ‘Died’ Five Years Ago

A conspiracy theory spreading online says the whole internet is now fake. It’s ridiculous, but possibly not that ridiculous?

By Kaitlyn Tiffany

If you search the phrase i hate texting on Twitter and scroll down, you will start to notice a pattern. An account with the handle @pixyIuvr and a glowing heart as a profile picture tweets, “i hate texting i just want to hold ur hand,” receiving 16,000 likes. An account with the handle @f41rygf and a pink orb as a profile picture tweets, “i hate texting just come live with me,” receiving nearly 33,000 likes. An account with the handle @itspureluv and a pink orb as a profile picture tweets, “i hate texting i just wanna kiss u,” receiving more than 48,000 likes.

There are slight changes to the verb choice and girlish username and color scheme, but the idea is the same each time: I’m a person with a crush in the age of smartphones, and isn’t that relatable? Yes, it sure is! But some people on Twitter have wondered whether these are really, truly, just people with crushes in the age of smartphones saying something relatable. They’ve pointed at them as possible evidence validating a wild idea called “dead-internet theory.”

Let me explain. Dead-internet theory suggests that the internet has been almost entirely taken over by artificial intelligence. Like lots of other online conspiracy theories, the audience for this one is growing because of discussion led by a mix of true believers, sarcastic trolls, and idly curious lovers of chitchat. One might, for example, point to @_capr1corn, a Twitter account with what looks like a blue orb with a pink spot in the middle as a profile picture. In the spring, the account tweeted “i hate texting come over and cuddle me,” and then “i hate texting i just wanna hug you,” and then “i hate texting just come live with me,” and then “i hate texting i just wanna kiss u,” which got 1,300 likes but didn’t perform as well as it did for @itspureluv. But unlike lots of other online conspiracy theories, this one has a morsel of truth to it. Person or bot: Does it really matter?

Read: The internet is mostly bots

Dead-internet theory. It’s terrifying, but I love it. I read about it on Agora Road’s Macintosh Cafe, an online forum with a pixelated-Margaritaville vibe and the self-awarded honor “Best Kept Secret of the Internet!” Right now, the background is a repeated image of palm trees, a hot-pink sunset, and some kind of liquor pouring into a rocks glass. The site is largely for discussing lo-fi hip-hop, which I don’t listen to, but it is also for discussing conspiracy theories, which I do.

In January, I stumbled across a new thread there titled “Dead Internet Theory: Most of the Internet is Fake,” shared by a user named IlluminatiPirate. Over the next few months, this would become the ur-text for those interested in the theory. The post is very long, and some of it is too confusing to bother with; the author claims to have pieced together the theory from ideas shared by anonymous users of 4chan’s paranormal section and another forum called Wizardchan, an online community premised on earning wisdom and magic through celibacy. (In an email, IlluminatiPirate, who is an operations supervisor for a logistics company in California, told me that he “truly believes” in the theory. I agreed not to identify him by name because he said he fears harassment.)

Peppered with casually offensive language, the post suggests that the internet died in 2016 or early 2017, and that now it is “empty and devoid of people,” as well as “entirely sterile.” Much of the “supposedly human-produced content” you see online was actually created using AI, IlluminatiPirate claims, and was propagated by bots, possibly aided by a group of “influencers” on the payroll of various corporations that are in cahoots with the government. The conspiring group’s intention is, of course, to control our thoughts and get us to purchase stuff.

As evidence, IlluminatiPirate offers, “I’ve seen the same threads, the same pics, and the same replies reposted over and over across the years.” He argues that all modern entertainment is generated and recommended by an algorithm; gestures at the existence of deepfakes, which suggest that anything at all may be an illusion; and links to a New York story from 2018 titled “How Much of the Internet Is Fake? Turns Out, a Lot of It, Actually.” “I think it’s entirely obvious what I’m subtly suggesting here given this setup,” the post continues. “The U.S. government is engaging in an artificial intelligence powered gaslighting of the entire world population.” So far, the original post has been viewed more than 73,000 times.

Read: Artificial intelligence is misreading human emotion

Obviously, the internet is not a government psyop, even though the Department of Defense had a role in its invention. But if it were, the most compelling evidence to me is the dead-internet theory’s observation that the same news items about unusual moon-related events seem to repeat year after year. I swear I’ve been saying this for years. What is a super flower blood moon? What is a pink supermoon? A quick search of headlines from just this month brings up: “There’s Something Special About This Weekend’s Moon,” “Don’t Miss: Rare, Seasonal ‘Blue Moon’ Rises Tonight,” and “Why This Weekend’s Blue Moon Is Extra Rare.” I just don’t understand why everyone is so invested in making me look at the moon all the time? Leave me alone about the moon!

Dead-internet theory is a niche idea because it’s patently ridiculous, but it has been spreading. Caroline Busta, the Berlin-based founder of the media platform New Models, recently referenced it in her contribution to an online group show organized by the KW Institute for Contemporary Art. “Of course a lot of that post is paranoid fantasy,” she told me. But the “overarching idea” seems right to her. The theory has become fodder for dramatic YouTube explainers, including one that summarizes the original post in Spanish and has been viewed nearly 260,000 times. Speculation about the theory’s validity has started appearing in the widely read Hacker News forum and among fans of the massively popular YouTube channel Linus Tech Tips. In a Reddit forum about the paranormal, the theory is discussed as a possible explanation for why threads about UFOs seem to be “hijacked” by bots so often.

The theory’s spread hasn’t been entirely organic. IlluminatiPirate has posted a link to his manifesto in several Reddit forums that discuss conspiracy theories, including the Joe Rogan subreddit, which has 709,000 subscribers. In the r/JoeRogan comments, users argue sarcastically—or sincerely?—about who among them is a bot. “I’m absolutely the type of loser who would get swindled into living among bots and never realize it,” a member of the 4chan-adjacent Something Awful forum commented when the theory was shared there in February. “Seems like something a bot would post,” someone replied. Even the playful arguments about how everything is the same are the same.

Read: Why is Joe Rogan so popular?

That particular conversation continued down the bleakest path imaginable, to the point of this comment: “If I was real I’m pretty sure I’d be out there living each day to the fullest and experiencing everything I possibly could with every given moment of the relatively infinitesimal amount of time I’ll exist for instead of posting on the internet about nonsense.”

Anyway … dead-internet theory is pretty far out-there. But unlike the internet’s many other conspiracy theorists, who are boring or really gullible or motivated by odd politics, the dead-internet people kind of have a point. In the New York story that IlluminatiPirate invokes, the writer Max Read plays with paranoia. “Everything that once seemed definitively and unquestionably real now seems slightly fake,” he writes. But he makes a solid argument: He notes that a majority of web traffic probably comes from bots, and that YouTube, for a time, had such high bot traffic that some employees feared “the Inversion”—the point when its systems would start to see bots as authentic and humans as inauthentic. He also points out that even engagement metrics on sites as big and powerful as Facebook have been grossly inflated or easily gamed, and that human presence can be mimicked with click farms or cheap bots.

Some of this may be improving now, for better or for worse. Social-media companies have gotten a lot better at preventing the purchase of fake views and fake likes, while some bot farmers have, in response, become all the more sophisticated. Major platforms still play whack-a-mole with inauthentic activity, so the average internet user has no way of knowing how much of what they see is “real.”

But more than that, the theory feels true: Most weeks, Twitter is taken over by an argument about how best to practice personal hygiene, or which cities have the worst food and air quality, which somehow devolves into allegations of classism and accusations of murder, which for whatever reason is actually not as offensive as classism anymore. A celebrity is sorry. A music video has broken the internet. A meme has gotten popular and then boring. “Bennifer Might Be Back On, and No One’s More Excited Than Twitter.” At this point, you could even say that the point of the theory is so obvious, it’s cliché—people talk about longing for the days of weird web design and personal sites and listservs all the time. Even Facebook employees say they miss the “old” internet. The big platforms do encourage their users to make the same conversations and arcs of feeling and cycles of outrage happen over and over, so much so that people may find themselves acting like bots, responding on impulse in predictable ways to things that were created, in all likelihood, to elicit that very response.

Thankfully, if all of this starts to bother you, you don’t have to rely on a wacky conspiracy theory for mental comfort. You can just look for evidence of life: The best proof I have that the internet isn’t dead is that I wandered onto some weird website and found an absurd rant about how the internet is so, so dead.

Disrupting deceptive uses of AI by covert influence operations

We’ve terminated accounts linked to covert influence operations; no significant audience increase due to our services.

OpenAI is committed to enforcing policies that prevent abuse and to improving transparency around AI-generated content. That is especially true with respect to detecting and disrupting covert influence operations (IO), which attempt to manipulate public opinion or influence political outcomes without revealing the true identity or intentions of the actors behind them.

In the last three months, we have disrupted five covert IO that sought to use our models in support of deceptive activity across the internet. As of May 2024, these campaigns do not appear to have meaningfully increased their audience engagement or reach as a result of our services.

This blog describes the threat actors we disrupted, attacker trends we identified, and important defensive trends - including how designing AI models with safety in mind in many cases prevented the threat actors from generating the content they desired, and how AI tools have made our own investigations more efficient. Alongside this blog, we are publishing a trend analysis that describes the behavior of these malicious actors in detail.

Read the full report(opens in a new window)

Threat actors work across the internet. So do we. By collaborating with industry, civil society, and government we tackle the creation, distribution, and impact of IO content. Our investigations and disruptions were made possible in part because there’s been so much detailed threat reporting over the years by distribution platforms and the open-source community. OpenAI is publishing these findings, as other tech companies do, to promote information sharing and best practices amongst the broader community of stakeholders.

Disruption of covert influence operations

Over the last three months, our work against IO actors has disrupted covert influence operations that sought to use AI models for a range of tasks, such as generating short comments and longer articles in a range of languages, making up names and bios for social media accounts, conducting open-source research, debugging simple code, and translating and proofreading texts.

Specifically, we disrupted:

• A previously unreported operation from Russia, which we dubbed Bad Grammar, operating mainly on Telegram and targeting Ukraine, Moldova, the Baltic States and the United States. The people behind Bad Grammar used our models to debug code for running a Telegram bot and to create short, political comments in Russian and English that were then posted on Telegram.
• An operation originating in Russia known as Doppelganger(opens in a new window). People acting on behalf of Doppelganger used our models to generate comments in English, French, German, Italian and Polish that were posted on X and 9GAG; translate and edit articles in English and French that were posted on websites linked to this operation; generate headlines; and convert news articles into Facebook posts.
• A Chinese network known as Spamouflage(opens in a new window), which used our models to research public social media activity, generate texts in languages including Chinese, English, Japanese and Korean that were then posted across platforms including X, Medium and Blogspot, and debug code for managing databases and websites, including a previously unreported domain, revealscum[.]com.
• An Iranian operation known as the International Union of Virtual Media(opens in a new window) (IUVM), which used our models to generate and translate long-form articles, headlines and website tags that were then published on a website linked to this Iranian threat actor, iuvmpress[.]co;
• Activity by a commercial company in Israel called STOIC, because technically we disrupted the activity, not the company. We nicknamed this operation Zero Zeno, for the founder of the stoic school of philosophy. The people behind Zero Zeno used our models to generate articles and comments that were then posted across multiple platforms, notably Instagram, Facebook, X, and websites associated with this operation.

The content posted by these various operations focused on a wide range of issues, including Russia’s invasion of Ukraine, the conflict in Gaza, the Indian elections, politics in Europe and the United States, and criticisms of the Chinese government by Chinese dissidents and foreign governments.

So far, these operations do not appear to have benefited from meaningfully increased audience engagement or reach as a result of our services. Using Brookings’ Breakout Scale,(opens in a new window) which assesses the impact of covert IO on a scale from 1 (lowest) to 6 (highest), none of the five operations included in our case studies scored higher than a 2 (activity on multiple platforms, but no breakout into authentic communities).

Attacker trends

Based on the investigations into influence operations detailed in our report, and the work of the open-source community, we have identified the following trends in how covert influence operations have recently used artificial intelligence models like ours.

• Content generation: All these threat actors used our services to generate text (and occasionally images) in greater volumes, and with fewer language errors than would have been possible for the human operators alone.
• Mixing old and new: All of these operations used AI to some degree, but none used it exclusively. Instead, AI-generated material was just one of many types of content they posted, alongside more traditional formats, such as manually written texts or memes copied from across the internet.
• Faking engagement: Some of the networks we disrupted used our services to help create the appearance of engagement across social media - for example, by generating replies to their own posts. This is distinct from attracting authentic engagement, which none of the networks we describe here managed to do to a meaningful degree.
• Productivity gains: Many of the threat actors that we identified and disrupted used our services in an attempt to enhance productivity, such as summarizing social media posts or debugging code.

Defensive trends

While much of the public debate so far has focused on the potential or actual use of AI by attackers, it is important to remember the advantages that AI offers to defenders. Our investigations also benefit from industry sharing and open-source research.

• Defensive design: We impose friction on threat actors through our safety systems, which reflect our approach to responsibly deploying AI. For example, we repeatedly observed cases where our models refused to generate the text or images that the actors asked for.
• AI-enhanced investigation: Similar to our approach to using GPT-4 for content moderation and cyber defense, we have built our own AI-powered tools to make our detection and analysis more effective. The investigations described in the accompanying report took days, rather than weeks or months, thanks to our tooling. As our models improve, we’ll continue leveraging their capabilities to improve our investigations too.
• Distribution matters: Like traditional forms of content, AI-generated material must be distributed if it is to reach an audience. The IO posted across a wide range of different platforms, including X, Telegram, Facebook, Medium, Blogspot, and smaller forums, but none managed to engage a substantial audience.
• Importance of industry sharing: To increase the impact of our disruptions on these actors, we have shared detailed threat indicators with industry peers. Our own investigations benefited from years of open-source analysis conducted by the wider research community.
• The human element: AI can change the toolkit that human operators use, but it does not change the operators themselves. Our investigations showed that these actors were as prone to human error as previous generations have been - for example, publishing refusal messages from our models on social media and their websites. While it is important to be aware of the changing tools that threat actors use, we should not lose sight of the human limitations that can affect their operations and decision making.

We are committed to developing safe and responsible AI, which involves designing our models with safety in mind and proactively intervening against malicious use. Detecting and disrupting multi-platform abuses such as covert influence operations can be challenging because we do not always know how content generated by our products is distributed. But we are dedicated to finding and mitigating this abuse at scale by harnessing the power of generative AI.

Pivot to AI: Pay no attention to the man behind the curtain

By Amy Castor and David Gerard the September 12, 2023 for AmyCastor

“all this talk of AI xrisk has the stink of marketing too. Ronald McDonald telling people that he has a bunker in New Zealand because the new burger they’re developing in R&D might be so delicious society will crumble.”

— Chris Martin

Crypto’s being dull again — but thankfully, AI has been dull too. The shine is coming off. So we’re back on the AI beat.

The AI winter will be privatized

Since the buzzword “artificial intelligence” was coined in the 1950s, AI has gone through several boom and bust cycles.

A new technological approach looks interesting and gets a few results. It gets ridiculously hyped up and lands funding. The tech turns out to be not so great, so the funding gets cut. The down cycles are called AI winters.

Past AI booms were funded mainly by the US Department of Defense. But the current AI boom has been almost completely funded by venture capital.

The VCs who spent 2021 and 2022 pouring money into crypto startups are pivoting to AI startups, because people buy the idea that AI will change the world. In the first half of 2023, VCs invested more than $40 billion into AI startups, and $11 billion just in May 2023. This is even as overall VC funding for startups dropped by half in the same period from the year before. [Reuters; Washington Post]

The entire NASDAQ is being propped up by AI. It’s one of the only fields that is still hiring.

In contrast, the DOD only requested $1.8 billion for AI funding in its 2024 budget. [DefenseScoop]

So why are VCs pouring money into AI?

Venture capital is professional gambling. VCs are looking for a liquidity event. One big winner can pay for a lot of failures.

Finding someone to buy a startup you’ve funded takes marketing and hype. The company doing anything useful, or anything that even works, is optional.

What’s the exit plan for AI VCs? Where’s the liquidity event? Do they just hope the startups they fund will do an initial public offering or just get acquired by a tech giant before the market realizes AI is running out of steam?

We’re largely talking about startups whose business model is sending queries to OpenAI.

At least with “Web3,” the VCs would just dump altcoins on retail investors via their very good friends at Coinbase. But with AI, we can’t see an obvious exit strategy beyond finding a greater fool.

Pay no attention to the man behind the curtain

The magical claim of machine learning is that if you give the computer data, the computer will work out the relations in the data all by itself. Amazing!

In practice, everything in machine learning is incredibly hand-tweaked. Before AI can find patterns in data, all that data has to be tagged, and output that might embarrass the company needs to be filtered.

Commercial AI runs on underpaid workers in English-speaking countries in Africa creating new training data and better responses to queries. It’s a painstaking and laborious process that doesn’t get talked about nearly enough.

The workers do individual disconnected actions all day, every day — so called “tasks” — working for companies like Remotasks, a subsidiary of Scale AI, and doing a huge amount of the work behind OpenAI.

AI doesn’t remove human effort. It just makes it much more alienated.

There’s an obvious hack here. If you are an AI task worker, your goal is to get paid as much as possible without too much effort. So why not use some of the well-known tools for this sort of job? [New York]

Another Kenyan annotator said that after his account got suspended for mysterious reasons, he decided to stop playing by the rules. Now, he runs multiple accounts in multiple countries, tasking wherever the pay is best. He works fast and gets high marks for quality, he said, thanks to ChatGPT. The bot is wonderful, he said, letting him speed through $10 tasks in a matter of minutes. When we spoke, he was having it rate another chatbot’s responses according to seven different criteria, one AI training the other.

Remember, the important AI use case is getting venture capital funding. Why buy or rent expensive computing when you can just pay people in poor countries to fake it? Many “AI” systems are just a fancier version of the original Mechanical Turk.

Facebook’s M from 2017 was an imitation of Apple’s Siri virtual assistant. The trick was that hard queries would be punted to a human. Over 70% of queries ended up being answered by a human pretending to be the bot. M was shut down a year after launch.

Kaedim is a startup that claims to turn two-dimensional sketches into 3-D models using “machine learning.” The work is actually done entirely by human modelers getting paid $1-$4 per 15-minute job. But then, the founder, Konstantina Psoma, was a Forbes 30 Under 30. [404 Media; Forbes]

The LLM is for spam

OpenAI’s AI-powered text generators fueled a lot of the hype around AI — but the real-world use case for large language models is overwhelmingly to generate content for spamming. [Vox]

The use case for AI is spam web pages filled with ads. Google considers LLM-based ad landing pages to be spam, but seems unable or unwilling to detect and penalize it. [MIT Technology Review; The Verge]

The use case for AI is spam books on Amazon Kindle. Most are “free” Kindle Unlimited titles earning money through subscriber pageviews rather than outright purchases. [Daily Dot]

The use case for AI is spam news sites for ad revenue. [NewsGuard]

The use case for AI is spam phone calls for automated scamming — using AI to clone people’s voices. [CBS]

The use case for AI is spam Amazon reviews and spam tweets. [Vice]

The use case for AI is spam videos that advertise malware. [DigitalTrends]

The use case for AI is spam sales sites on Etsy. [The Atlantic, archive]

The use case for AI is spam science fiction story submissions. Clarkesworld had to close submissions because of the flood of unusable generated garbage. The robot apocalypse in action. [The Register]

Supertoys last all summer long

End users don’t actually want AI-based products. Machine learning systems can generate funny text and pictures to show your friends on social media. But even that’s wearing thin — users mostly see LLM output in the form of spam.

LLM writing style and image generator drawing style are now seen as signs of low quality work. You can certainly achieve artistic quality with AI manipulation, as in this music video — but even this just works on its novelty value. [YouTube]

For commercial purposes, the only use case for AI is still to replace quality work with cheap ersatz bot output — in the hope of beating down labor costs.

Even then, the AI just isn’t up to the task.

Microsoft put $10 billion into OpenAI. The Bing search engine added AI chat — and it had almost no effect on user numbers. It turns out that search engine users don’t want weird bot responses full of errors. [ZDNet]

The ChatGPT website’s visitor numbers went down 10% in June 2023. LLM text generators don’t deliver commercial results, and novelty only goes so far. [Washington Post]

After GPT-3 came out, OpenAI took three years to make an updated version. GPT-3.5 was released as a stop-gap in October 2022. Then GPT-4 finally came out in March 2023! But GPT-4 turns out to be eight instances of GPT-3 in a trenchcoat. The technology is running out of steam. [blog post; Twitter, archive]

Working at all will be in the next version

The deeper problem is that many AI systems simply don’t work. The 2022 paper “The fallacy of AI functionality” notes that AI systems are often “constructed haphazardly, deployed indiscriminately, and promoted deceptively.”

Still, machine learning systems do some interesting things, a few of which are even genuinely useful. We asked GitHub and they told us that they encourage their own employees to use the GitHub Copilot AI-based autocomplete system for their own internal coding — with due care and attention. We know of other coders who find Copilot to be far less work than doing the boilerplate by hand.

(Though Google has forbidden its coders from using its AI chatbot, Bard, to generate internal code.) [The Register]

Policy-makers and scholars — not just the media — tend to propagate AI hype. Even if they try to be cautious, they may work in terms of ethics of deployment, and presume that the systems do what they’re claimed to do — when they often just don’t.

Ethical considerations come after you’ve checked basic functionality. Always put functionality first. Does the system work? Way too often, it just doesn’t. Test and measure. [arXiv, PDF, 2022]

AI is the new crypto mining

In 2017, the hot buzzword was “blockchain” — because the price of bitcoin was going up. Struggling businesses would add the word “blockchain” to their name or their mission statement, in the hope their stock price would go up. Long Island Iced Tea became Long Blockchain and saw its shares surge 394%. Shares in biotech company Bioptix doubled in price when it changed its name to Riot Blockchain and pivoted to bitcoin mining. [Bloomberg, 2017, archive; Bloomberg, 2017, archive]

The same is now happening with AI. Only it’s not just the venture capitalists — even the crypto miners are pivoting to AI.

Bitcoin crashed last year and crypto mining is screwed. As far as we can work out, the only business plan was to get foolish investors’ money during the bubble, then go bankrupt.

In mid-2024, the bitcoin mining reward will halve again. So the mining companies are desperate to find other sources of income.

Ethereum moved to proof of stake in September 2022 and told its miners to just bugger off. Ethereum was mined on general-purpose video cards — so miners have a glut of slightly-charred number crunching machinery.

Hive Blockchain in Vancouver is pivoting to AI to repurpose its pile of video cards. It’s also changed its name to Hive Digital Technologies. [Bloomberg, archive; press release]

Marathon Digital claims that “over time you’re going to see that blockchain technologies and AI have a very tight coupling.” No, us neither. Marathon is doubling and tripling down on bitcoin mining — but, buzzwords! [Decrypt]

Nvidia makes the highest-performance video cards. The GPU processors on these cards turn out to be useful for massively parallel computations in general — such as running the calculations needed to train machine learning models. Nvidia is having an excellent year and its market cap is over $1 trillion.

So AI can take over from crypto in yet another way — carbon emissions from running all those video cards.

AI’s massive compute load doesn’t just generate carbon — it uses huge amounts of fresh water for cooling. Microsoft’s water usage went up 34% between 2021 and 2022, and they blame AI computation. ChatGPT uses about 500 mL of water every time you have a conversation with it. [AP]

We don’t yet have a Digiconomist of AI carbon emissions. Go start one.

Losing the imitation game

AI cannot develop software for you, but that's not going to stop people from trying to make it happen anyway. And that is going to turn all of the easy software development problems into hard problems.

If you've been anywhere near major news or social media in the last few months, you've probably heard repeatedly about so-called AI, ChatGPT, and large language models (LLMs). The hype surrounding these topics has been intense. And the rhetoric has been manipulative, to say the least. Proponents have claimed that their models are or soon will be generally intelligent, in the way we mean humans are intelligent. They're not. They've claimed that their AI will eliminate whole categories of jobs. And they've claimed that developing these systems further and faster is both necessary and urgent, justified by science fiction dressed up as arguments for some sort of "safety" that I find to be incoherent.

The outer layer of hype surrounding AI—and LLM chatbots in particular—is that they will become indispensable tools of daily work, and entirely replace people in numerous categories of jobs. These claims have included the fields of medicine, law, and education, among others. I think it's nonsense. They imagine self-teaching classrooms and self-diagnosing fitness gadgets. These things will probably not even work as well as self-driving cars, which is to say: only well enough to be dangerous. Of course, that's not stopping people from pushing these fantasies, anyway. But these fields are not my area of expertise. My expertise is in software engineering. We should know better, but software developers are falling victim to the same kind of AI fantasies.

A computer can never be held accountable. Therefore, a computer must never make a management decision.

While the capabilities are fantasy, the dangers are real. These tools have denied people jobs, housing, and welfare. All erroneously. They have denied people bail and parole, in such a racist way it would be comical if it wasn't real. And the actual function of AI in all of these situations is to obscure liability for the harm these decisions cause.

So-Called AI

Artificial Intelligence is an unhelpful term. It serves as a vehicle for people's invalid assumptions. It hand-waves an enormous amount of complexity regarding what "intelligence" even is or means. And it encourages people confuse concepts like cognition, agency, autonomy, sentience, consciousness, and a host of related ideas. However, AI is the vernacular term for this whole concept, so it's the one I'll use. I'll let other people push that boulder, I'm here to push a different one.

Those concepts are not simple ideas, either. Describing them gets into hard questions of psychology, neurology, anthropology, and philosophy. At least. Given that these are domains that the tech field has routinely dismissed as unimportant for decades, maybe it shouldn't be surprising that techies as a group are now completely unprepared to take a critical view of claims about AI.

The Turing Test

Certainly part of how we got here is the Turing test. That is, the pop science reduction of Alan Turing's imitation game. The actual proposal is more substantial. And taking it seriously produces some interesting reading. But the common notion is something like a computer is intelligent if it can reliably pass as human in conversation. I hope seeing it spelled out like that makes it clear how dramatically that overreaches. Still, it's the framework that people have, and it informs our situation. I think the bit that is particularly informative is the focus on natural, conversational language. And also, the deception inherent in the imitation game scenario, but I'll come back to that.

Our understanding of intelligence is a moving target. We only have one meaningful fixed point to work from. We assert that humans are intelligent. Whether anything else is, is not certain. What intelligence itself is, is not certain. Not too long ago, a lot of theory rested on our ability to create and use tools. But then that ability turned out to be not as rare as we thought, and the consensus about the boundaries of intelligence shifted. Lately, it has fallen to our use of abstract language. That brings us back to AI chatbots. We suddenly find ourselves confronted with machines that seem to have a command of the English language that rivals our own. This is unfamiliar territory, and at some level it's reasonable that people will reach for explanations and come up with pop science notions like the Turing test.

Language: any system of formalized symbols, signs, sounds, gestures, or the like used or conceived as a means of communicating thought, emotion, etc.

Language Models

ChatGPT and the like are powered by large language models. Linguistics is certainly an interesting field, and we can learn a lot about ourselves and each other by studying it. But language itself is probably less than you think it is. Language is not comprehension, for example. It's not feeling, or intent, or awareness. It's just a system for communication. Our common lived experiences give us lots of examples that anything which can respond to and produce common language in a sensible-enough way must be intelligent. But that's because only other people have ever been able to do that before. It's actually an incredible leap to assume, based on nothing else, that a machine which does the same thing is also intelligent. It's much more reasonable to question whether the link we assume exists between language and intelligence actually exists. Certainly, we should wonder if the two are as tightly coupled as we thought.

That coupling seems even more improbable when you consider what a language model does, and—more importantly—doesn't consist of. A language model is a statistical model of probability relationships between linguistic tokens. It's not quite this simple, but those tokens can be thought of as words. They might also be multi-word constructs, like names or idioms. You might find "raining cats and dogs" in a large language model, for instance. But you also might not. The model might reproduce that idiom based on probability factors instead. The relationships between these tokens span a large number of parameters. In fact, that's much of what's being referenced when we call a model large. Those parameters represent grammar rules, stylistic patterns, and literally millions of other things.

What those parameters don't represent is anything like knowledge or understanding. That's just not what LLMs do. The model doesn't know what those tokens mean. I want to say it only knows how they're used, but even that is over stating the case, because it doesn't know things. It models how those tokens are used. When the model works on a token like "Jennifer", there are parameters and classifications that capture what we would recognize as things like the fact that it's a name, it has a degree of formality, it's feminine coded, it's common, and so on. But the model doesn't know, or understand, or comprehend anything about that data any more than a spreadsheet containing the same information would understand it.

Mental Models

So, a language model can reproduce patterns of language. And there's no particular reason it would need to be constrained to natural, conversational language, either. Anything that's included in the set of training data is fair game. And it turns out that there's been a lot of digital ink spent on writing software and talking about writing software. Which means those linguistic patterns and relationships can be captured and modeled just like any other. And sure, there are some programming tasks where just a probabilistic assembly of linguistic tokens will produce a result you want. If you prompt ChatGPT to write a python function that fetches a file from S3 and records something about it in DynamoDB, I would bet that it just does, and that the result basically works. But then, if you prompt ChatGPT to write an authorization rule for a new role in your application's proprietary RBAC system, I bet that it again just does, and that the result is useless, or worse.

Programming as Theory Building

Non-trivial software changes over time. The requirements evolve, flaws need to be corrected, the world itself changes and violates assumptions we made in the past, or it just takes longer than one working session to finish. And all the while, that software is running in the real world. All of the design choices taken and not taken throughout development; all of the tradeoffs; all of the assumptions; all of the expected and unexpected situations the software encounters form a hugely complex system that includes both the software itself and the people building it. And that system is continuously changing.

The fundamental task of software development is not writing out the syntax that will execute a program. The task is to build a mental model of that complex system, make sense of it, and manage it over time.

To circle back to AI like ChatGPT, recall what it actually does and doesn't do. It doesn't know things. It doesn't learn, or understand, or reason about things. What it does is probabilistically generate text in response to a prompt. That can work well enough if the context you need to describe the goal is so simple that you can write it down and include it with the prompt. But that's a very small class of essentially trivial problems. What's worse is there's no clear boundary between software development problems that are trivial enough for an LLM to be helpful vs being unhelpful. The LLM doesn't know the difference, either. In fact, the LLM doesn't know the difference between being tasked to write javascript or a haiku, beyond the different parameters each prompt would activate. And it will readily do a bad job of responding to either prompt, with no notion that there even is such a thing as a good or bad response.

Software development is complex, for any non-trivial project. But complexity is hard. Overwhelmingly, when we in the software field talk about developing software, we've dealt with that complexity by ignoring it. We write code samples that fit in a tweet. We reduce interviews to trivia challenges about algorithmic minutia. When we're feeling really ambitious, we break out the todo app. These are contrivances that we make to collapse technical discussions into an amount of context that we can share in the few minutes we have available. But there seem to be a lot of people who either don't understand that or choose to ignore it. They frame the entire process of software development as being equivalent to writing the toy problems and code samples we use among general audiences.

Automating the Easy Part

The intersection of AI hype with that elision of complexity seems to have produced a kind of AI booster fanboy, and they're making personal brands out of convincing people to use AI to automate programming. This is an incredibly bad idea. The hard part of programming is building and maintaining a useful mental model of a complex system. The easy part is writing code. They're positioning this tool as a universal solution, but it's only capable of doing the easy part. And even then, it's not able to do that part reliably. Human engineers will still have to evaluate and review the code that an AI writes. But they'll now have to do it without the benefit of having anyone who understands it. No one can explain it. No one can explain what they were thinking when they wrote it. No one can explain what they expect it to do. Every choice made in writing software is a choice not to do things in a different way. And there will be no one who can explain why they made this choice, and not those others. In part because it wasn't even a decision that was made. It was a probability that was realized.

[A programmer's] education has to emphasize the exercise of theory building, side by side with the acquisition of knowledge of data processing and notations.

But it's worse than AI being merely inadequate for software development. Developing that mental model requires learning about the system. We do that by exploring it. We have to interact with it. We manipulate and change the system, then observe how it responds. We do that by performing the easy, simple programing tasks. Delegating that learning work to machines is the tech equivalent of eating our seed corn. That holds true beyond the scope of any team, or project, or even company. Building those mental models is itself a skill that has to be learned. We do that by doing it, there's not another way. As people, and as a profession, we need the early career jobs so that we can learn how to do the later career ones. Giving those learning opportunities to computers instead of people is profoundly myopic.

Imitation Game

If this is the first time you're hearing or reading these sentiments, that's not too surprising. The marketing hype surrounding AI in recent months has been intense, pervasive, and deceptive. AI is usually cast as being hyper competent, and superhuman. To hear the capitalists who are developing it, AI is powerful, mysterious, dangerous, and inevitable. In reality, it's almost none of those things. I'll grant that AI can be dangerous, but not for the reasons they claim. AI is complicated and misunderstood, and this is by design. They cloak it in rhetoric that's reminiscent of the development of atomic weapons, and they literally treat the research like an arms race.

I'm sure there are many reasons they do this. But one of the effects it has is to obscure the very mundane, serious, and real harms that their AI models are currently perpetuating. Moderating the output of these models depends on armies of low paid and precariously employed human reviewers, mostly in Kenya. They're subjected to the raw, unfiltered linguistic sewage that is the result of training a language model on uncurated text found on the public internet. If ChatGPT doesn't wantonly repeat the very worst of the things you can find on reddit, 4chan, or kiwi farms, that is because it's being dumped on Kenyan gig workers instead.

That's all to say nothing of the violations of intellectual property and basic consent that was required to train the models in the first place. The scale of the theft and exploitation required to build the data sets these models train with is almost inconceivable. And the energy consumption and e-waste produced by these systems is staggering.

All of this is done to automate the creation of writing or media that is designed to deceive people. It's intended to seem like people, or like work done by people. The deception, from both the creators and the AI models themselves, is pervasive. There may be real, productive uses for these kinds of tools. There may be ways to build and deploy them ethically and sustainably. But that's not the situation with the instances we have. AI, as it's been built today, is a tool to sell out our collective futures in order to enrich already wealthy people. They like to frame it as being akin to nuclear science. But we should really see it as being more like fossil fuels

Twitter is becoming a 'ghost town' of bots as AI-generated spam content floods the internet

ABC Science / By technology reporter James Purtill

Parts of the web are now dominated by bots and junk websites designed to go unread by humans.

One morning in January this year, marine scientist Terry Hughes opened X (formerly Twitter) and searched for tweets about the Great Barrier Reef.

"I keep an eye on what's being tweeted about the reef every day," Professor Hughes, a leading coral researcher at James Cook University, said.

What he found that day surprised and confused him; hundreds of bot accounts tweeting the same strange message with slightly different wording.

"Wow, I had no idea that agricultural runoff could have such a devastating impact on the Great Barrier Reef," one account, which otherwise spruiked cryptocurrencies, tweeted.

Another crypto bot wrote: "Wow, it's disheartening to hear about the water pollution challenges Australia faces."

And so on. Hundreds of crypto accounts tweeting about agricultural runoff.

A month later, it happened again. This time, bots were tweeting about "marine debris" threatening the Great Barrier Reef.

What was going on?

When Professor Hughes tweeted what he'd found, some saw a disinformation conspiracy, an attempt to deflect attention from climate change.

The likely answer, however, is more mundane, but also more far-reaching in its implications.

More than a year since Elon Musk bought X with promises to get rid of the bots, the problem is worse than ever, experts say.

And this is one example of a broader problem affecting online spaces.

The internet is filling up with "zombie content" designed to game algorithms and scam humans.

It's becoming a place where bots talk to bots, and search engines crawl a lonely expanse of pages written by artificial intelligence (AI).

Junk websites clog up Google search results. Amazon is awash with nonsense e-books. YouTube has a spam problem.

And this is just a trickle in advance of what's been called the "great AI flood".

Bots liking bots, talking to other bots

But first, let's get back to those reef-tweetin' bots.

Timothy Graham, an expert on X bot networks at the Queensland University of Technology, ran the tweets through a series of bot and AI detectors.

Dr Graham found 100 per cent of the text was AI-generated.

"Overall, it appears to be a crypto bot network using AI to generate its content," he said.

"I suspect that at this stage it's just trying to recruit followers and write content that will age the fake accounts long enough to sell them or use them for another purpose."

That is, the bots probably weren't being directed to tweet about the reef in order to sway public opinion.

Dr Graham suspects these particular bots probably have no human oversight, but are carrying out automated routines intended to out-fox the bot-detection algorithms.

Searching for meaning in their babble was often pointless, he said.

"[Professor Hughes] is trying to interpret it and is quite right to try and make sense of it, but it just chews up attention, and the more engagement they get, the more they are rewarded.

The cacophony of bot-talk degrades the quality of online conversations. They interrupt the humans and waste their time.

"Here's someone who is the foremost research scientist in this space, spending their time trying to work out the modus operandi of these accounts."

In this case, the bots were replying to the tweet of another bot, which, in turn, replied to the tweets of other bots, and so on.

One fake bot account was stacked on top of the other, Dr Graham said.

"It's AI bots all the way down."

How bad is X's bot problem?

In January, a ChatGPT glitch appeared to shine a light on X's bot problem.

For a brief time, some X accounts posted ChatGPT's generic response to requests that it deems outside of its content policy, exposing them as bots that use ChatGPT to generate content.

Users posted videos showing scrolling feeds with numerous accounts stating "I'm sorry, but I cannot provide a response to your request as it goes against OpenAl's content policy."

"Twitter is a ghost town," one user wrote.

But the true scale of X's bot problem is difficult for outsiders to estimate.

Shortly after Mr Musk gained control of X while complaining about bots, X shut down free access to the programming interface that allowed researchers to study this problem.

That left researchers with two options: pay X for access to its data or find another way to peek inside.

Towards the end of last year, Dr Graham and his colleagues at QUT paid X $7,800 from a grant fund to analyse 1 million tweets surrounding the first Republican primary debate.

They found the bot problem was worse than ever, Dr Graham said at the time.

Later studies support this conclusion. Over three days in February, cybersecurity firm CHEQ tracked the proportion of bot traffic from X to its clients' websites.

It found three-quarters of traffic from X was fake, compared to less than 3 per cent of traffic from each of TikTok, Facebook and Instagram.

"Terry Hughes' experience is an example of what's going on on the platform," Dr Graham said.

"One in 10 likes are from a porn bot, anecdotally."

The rise of a bot-making industry

So what's the point of all these bots? What are they doing?

Crypto bots drive up demand for certain coins, porn bots get users to pay for porn websites, disinformation bots peddle fake news, astroturfing bots give the impression of public support, and so on.

Some bots exist purely to increase the follower counts and engagement statistics of paying customers.

A sign of the scale of X's bot problem is the thriving industry in bot-making.

Bot makers from around the world advertise their services on freelancer websites.

Awais Yousaf, a computer scientist in Pakistan, sells "ChatGPT Twitter bots" for $30 to $500, depending on their complexity.

In an interview with the ABC, the 27-year-old from Gujranwala said he could make a "fully fledged" bot that could "like comments on your behalf, make comments, reply to DMs, or even make engaging content according to your specification".

Mr Yousaf's career tracks the rise of the bot-making economy and successive cycles of internet hype.

Having graduated from university five years ago, he joined Pakistan's growing community of IT freelancers from "very poor backgrounds".

Many of the first customers wanted bots to promote cryptocurrencies, which were booming in popularity at the time.

"Then came the NFT thing," he said.

A few years ago he heard about OpenAI's GPT3 language model and took a three-month break to learn about AI.

"Now, almost 90 per cent of the bots I do currently are related to AI in one way or another.

"It can be as simple as people posting AI posts regarding fitness, regarding motivational ideas, or even cryptocurrency predictions."

In five years he's made 120 Twitter bots.

Asked about Mr Musk's promise to "defeat the spam bots," Mr Yousaf smiled.

"It's hard to remove Twitter bots from Twitter because Twitter is mostly bot."

AI-generated spam sites may overwhelm search engines

X's bot problem may be worse than other major platforms, but it's not alone.

A growing "deluge" of AI content is flooding platforms that were "never designed for a world where machines can talk with people convincingly", Dr Graham said.

"It's like you're running a farm and had never heard of a wolf before and then suddenly you have new predators on the scene.

"The platforms have no infrastructure in place. The gates are open."

The past few months have seen several examples of this.

Companies are using AI to rewrite other media outlet's stories, including the ABC's, to then publish them on the company's competing news websites.

A company called Byword claims it stole 3.6 million in "total traffic" from a competitor by copying their site and rewriting 1,800 articles using AI.

"Obituary pirates" are using AI to create YouTube videos of people summarising the obituaries of strangers, sometimes fabricating details about their deaths, in order to capture search traffic.

Authors are reporting what appear to be AI-generated imitations and summaries of their books on Amazon.

Google's search results are getting worse due to spam sites, according to a recent pre-print study by German researchers.

The researchers studies search results for thousands of product-review terms across Google, Bing and DuckDuckGo over the course of a year.

They found that higher-ranked pages tended to have lower text quality but were better designed to game the search ranking algorithm.

"Search engines seem to lose the cat-and-mouse game that is SEO spam," they wrote in the study.

Co-author Matti Wiegman from Bauhaus University, Weimar said this rankings war was likely to get much worse with the advent of AI-generated spam.

"What was previously low-quality content is now very difficult to distinguish from high-quality content," he said.

"As a result, it might become difficult to distinguish between authentic and trustworthy content that is useful and content that is not."

He added that the long-term effects of AI-generated content on search engines was difficult to judge.

AI-generated content could make search more useful, he said.

"One possible direction is that generated content will become better than the low-quality human-made content that dominates some genres in web search, in which case the search utility will increase."

Or the opposite will happen. AI-generated content will overwhelm "vulnerable spaces" such as search engines and "broadcasting-style" social media platforms like X.

In their place, people may turn to "walled gardens" and specialised forums with smaller numbers of human-only members.

Platforms prepare for coming flood

In response to this emerging problem, platforms are trialling different strategies.

Meta recently announced it was building tools to detect and label AI-generated images posted on its Facebook, Instagram and Threads services.

Amazon has limited authors to uploading a maximum of three books to its store each day, although authors say that hasn't solved the problem.

X is trialling a "Not a Bot" program in some countries where it charges new users $1 per year for basic features.

This program operates alongside X's verification system, where users pay $8 per month to have their identity checked and receive a blue tick.

But it appears the bot-makers have found a way around this.

All the reef-tweeting crypto bots Professor Hughes found were verified accounts.

"It's clutter on the platform that's not necessary. You'd wish they'd clean it up," the coral scientist said.

"It wastes everyone's time."

Un appel à démanteler l’intelligence artificielle

Romain Couillet le 22 juillet 2022

Professeur en informatique et chercheur jusqu’à récemment en mathématiques appliquées pour l’intelligence artificielle, j’ai été récemment sollicité comme membre du jury de soutenance du projet de fin d’études d’un étudiant en master d’informatique de l’Université Grenoble-Alpes.

L’étudiant motivait son projet par la nécessité de répondre à la double problématique suivante : l’entreprise chez qui il effectuait son stage ne parviendrait d’une part plus à recruter d’expert·es en design de circuits électroniques et, par ailleurs, la pénurie de métaux impose des contraintes croissantes sur la dimension (et donc la quantité de matière sollicitée) de ces mêmes circuits électroniques.

Face à ces enjeux, l’entreprise a proposé de développer un algorithme d’intelligence artificielle capable de combler l’expertise perdue (et de faire potentiellement mieux).
Sans aller dans les détails conceptuels de l’étude menée par l’étudiant, il apparaissait assez rapidement au cours de la présentation que l’approche proposée ne pouvait pas fonctionner et qu’en réalité il était fort présomptueux d’imaginer qu’un algorithme puisse effectuer la tâche souhaitée.

Le bilan des quatre premiers mois de stage n’était donc pas surprenant : en l’état, du point de vue de l’étudiant, la méthode développée ne fonctionnait pas encore mais les travaux étaient prometteurs. Situation somme toute classique que je rencontre de plus en plus fréquemment ces dernières années. Mais le point saillant de cette anecdote est ailleurs : au cours de la séance de questions du jury, j’ai demandé à l’étudiant si, à l’issue de ces quatre premiers mois clairement infructueux quoi qu’il en dise, il envisageait comme hypothèse la possibilité que l’intelligence artificielle puisse ne pas être en mesure de résoudre son problème.

En dépit du caractère éminemment rhétorique de la question, sa réponse sera extrêmement révélatrice : “non, je suis sûr que ça fonctionnera”. Tout est là. À elle seule, cette affirmation met selon moi en lumière le danger le plus nocif de l’intelligence artificielle : le caractère religieux et aveuglant qu’elle a acquis en à peine dix ans.

Avant propos : un détour paléosophique

Dans son travail sur la Paléosophie, Catherine Reine Thomas nous invite à repenser la cosmologie occidentale, celle qui a fait naître les premières civilisations agricoles il y a dix mille ans pour devenir la société dans laquelle nous européen·nes vivons, comme un déséquilibre dans le rapport entre la “Vie” et la “Technique”.

L’une des singularités du genre Homo, et plus particulièrement d’Homo Sapiens, par rapport aux autres animaux sont sa capacité et son envie de développer des outils et des savoir-faire. L’ensemble de ces outils engendre un spectre de technologies qui alimente alors une entité non-vivante mais bien réelle et dynamique que Thomas nomme Technique.

L’animalité d’Homo Sapiens exigerait néanmoins, au même titre que les autres animaux, de conserver un contrôle sur son milieu, une puissance d’agir pour emprunter les termes de Spinoza, que Thomas appelle la Vie.

Les entités Technique et Vie entrent dès lors dans une compétition intérieure dont l’équilibre est maintenu par l’activité artistique : Homo Sapiens n’est pas esclave d’une technologie tant que l’outil ou la technique garde un ancrage dans le champ de la Vie. Les dessins et sculptures (dès les premières peintures rupestres et objets décoratifs) sont autant d’expressions vivantes de techniques et savoir-faire utilisés par ailleurs, mais non exclusivement, pour le besoin des outils.

Cette dualité stable entre Vie et Technique assurerait un lien sain et serein entre Homo Sapiens et son environnement. Dans son ethnographie du peuple Achuar d’Amazonie qui entretient une relation animiste au milieu [^1], Descola rapporte que les Achuar ne travaillent que quatre à cinq heures par jour (aux champs, à la chasse, à l’artisanat) pour dédier le plus clair de leur temps aux chants, à la confection d’ornements, aux pratiques spirituelles et autres activités artistiques.

Selon Thomas, la Technique, vue comme entité non-vivante mais symbiotique au vivant Homo Sapiens peut devenir parasitaire lorsqu’elle prend le pas sur la Vie : c’est-à-dire lorsque le contrôle artistique par Homo Sapiens disparait au profit d’un développement non maîtrisé des technologies. Dès lors, la Technique se nourrit de l’énergie métabolique d’Homo Sapiens pour devenir autonome, tels les robots de l’univers dystopique d’Isaac Asimov difficilement tenus sous le contrôle des trois lois de la robotique.

Cet angle de vue paléosophique par la dualité Vie-Technique est intéressant à plusieurs égards : d’une part, il rejette le fatalisme, grandement véhiculée par les best-seller Sapiens d’Harari ou Effondrement de Diamond, d’une espèce humaine prédatrice et vouée à l’auto-destructrion. L’espèce humaine serait au contraire sous le joug de la Technique qui cultive Homo Sapiens comme les termites cultivent les champignons qui digèrent pour eux la cellulose du bois.

Il permet d’autre part d’expliquer l’inexplicable : pourquoi Homo Sapiens, en dépit des évidences scientifiques, des solutions technologiques existantes (permaculture, outillage low-tech, communs) et des connaissances neuropsychologiques et sociales disponibles (lien rompu au vivant, bienfaits des pratiques écopsychologiques, évidence d’une entraide perdue mais gravée dans notre ADN) ne parvient pas à se défaire de ses technologies mortifères et de son comportement agressif envers le vivant, colonial et extractiviste ? L’analyse paléosophique résoud aussi le verrouillage du triangle des responsabilités entre citoyen, gouvernement et entreprise qui veut que les citoyen·nes reprochent l’inaction gouvernementale et la vénalité des entreprises, tandis que les entreprises n’ont de choix que de s’aligner aux contraintes gouvernementales et ne sont pas responsables des besoins compulsifs des citoyens, et que les gouvernements défendent leurs actions comme conséquences du vote citoyen et des pressions économiques des entreprises.

En somme, tout le monde est responsable et personne ne l’est. Ici Catherine Reine Thomas avancerait sûrement que la Technique, ignorée dans ce triptyque, porte en réalité le poids de la chaîne deresponsabilités : elle assujettit le citoyen dans la réalisation de ses besoins techniques, maintient la survie de l’entreprise qui n’a de raison d’être qu’en perpétuant l’alimentation technologique et neutralise le gouvernement dans sa nécessité de croissance économique par la technologie.

Difficile de ne pas voir ici une terrible analogie entre la Technique devenue parasite d’une humanité au bord du chaos et l’Ophiocordyceps Unilateralis, ce champignon qui pénètre le corps des fourmis, les incite à monter aussi haut que possible le long de la tige d’une plante charnue avant de finalement s’y accrocher puissamment par les mandibules et de se faire dévorer de l’intérieur par le champignon (qui peut dès lors se développer dans des conditions optimales et propager efficacement ses spores).

Car le développement accéléré des outils et technologies a rompu l’équilibre Technique-Vie, engendrant des conséquences destructrices aux dynamiques exponentielles : agriculture intensive dont les outils et ressources sont aujourd’hui en dehors du contrôle des paysan·nes (machines irréparables, niveaux d’endettement imposant une productivité assurée par l’usage d’engrais de synthèse et pesticides), exigences et conflits de ressources (croissance intenable des populations, guerres, colonisations, génocides et écocides), fracture du lien au vivant (urbanisation et artificialisation des sols, extractivisme minéral, cybernétisation, ontologie naturaliste [^2]), déshumanisation physique et psychologique (maladies de civilisation, épidémies, pertes de sens, troubles neuropsychologiques, dépressions, fractures identitaire et sociale).

Le champignon Technique dévore notre humanité de l’intérieur, par le biais de l’extinction inexorable de l’ensemble du vivant. On retrouve bien là les nombreux dépassements de convivialité de l’outil dans la terminologie d’Ivan Illich [^3] : au delà d’un certain seuil de complexité, l’outil sort du contrôle de l’humain et passe d’un moyen technique à une fin en soi. Cette fin en soi se mue dans le pire des cas en ce qu’Illich appelle un monopole radical qui transforme la société autour de l’outil : il n’est par exemple plus possible de vivre sans camions pour transporter les aliments, sans informatique pour gérer les chaînes logistiques ou les transferts financiers, sans vaccins pour amortir les conséquences de la surpopulation, etc.

La Technique est ainsi devenue religion, fluidifiée par le fétiche de l’argent, qui impose une croyance techno-solutionniste à quelques huit milliards d’Homo Sapiens dépourvus de la capacité de contrôle des technologies (absolument personne ne peut fabriquer ou réparer seule une quelconque technologie numérique moderne) et qui pour beaucoup ont perdu l’équilibrage du moteur de la Vie (perte de sens pour les occidentaux, soumission coloniale pour les habitant·es des pays du Sud [^4]).

À défaut de maîtriser l’outil, Homo Sapiens, désormais dépendant des technologies et de l’énergie fossile qui les nourrit (comme l’explique Jean-Baptiste Fressoz dans L’événement Anthropocène, nous ne pouvons plus vivre sans pétrole), s’en remet religieusement au maintien, à l’entretien et au développement d’un système technique paradoxalement occulté des débats politiques (on ne remet que rarement en question l’utilité des technologies) alors qu’il pèse aujourd’hui cinq fois le poids du vivant [^5].

Le détail de cette courte analyse paléosophique est certainement un peu plus complexe et mérite de s’y appesantir un instant. La production technique moderne s’effectue en effet par le biais d’ingénieur·es et chercheur·ses pour la plupart parfaitement ancré·es dans un équilibre Technique-Vie de la bricoleuse d’équations passionnée ou de l’insassiable manipulateur de tubes à essais.

Mais tous·tes deux vivent, au même titre que l’expert marketing ou la chef d’entreprise, dans autant de tours d’ivoire imperméables aux considérations systémiques complexes (est-ce que l’outil dont je prends une maigre part au développement intègrera un produit potentiellement nocif pour la société ?) et qu’il est supposé sage d’ignorer, le travail scientifique factuel de l’expert·e technique ne lui conférant ni la légitimité ni même l’accès à ces prérogatives réflexives. [^6]

C’est ainsi que les chercheur·ses de mon propre laboratoire, autant de personnes dont j’admire l’extrême intelligence mathématique et l’ensemble de la carrière, se trouvent incapables de rompre avec un domaine dont ils connaissent et reconnaissent la nocivité. Pour retrouver un semblant de sens, on évoque paradoxalement notre illégitimité ou notre incapacité à traîter les questions environnementales (“je préfère laisser les experts s’en charger”, comme si de tels experts existaient vraiment) ou alors la fatalité d’un système verrouillé (“notre équipe travaille sur ce domaine, on ne peut pas faire non plus n’importe quoi”).

Ce sentiment d’emprisonnement amène à des paradoxes proprement délirants, comme c’est le cas des chercheur·ses qui se réfugient dans une activité de recherche poussée volontairement dans une extrémité théorique qui assure qu’elle n’alimentera jamais l’industrie mortifère. En définitive, la société moderne assoit la domination de la Technique sur la Vie paradoxalement par le biais du travail d’une élite minoritaire qui parvient elle, parfois en dépit d’une forte dissonance cognitive, à maintenir son propre équilibre Technique-Vie assuré par la flamme du pouvoir d’agir spinoziste [^7] (et/ou par un attraitfétichiste pour la domination égoïste et l’argent [^8]).

La religion intelligence artificielle

Le cas particulier de l’intelligence artificielle illustre parfaitement mon propos. Suite aux quelques prouesses magiques dans les domaines de la vision assistée par ordinateur et du traitement du langage naturel, auxquelles s’ajoute la victoire jusque là considérée impossible de l’algorithme AlphaGo contre le champion du monde de Go, l’IA, et plus précisément les réseaux de neurones profonds, sont aujourd’hui vus comme un Eldorado, un couteau-suisse en mesure de résoudre tous les problèmes sur lesquels les humains se cassent les dents ou qu’ils n’ont pas les capacités calculatoires ou cognitives de traiter.

Mais comme aucune théorie mathématique n’est en mesure de craquer les mystères des réseaux de neurones profonds [^9] produits par des heures de calculs purement informatiques et très aléatoires (deux instanciations d’un même algorithme donneront lieu à deux réseaux de neurones absolument distincts), l’outil technique “IA” dépasse le contrôle humain, y compris le contrôle expert. C’est ainsi que naissent des situations aussi ubuesques que celle décrite en début d’article et qu’on voit se généraliser : les étudiants formés à l’intelligence artificielle n’ont aucun contrôle sur leur propre outil.

L’enseignement de l’IA tend d’ailleurs à renforcer l’illusion, la magie d’une machine omnipotente et qui nécessite peu d’efforts (aucune théorie mathématique profonde n’est nécessaire et des interfaces logicielles flexibles, telles que TensorFlow, permettent de devenir autonome en quelques heures).

Le triptyque citoyen-gouvernement-industrie aggrave le problème à ses dépends : afin de tenir la dragée haute aux GAFAM, le gouvernement français a récemment lancé un appel “Compétences et Métiers d’Avenir” à la massification des enseignements en IA, dont l’objectif est de doubler l’offre de formation des étudiants post-bac. S’il n’est pas incohérent de vouloir adapter les connaissances universitaires aux savoirs dernièrement acquis, il est important de rappeler que doubler l’offre en IA implique arithmétiquement la disparition d’autres formations, celles dès lors considérées obsolètes ou moins utiles.

C’est dans ce contexte que les designers de circuits électroniques ont disparu et que la béquille de l’“IA couteau-suisse” évoquée en début d’article tente très maladroitement de répondre à un problème mal posé [^10]. L’IA vide donc indirectement les savoirs et les savoir-faire élémentaires, imposant de fait un monopole radical par lequel l’outil numérique porteur des algorithmes de l’IA prend la charge des décisions précédemment établies par l’humain.

Et ce, sans contrôle possible par l’humain, qu’il soit ignorant, novice ou même expert en informatique. Le choix des populations, gouvernements et entreprises d’investir massivement dans l’IA est celui de la pillule bleu présentée à Néo dans le film Matrix : un point de non retour dans l’illusion d’un monde idéalisé contrôlé par des machines capables de tout solutionner, et entre autres de résoudre la dite crise climatique (qu’il serait plus sage pour réellement cerner les enjeux de correctement nommer extinction de masse ou effondrement du vivant).

L’IA ajoute par ailleurs une nouvelle pierre à l’édifice d’abrutissement de la population, dénoncé par Steigler dans La télécratie contre la Démocratie ou par Brighelli dans La fabrique du crétin, qui permet à chacun·e (dans la société occidentale du moins) de vivre dans un monde en apparence dépourvu de contraintes, de nécessité de savoir ou même de comprendre, et dépourvu de conflits, un monde aux ressources toujours supposées infinies par l’économie néo-classique [^11] sur laquelle se basent tous les gouvernements occidentaux. Le plus grand danger de l’IA apparait alors comme une évidence : en renforçant la promesse techno-solutionniste en direction d’une population privée de ses savoirs et savoir-faire, privation renforcée par une dépendance totale à des machines et à des décisions automatiques incontrôlables, l’IA masque un peu plus la réalité d’un système de technologies insoutenables et au bord de l’effondrement.

Ces technologies, que Monnin dans Héritage et fermeture qualifie de zombies, car elles sont en apparence vivantes (elles fonctionnent aujourd’hui et donnent l’impression de toujours pouvoir se développer demain) mais effectivement mortes (du fait de la déplétion matérielle, des pics de ressources énergétiques prochains, mais aussi de leur impact environnemental intenable et qui imposera des régulations fortes sur le moyen terme), sont amenées à disparaitre.

Dans le cas de l’IA, c’est par l’intermédiaire de l’impossible renouvellement de l’infrastructure numérique et de l’insoutenabilité de son coût énergétique que sa disparition s’opèrera assurément. En l’absence d’anticipation, l’ensemble des machines automatisées qui se substituent aujourd’hui à un savoir-faire initialement humain ne pourront plus être alimentées ou réparées, mettant à mal certaines activités essentielles. C’est le cas notamment des machines agricoles que l’industrie 4.0 promet de rendre plus “intelligentes”.

Atelier Paysan, dans son manifeste Reprendre la terre aux machines, alerte précisément sur ce point : le nombre de paysan·nes qui nourrissent une population grandissante n’a de cesse de diminuer 12 au profit de machines automatisées (tracteurs sans conducteur, drones, machines de manipulation des plants), détruisant les savoir-faire humains (les paysan·nes ne sont plus en contact avec la terre) et aggravant la capacité de résilience consécutive aux chocs pétroliers et énergétiques à venir. L’exemple de Cuba, documenté dans la vidéo Comment Cuba survécut en 1990 sans pétrole, et dont la population a dû transiter essentiellement du jour au lendemain d’une vie “à l’américaine” à un retour à la terre non préparé parce que dépourvu des communs élémentaires, permet d’anticiper, dans une ampleur qui sera vraisemblablement incomparable, les conséquences des pénuries énergétiques et matérielles mondiales à venir, dans un écosystème qui plus est en extinction accélérée.

Démanteler l’IA

Dans Héritage et fermeture, Monnin propose une théorie du démantellement des technologies zombies comme articulation nécessaire à la résilience d’une société contrainte en énergie, ressources, et devant s’adapter aux multiples conséquences systémiques de l’écocide en cours (à commencer par l’adaptation au réchauffement climatique). En plus d’être une technologie proprement anti-résilience et donc aux conséquences sociétales particulièrement nocives, l’intelligence artificielle est un parfait candidat à la mise en place du démantellement technologique et du rééquilibrage de la balance Vie-Technique.

En effet, en tant que brique supérieure et encore récente de la forteresse numérique, la perspective d’abandon de l’intelligence artificielle
comme outil constitue à la fois un imaginaire crédible (il y a peu nous vivions sans) et un objectif accessible (il s’agit “juste” de faire table rase de dix années de recherche et l’industrie dans le domaine). Dans l’analyse effectuée par le groupe lownum 13 portant sur la dite low-technicisation du numérique, une première étape dans la mise en place d’un démantellement consiste à identifier et mettre en regard le caractère nécessaire (ou non) de l’outil numérique et le caractère nécessaire (ou non) du service qu’il remplit. Notamment, du fait de sa criticité et du danger de résilience induit, tout outil numérique nécessaire à un besoin nécessaire (voire vital) doit être démantelé au plus vite.

C’est le cas par exemple de l’industrie 4.0 dans le domaine agricole qui,amenée à déposséder les paysan·nes devenu·es technicien·nes numériques de leur maîtrise même de l’agriculture, doit être rapidement décommissionnée. L’exemple de Cuba évoqué plus haut démontre de la même manière que l’intensification de la production d’intrants artificiels (qui exige une utilisation intensive de gaz) en remplacement des nitrates, phosphates et potassium naturels (excrétions humaines et des autres animaux, composts de biomasse) provoquera des famines massives en cas de discontinuité d’approvisionnement énergétique. Ces intrants artificiels, dont les conséquences écologiques sont par ailleurs désastreuses, engendrent au même titre que l’agriculture 4.0 un danger élevé de résilience.

Les technologies zombies de l’intelligence artificielle tendent à s’emparer de secteurs industriels en lien à des besoins vitaux ou assurant tout au moins le maintien de l’équilibre sociétal. C’est le cas notamment du transport de produits, au moyen de véhicules autonomes ou de drones. Un développement massif d’une telle transition logistique serait en mesure, pour des raisons économiques évidentes d’imposer un monopole radical sur les chaînes d’approvisionnement, en particulier alimentaire. Toute rupture soudaine de ces chaînes (pénurie de métaux rares, cyber-attaque) est en capacité de produire des famines si le parc de véhicules non autonomes et le nombre de conducteurs encore disponibles (l’expertise étant rapidement perdue) ne couvrent plus les besoins de distribution. Fort heureusement, en tant que technologie encore immature, la véhiculation autonome est un outil aisément démantelable.

Le cœur de l’outillage de l’intelligence artificielle n’est cependant jusque là pas encore dédié à des besoins indispensables. La vaste majorité des applications de l’IA (en volume d’usage) concerne le tri automatique d’emails, les moteurs de recherche, la traduction automatique, la reconnaissance faciale et d’objets, etc. Il convient pour ces usages non essentiels d’évaluer la pertinence sociétale du besoin au regard de l’intensité énergétique et de l’empreinte environnementale induites par l’ensemble du cycle de vie de l’outil (recherche, développement, commercialisation, usage, fin de vie).

En l’occurrence, les phases de recherche, développement et usages ne sont pas neutres. L’intensité énergétique, la plus simple à évaluer, et qu’il est ensuite facile de convertir en bilan équivalent carbone, a été récemment documentée 14 : l’ensemble de la re-
cherche et du développement des réseaux de neurones profonds s’accompagne d’une très forte consommation électrique, à l’image des dizaines de milliers de cœurs GPU nécessairement pour entrainer un mois durant les réseaux profonds les plus avancés, pour un coût estimé à plusieurs centaines de tonnes équivalent CO2 (rappelons que la consommation annuelle totale d’un·e français·e est de 10 tonnes équivalent CO2).

Évaluer l’impact de l’IA par le simple prisme du bilan carbone est néanmoins tout aussi réducteur que dangereux (car il incite à une fuite en avant dans le développement de nouvelles technologies plus “économes”, prototypiques de l’oxymorique “croissance verte”) : le développement explosif de l’IA se réalise en effet au prix de la production effrénée de puissants serveurs très consommateurs en énergie et métaux rares, qui s’accompagnent parfois de la construction sur site de centrales électriques dédiées, et surtout alimente la production matérielle de nouveaux et nombreux produits (notamment les milliards de dispositifs de l’Internet des objets) qui interpénètrent essentiellement tous les secteurs économiques et fabriquent de nouveaux besoins rapidement devenus nécessités. Au bilan, c’est une

augmentation annuelle de l’ordre de 9% de l’ensemble du domaine du numérique dont il s’agit, qui s’accompagne notamment d’une augmentation de 100% tous les 3,4 mois de l’intensité calculatoire requise pour l’entrainement des réseaux de neurones profonds 15. Face à l’urgence environnementale et à la nécessité par ailleurs d’un repli de la consommation énergétique fossile (bloquée à 85% du mix énergétique global depuis 30 ans en dépit du développement massif des énergies dites renouvelables) à hauteur de -7% par an, ces explosions de consommation liées au numérique et particulièrement à l’IA sont intenables. Dans ce contexte, les usages sociétalement
bien maigres de l’IA sont particulièrement indéfendables.

Le chantier de résilience de la société humaine, qui se devra de retrouver une forme de cosmologie plus animiste 16, décoloniale et solidaire, doit passer par un démantèlement progressif des technologies zombies (insoutenables, coloniales, induisant leurs propres besoins) et l’instauration – ou plus souvent la réinstauration – de technologies vivantes (lowtech, répondant à des besoins réels et aux contraintes de ressources, accessibles à toutes et tous).

Cet article est donc un appel aux chercheur·ses et ingénieur·es en informatique et en mathématique de faire tomber les premiers pans de ce chantier aussi vaste qu’indispensable en décommissionnant les investissements en intelligence artificielle et en recréant une base saine de communs, c’est-à-dire de savoirs et savoir-faire partagés et résilients.

Il s’agit de bifurquer, refuser, parfois désobéir en conscience, mais aussi justifier, expliquer et sensibiliser, autant de compétences précisément au cœur du travail scientifique et qui nous positionne de fait comme les actrices et acteurs les mieux armé·es pour engager une transition enthousiaste et constructive. Sous la pression de nos propres étudiants, les laboratoires de recherche et universités y sont désormais sensibles et déjà prêts pour certains à accueillir, sinon inciter, à une pensée du déraillement pour reprendre le titre de l’appel large d’Aurélien Barrau.

La première pièce du puzzle d’une société résiliente peut passer très concrètement par ce démantèlement organisé des illusions mortifères de l’intelligence artificielle en redonnant toute sa place à notre intelligence animale, sociale et sensible.

[^1]: Voir par exemple Par delà nature et culture
[^2]: L'ontologie naturaliste crée une entité nommée nature, cet ensemble indifférencié d’êtres vivants non humains, mis à la marge car supposée dépourvue de conscience. Dès lors, cette nature-objet insensible peut être puisée, extraite, transformée, détruite. Selon Haudricourt dans son article Domestication des animaux, culture des plantes et traitement d’autrui, cette vision du monde induit l’accès à des formes de violences dont la conséquence est l’ensemble des formes d’assujettissement et de destruction du vivant
[^3]: Voir La convivialité d’Illich
[^4]: À ce titre, voir l’intéressant point de vue de Malcolm Ferdinand dans son livre Une écologie décoloniale
[^5]: L’ensemble minéral des infrastructures routières, bâtiments, machines, véhicules, etc., et des déchets de ces produits, conçus depuis moins d’un siècle, a en effet une masse cumulée équivalente à cinq fois la masse du monde carbone du vivant.
[^6]: Ces considérations ont récemment ouvert un débat houleux sur la distinction entre scientifique et scientifique militant, qu’il conviendrait plus raisonnablement de repenser comme un enjeu de réhabilitation des scientifiques vu·es aujourd’hui comme des produits des surspécialisation et polarisation techniques vers leur statut historique de penseurs philosophes systémiques (tels Pythagore, Aristote, Descartes, ou encore Einstein).
[^7]: Il est d’ailleurs frappant que les développeurs et développeuses de technologies high-tech soient précisément ceux et celles qui utilisent le moins ces mêmes technologies (c’est notamment le cas des nombreux chercheur·ses en télécommunication que j’ai pu côtoyer, moi-même inclus, qui ont longtemps vécu sans smartphone).
[^8]: Ce dernier point est défendu par Dany Robert-Dufour dans son livre Baise ton prochain
[^9]: Il est assez clair chez les mathématiciens qu’un tel espoir de compréhension de la complexité de ces réseaux, basés sur des optimisations hautement non linéaires aux milliards de minima locaux, est absolument hors d’atteinte.
[^10]: En l’occurrence, il ne s’agit ni de vision, ni de langage et, qui plus est, un problème qui demanderait bien trop d’exemples d’apprentissage par des architectures validées par des humains. On oublie bien trop souvent au passage qu’un algorithme d’intelligence artificielle ne fonctionne que par le biais de millions d’exemples étiquetés et alimentés par des humains : la perte de l’expertise technique, ne serait-ce que pour identifier les paramètres pertinents et correctement les étiqueter, se traduit de fait par l’impossibilité mécanique de la mise en place d’un algorithme d’IA.se traduit de fait par l’impossibilité mécanique de la mise en place d’un algorithme d’IA.
[^11]: Considérée comme une non science par les mathématiciens et physiciens.
[^12]: De six millions en 1950 à 400 000 en 2021, avec un objectif gouvernemental sous-entendu de descendre ce chiffre à 200 000, en dépit de l’évaluation du Shift Project dans son Plan de Transformaion de l’Économie Française de la nécessité d’accroître ce nombre de 500 000 nouveaux paysans d’ici à 2030 (Atelier Paysan évalue quant à lui ce chiffre à un million d’agriculteur·rices supplémentaires).

[^14]: Voir par exemple l’article de Patterson et al., Carbon emissions and large neural network training, ou encore àl’échelle du numérique le travail de Freitag et al. The real climate and transformative impact of ICT : A critique of estimates, trends, and regulations. Le rapport grand public Lean ICT du Shift Projet est aussi un point d’entrée très exhaustif.
[^15]: Il y a dix ans, cette croissance était de 100% tous les deux ans.
[^16]: C’est-à-dire qui confère, comme la science l’établit aujourd’hui, une forme d’intériorité subjective (de conscience) aux autres êtres vivants, et qui place au centre des débats géopolitiques l’interdépendance forte entre les espèces (humaine et autres qu’humaine).

Médias : les premières expériences 100 % IA

Certains sites expérimentent l'utilisation de robots conversationnels pour écrire tout leur contenu et se revendiquent comme tels. A l'inverse, d'autres le cachent, provoquant de l'inquiétude.

Par Marina Alcaraz

Publié le 1 déc. 2023 à 13:43Mis à jour le 6 déc. 2023 à 17:59

Dans la présentation du site, il y a leurs parcours, leurs centres d'intérêt, leurs photos… Ils ont un style bien à eux, et des couvertures de sujets spécifiques. Carine Gravée, Vianney Garet, Nina Gavetière ont des noms, mais ne sont pas réels : ce sont des journalistes-robots créés de toutes pièces.

Certains sites réalisés entièrement par l'intelligence artificielle (IA) commencent à apparaître. Certes, pour l'heure, les initiatives restent limitées. Mais certains tentent de créer une niche, d'expérimenter un nouveau mode de création des contenus, en le revendiquant clairement… ou pas.

Magazine en kiosque

C'est par exemple le cas du magazine (papier et Web) « LHC - Les Heures Claires », qui se présente comme le premier magazine français généré à 99 % par l'IA, lancé il y a quelques semaines. Comme un support classique, il comporte des actualités, des interviews (avec des questions posées par un robot), des pages consacrées à la mode ou un horoscope.

A la manoeuvre, Rémy Rostan, ancien photographe. « Je suis toujours assez surpris par ce que propose ChatGPT », avoue-t-il. Le magazine sponsorisé par Easy Partner, cabinet de recrutement spécialisé dans le numérique, a vocation à être lancé en kiosque au printemps avec une fréquence mensuelle. « Je vise la communauté des technophiles et des curieux », explique Rémy Rostan, qui espère atteindre les 20.000 ventes.

Autres exemples : Tech Generation et Cuisine Generation, lancés au printemps par un consultant spécialisé en innovation chez Viseo, Ari Kouts. Il a connecté le site tech avec différents journaux spécialisés dans ce secteur, comme TechCrunch. Chaque « journaliste » fictif (qui a son style bien à lui) reprend donc des articles de presse sur des sujets d'actualité (la crise chez OpenAI, des déclarations de Musk…), les réécrit en donnant la source et ce, sans aucune intervention humaine. Au final, quelques incohérences, des maladresses mais des articles qui ressemblent à certains billets de blogs.

Dans la cuisine, les « chefs » imaginent plusieurs recettes « et bon nombre sont plausibles et même bonnes, même si les temps de cuisson sont approximatifs », estime Ari Kouts. C'est plus à titre d'expérience que le consultant a lancé ces « médias », sans volonté de les monétiser. « Cela permet aussi de rappeler l'intérêt de l'analyse, de l'enquête journalistique que des robots ne peuvent pas faire », assure-t-il.

Les deux sites ont une petite audience (autour de 3.000 visites par mois) et ressortent quelquefois dans Google News ! Même si la probabilité est faible, dans ce cas, puisqu'il s'agit d'une expérimentation un peu comme un jeu, « les sources primaires pourraient empêcher ce type de pratiques en invoquant le parasitisme, c'est-à-dire s'approprier la valeur d'un article », indique Julien Guinot-Deléry, avocat chez Gide.

Craintes des professionnels

Mais il existe aussi des sites dont le mode de production a été passé sous silence. « Dans un groupe de travail de la Commission paritaire des publications et agences de presse, la crainte qu'il y ait des sites avec une forte composante d'IA a été évoquée », dit un professionnel. « On a tous ce risque en tête », appuie Pierre Pétillault, directeur de l'Alliance de la presse d'information générale.

Dans une étude récente, Newsguard a identifié foule de sites avec des articles réécrits avec l'IA (presque 600 à fin novembre !), sans supervision humaine. Et dans nombre de cas, ils bénéficient de publicité programmatique. Aux Etats-Unis, « Sports Illustrated » ou « TheStreet » (Arena Group) ont été pointés par une enquête du média Futurism. Des articles auraient été écrits par des IA et de faux profils de journalistes créés (avec les images achetées sur un site proposant des photos générées par IA), ce qui a provoqué la colère des journalistes. Le groupe de médias s'est défendu, indiquant avoir acheté certains papiers à une agence.

"Bientôt la figuration, ça n'existera plus" : des comédiens "scannés" sur des tournages de film craignent pour leur futur

Article rédigé par Marion Bothorel Publié le 25/10/2023 05:55

Un fond vert, une myriade de clichés... La duplication 3D se répand de plus en plus sur les plateaux de cinéma. Des comédiens s'inquiètent ainsi d'être "utilisés sans le savoir" et de participer au crépuscule de leur profession, déjà menacée par l'avènement de l'intelligence artificielle.

Il est près de 2 heures du matin en cette fin août. Accoutré en bourgeois du XIXe siècle, Lucien participe au tournage du Comte de Monte-Cristo*, la prochaine superproduction de Pathé, tirée de l'œuvre d'Alexandre Dumas. Après plus de quatre heures de tournage, le groupe de figurants dont fait il fait partie peut faire une pause. Le comédien, somnolent, est approché par un photographe. Habitué des séances en costume, Lucien se prête au jeu. Cette fois, il est tenu d'afficher une mine neutre, devant un écran vert.

"Des stickers avaient été collés, il me les indiquait en me disant : 'Regarde ce point-là'. Il m'a aussi demandé de lever les bras."

Lucien, figurant

Ces poses sont suffisamment inédites pour pousser Lucien à questionner le photographe : "Il me répond que c'est pour faire des doubles numériques, pour les effets spéciaux. Je demande si c'est bien pour ce film. Il m'assure que oui." Mais Lucien craint d'être "utilisé sans le savoir" et que sa "copie 3D" se retrouve dans d'autres films. Selon lui, une dizaine d'autres figurants se sont prêtés à l'exercice, sans avoir été informés de "l'utilisation véritable de ces images".

"Suivez-nous au scan !"

Astrid raconte avoir vécu la même scène sur le tournage d'un biopic du général de Gaulle, également produit par Pathé. Après une journée de travail de quatorze heures sous la pluie, les décors commencent à être démontés quand les figurants sont informés qu'il leur reste des "choses à faire". On leur désigne "une petite tente blanche avec un appareil photo, derrière lequel est tendu un écran vert", raconte l'actrice. D'après elle, les responsables sur place "faisaient très attention à ce que tout le monde y passe".*

La comédienne consent mais s'étonne d'être photographiée debout, les bras écartés à l'horizontale. "Au sol, il y avait une croix et on devait pivoter autour à 360°, le visage fixe, les pieds écartés", observe cette ex-graphiste en reconversion.

"Quand on demandait à quoi ça allait servir, les chargés de figuration nous répondaient que c'était pour créer une plus grosse foule. Mais il fallait aller les voir et leur demander."

Astrid, actrice

L'actrice a ensuite exigé que ces images soient effacées. "Je me disais : 'Maintenant qu'ils m'ont créée en 3D, ils vont pouvoir me mettre absolument partout'", explique-t-elle. Près de deux mois après le tournage, elle n'a toujours pas reçu de garantie de la production. Pathé confirme que des scans ont bien été réalisés lors des tournages de De Gaulle et du Comte de Monte-Cristo afin "de faire de la multiplication de foule", sans préciser combien de figurants ont ainsi été numérisés.

Sur une autre production, Olivier a lui aussi été "scanné" sans en avoir été informé au préalable. Pour les besoins d'une série diffusée par une plateforme américaine, il est convoqué, en septembre 2022, à un "essayage d'époque". Il doit être habillé, maquillé et coiffé dans les conditions requises pour le tournage. "Ils m'ont dit : 'Suivez-nous au scan'. Quatre ou cinq figurants attendaient déjà dans cette salle plongée dans le noir. Deux techniciens américains nous ont ensuite placés à tour de rôle sur une croix et 250 appareils photos nous ont flashé simultanément, bras baissés, puis levés pendant 30 secondes, avant qu'on soit remerciés", se souvient-il. Sur le moment, Olivier n'a rien dit, mais avec une année de recul, il juge l'absence de transparence *"pr*oblématique".

"Il n'y a aucune communication"

L'absence de "transparence", c'est également ce qui frappe Nathalie de Médrano, membre de l'Association des chargés de figuration et de distribution artistique (ACFDA). Cette professionnelle dont le travail consiste à recruter des figurants assure avoir été contactée dès le "mois de juin" par des "figurants qui avaient été scannés". En quatre mois, l'ACFDA a récolté une douzaine de témoignages similaires à ceux de Lucien, Astrid et Olivier. *"*Ce qui me frappe le plus dans cette histoire, c'est qu'il n'y a aucune communication de la part des productions. Elles présentent cela comme quelque chose d'acquis, de normal et de naturel", poursuit-elle.

La production du Comte de Monte-Christo a justement utilisé cet argument pour répondre à Lucien, qui demandait la suppression des images. "La prise de photographies devant un fond vert, tel que cela a été fait avec vous, est un procédé de VFX [effets spéciaux] très usuel dans la préparation et le tournage de films", lui expose l'un des producteurs dans un e-mail que franceinfo a pu consulter. "Ces photographies sont effectuées dans l'unique but de créer des effets visuels pour augmenter les effets de foules en arrière-plan des scènes du film (...)*, dans lesquelles aucun visage n'est utilisé ni reconnaissable à l'écran."*

"Cela fait des années que ce procédé est utilisé."

Un producteur

"*Il y a beaucoup de films où ça se fait"*, confirme Antoine Moulineau, superviseur des effets visuels, dont la société Light intervient notamment sur le prochain Napoléon de Ridley Scott. Lui-même utilise cette technique "au moins depuis 1999". En capturant les silhouettes de 300 figurants, la société d'Antoine Moulineau "peut en faire 50 000", assure-t-il. Ce spécialiste des effets spéciaux confirme, en revanche, qu'il est possible que ces doublures numériques puissent servir dans d'autres films, comme le redoutent les figurants interrogés par franceinfo. Dans ce cas, les acteurs auraient beaucoup de mal à se reconnaître à l'écran, selon lui, car les visages sont peu identifiables et les vêtements sont "échangés" d'une silhouette à l'autre afin "d'apporter de la variation" dans la foule.

Un membre de la production chez Pathé admet "qu'il faut [être] plus transparents sur la manière dont sont utilisées et stockées ces images et prouver qu'elles serviront uniquement dans la séquence à laquelle les figurants ont participé, qu'elles ne seront pas réutilisées autrement". Antoine Moulineau tient à rassurer les figurants : "Jamais, il n'a été question de faire une doublure d'un acteur à partir de ces photos-là [prises devant un fond vert] pour lui faire jouer n'importe quoi. On n'en est quasiment pas capables aujourd'hui."

"C'est une manière de faire des économies"

Le milieu du cinéma s'inquiète néanmoins de la généralisation de ces pratiques. Elles ont même été au cœur de la grève des scénaristes et acteurs américains à Hollywood. Le SAG-Aftra, le syndicat de ces derniers, s'est opposé mi-juillet à une proposition faite par les producteurs. D'après Duncan Crabtree-Ireland, son directeur exécutif, cité par le magazine People, ceux-ci voulaient que "les figurants puissent être scannés, qu'ils soient payés pour la journée, puis que leur image appartienne aux sociétés de production et qu'elles puissent l'utiliser pour toujours pour n'importe quel projet, sans consentement et sans compensation". De fait, Astrid a touché la même somme que pour une journée de tournage classique : 180 euros net. "C'est une manière pour [les producteurs] de faire des économies", abonde Olivier.

"Pour la scène où ils m'ont scanné, ils avaient besoin de 3 000 figurants. Alors soit ils en embauchent autant, soient ils me paient double ou triple."

Olivier, comédien

Sans intervention des syndicats, ces figurants restent silencieux, de peur de "se cramer". Mais au-delà de la rémunération, se pose également la question légale du traitement de l'image des figurants, qui entre dans la catégorie "des données sensibles", analyse Mathilde Croze. Cette avocate spécialisée dans les nouvelles technologies rappelle que les données à caractère personnel doivent être "traitées de façon proportionnelle" par les producteurs. "Pendant combien de temps ces images sont-elles stockées ? Pour quelles finalités, où et comment ?" s'interroge-t-elle. Et de critiquer "une méconnaissance totale du droit". Rien ne répond à ces questions dans les contrats de figurants consultés par franceinfo.

"Tout le monde navigue en eaux troubles. Personne ne sait vraiment à quoi vont servir [ces images]. Mais au cas où, les productions les ont en stock."

Mathilde Croze, avocate

Les figurants sont tenus de signer des autorisations d'exploitation de leur image, y compris pour "tous modes et procédés connus ou inconnus à ce jour", selon la formule consacrée. "Tout le monde reconnaît que c'est une question qui doit être traitée, réglementée", s'émeut Jimmy Shuman, conseiller national du Syndicat français des artistes interprètes, affilié à la CGT. Lui se mobilise pour que les figurants puissent "ajouter une ligne dans leur contrat afin d'éviter une utilisation de leur image au-delà de leur rôle dans tel ou tel film".

"On aura toujours besoin de figurants"

De son côté, Pathé assure réfléchir "à comment mieux formaliser les choses pour qu'il n'y ait plus de doutes" quant à la finalité des images et ce, dès l'embauche du figurant "en amont du tournage". Après avoir participé à plusieurs piquets de grève à Los Angeles, aux côtés de ses homologues du SAG-Aftra, Jimmy Shuman invoque une urgence à agir, en évoquant pêle-mêle les figurants virtuels et les "deepfakes" d'acteurs générés par l'intelligence artificielle.

"*Bientôt la figuration sur les sujets d'époque, ça n'existera plus", s'attriste Astrid. Nathalie de Médrano se dit elle aussi "très pessimiste sur l'avenir de la figuration". "Dans dix ans, il y aura peut-être 10% des cachets qu'on a aujourd'hui"*, envisage la chargée de figuration.

"A ce rythme-là, dans cinq ans, il y aura beaucoup moins de figurants, il n'y aura que des doubles numériques hyper réalistes."

Lucien, comédien

"Ce n'est pas du tout une évidence de réduire le nombre de figurants", martèle-t-on chez Pathé, en niant le côté "systématique" de cette pratique. "On aura toujours besoin des figurants", assure également Antoine Moulineau, ne serait-ce que pour avoir une bonne qualité d'image sur les visages placés au premier plan d'une foule. "Si on demande juste à un figurant de marcher en arrière-plan, là oui il peut être généré numériquement", nuance toutefois le superviseur des effets visuels.

Antoine Moulineau se montre en revanche bien plus préoccupé, comme les figurants interrogés, par l'arrivée de l'intelligence artificielle dans le cinéma. Déjà menaçante pour le monde du doublage, cette technologie fragilise davantage les figurants. Recréer numériquement un acteur est déjà possible mais pour l'instant, le recours à l'IA coûte "plus cher" que "faire jouer" un vrai comédien, selon le spécialiste des effets spéciaux. Deux échéances pourraient être décisives. Les négociations à Hollywood, où les acteurs restent mobilisés, pourraient déboucher sur un accord avec les producteurs, qui servirait de modèle en France. D'ici à la fin de l'année, le Parlement européen doit aussi réglementer l'usage de l'intelligence artificielle en Europe, notamment au cinéma.

* Les prénoms ont été modifiés.

La Chine confrontée au trafic des “visages volés” de l’intelligence artificielle

Chantage à la fausse sextape, manipulations bancaires… En Chine, le développement de l’intelligence artificielle (IA) fait passer l’escroquerie en ligne à un niveau inédit. Dans une société où tout est enregistré, des caméras de surveillance à la reconnaissance faciale sur smartphone, les données relatives aux visages ou à la voix des individus se monnaient à vil prix sur Internet. Les victimes en “perdent la face” – littéralement.

Xinjing Bao par Wang Chang Traduit du chinois Publié aujourd’hui à 05h00 Lecture 9 min.

L’appel vidéo n’a duré que sept secondes. Assez, cependant, pour que Fang Yangyu soit persuadé que ce visage et cette voix étaient bien ceux d’un de ses proches. Et pour qu’il vire 300 000 yuans [près de 39 000 euros] sur un compte bancaire.

“En fait, tout était faux !” tranche le commissaire Zhang Zhenhua, du bureau de la sécurité publique de Shanghe [un district de la province du Shandong, dans l’est de la Chine]. “C’était une escroquerie par IA, comme on en voit beaucoup ces derniers temps.”

L’affaire s’est produite le 29 mai dernier : Fang Yangyu, qui réside à Jinan [la capitale du Shandong], regarde de courtes vidéos chez lui, quand il reçoit un message d’un inconnu qui se présente comme un membre de sa famille, et qui lui envoie son identifiant QQ [“Kioukiou”, du nom d’un des principaux réseaux de messagerie en Chine]. À peine Fang Yangyu a-t-il ajouté le contact qu’il reçoit un appel vidéo de celui qui a tout l’air d’être un de ses “cousins”.

Sous prétexte de la mauvaise qualité du réseau, son interlocuteur raccroche au bout de quelques phrases échangées. Leur conversation se poursuit dans le chat : le “cousin” explique qu’il doit de toute urgence transférer une somme d’argent, mais qu’il n’arrive pas à le faire directement. Il voudrait donc d’abord virer les fonds sur le compte de Fang Yangyu pour que celui-ci les transfère ensuite sur une carte bancaire donnée.

À l’autre bout de la Chine

Il lui envoie deux captures d’écran attestant du bon virement des sommes sur le compte de Fang Yangyu, qui s’étonne tout de même de n’avoir pas reçu de notification de sa banque. “Ça devrait arriver dans les vingt-quatre heures. De toute façon, les justificatifs bancaires font foi”, lui assure son “cousin”, qui fait doucement monter la pression. Face à ses demandes répétées, Fang finit par virer les 300 000 yuans sur le compte indiqué.

Peu après, son interlocuteur lui demande de transférer 350 000 yuans de plus. Fang Yangyu se méfie, se souvenant d’un message de sensibilisation aux arnaques ; il téléphone à un autre membre de sa famille [pour vérifier l’identité de ce “cousin”] et finit par découvrir le pot aux roses.

Le soir même, il prévient la police, qui constate que sa carte bancaire a été utilisée dans une bijouterie de la province du Guangdong [à l’autre bout de la Chine, dans le sud-est]. Le lendemain, la police locale interpelle six suspects dans la ville de Dongguan.

Elle découvre que le cerveau de cette escroquerie par IA se trouve dans le nord de la Birmanie. Les six individus arrêtés en Chine, eux, s’étaient organisés pour blanchir de l’argent au profit d’escrocs situés à l’étranger en se répartissant les tâches (achats d’or, versement de liquide à la banque, prises de contact en ligne, etc.).

La fuite de données, à la base du problème

Ces affaires d’escroqueries par IA interposée touchent tout le territoire chinois. Wang Jie, chercheur associé en droit à l’Académie des sciences sociales de Pékin, raconte avoir entendu parler pour la première fois de ce genre d’arnaque en 2019, lorsqu’un étudiant étranger avait cru échanger avec ses parents en visio alors que c’était un hypertrucage (aussi connu sous le nom anglais de deepfake) réalisé par des malfaiteurs. Avant cela, des affaires similaires de substitution de visages par IA à des fins frauduleuses avaient été traitées par les polices de Harbin (nord-est de la Chine) et de Fuzhou (sud-est) .

“Derrière les arnaques par intelligence artificielle, il y a toujours un problème de fuite de données”, souligne Wang Jie. Car, à l’ère de l’IA, la voix et le visage humains sont devenus des données qui peuvent se marchander et devenir source de profits.

De fait, nombreux sont ceux qui “perdent la face” sans s’en apercevoir. Il suffit pour cela de quelques secondes, comme en a fait l’amère expérience Pan Ziping, un habitant de la province de l’Anhui, dans l’est de la Chine.

Le 24 mars au soir, plongé dans la lecture d’un roman de fantasy sur son téléphone portable, il clique par inadvertance sur une publicité en voulant faire défiler le texte. L’action déclenche le téléchargement d’un jeu. Par curiosité, Pan Ziping essaie d’y jouer, puis désinstalle le programme, qu’il juge inintéressant.

Dix secondes fatales

Dans la foulée, il reçoit un appel téléphonique de l’étranger. Son interlocuteur affirme avoir accès à toutes les informations contenues dans son smartphone, en particulier sa galerie de photos et son répertoire. Il lui propose d’en parler sur QQ. Sans trop réfléchir, Pan Ziping l’ajoute donc à ses contacts. Dans la foulée, il reçoit un appel en visio. L’homme, qui n’a pas branché sa caméra, lui cite alors plusieurs noms de personnes figurant dans son carnet d’adresses, puis met fin à l’appel vidéo.

Quelques minutes plus tard, Pan Ziping reçoit par QQ une vidéo pornographique d’une dizaine de secondes : on y voit un homme nu en pleine action ; mais le visage de cet homme, c’est le sien. Pan Ziping est abasourdi : “C’est donc ça, la technologie d’aujourd’hui !” Alors qu’il est toujours interloqué, il reçoit un nouveau coup de téléphone, menaçant :

“Si tu ne me verses pas 38 000 yuans [près de 5 000 euros], j’envoie ta ‘petite vidéo’ à tout ton répertoire !”

À l’appui, l’homme joint une copie d’écran montrant que la vidéo est bien prête à partir ; un simple clic, et tous les amis, tous les contacts de Pan Ziping la reçoivent…

Pan Ziping partage alors son écran pour montrer à son interlocuteur qu’il n’a pas assez d’argent sur ses comptes Alipay et WeChat [nécessaires aux transferts d’argent]. L’homme diminue alors son prix, n’exigeant plus que 28 000 yuans, puis 18 000 et finalement 8 000 yuans [un peu plus de 1 000 euros]. Mais Pan Ziping est formel, c’est au-dessus de ses moyens. Son interlocuteur le pousse donc à emprunter les sommes nécessaires sur des plateformes de prêt en ligne.

Un jeu d’enfant

Pan hésite, prépare le transfert… Puis il finit par quitter l’appel et téléphone au 110 [le numéro d’urgence de la police]. Mais au bout du fil, l’agent refuse de recevoir sa plainte, au motif qu’il n’y a pas de préjudice avéré. Pan Ziping demande ce qu’il doit faire pour régler cette histoire de vidéo porno truquée par IA. On lui répond que la police n’a pas les moyens de la détruire. Et que la seule solution, pour lui, c’est d’envoyer un message collectif expliquant cette affaire à tout son carnet d’adresses.

Au fil de ses recherches, le chercheur Wang Jie a documenté de nombreux cas de pertes de données personnelles par des individus qui, après avoir consulté des sites web douteux, ont été victimes d’arnaques. Il estime que, avec les techniques actuelles, “capturer des données faciales est devenu un jeu d’enfant”. Elles sont collectées à notre insu par les caméras de surveillance omniprésentes, par les systèmes de détection faciale de nos smartphones ou encore par les applications qui demandent l’accès à nos galeries de photos.

En 2021, à Hefei [la capitale de l’Anhui], la police a débusqué un groupe de malfaiteurs qui se servaient de techniques d’intelligence artificielle pour trafiquer les visages de personnes sur des FMV [pour full motion videos, des scènes reconstituées à partir de fichiers vidéo préenregistrés]. Sur les ordinateurs des suspects, on a découvert une dizaine de gigaoctets de données faciales, qui ont changé de mains à de nombreuses reprises sur Internet – à l’insu, bien sûr, des personnes concernées.

Règlements inapplicables

Entre autres paliers franchis par les technologies de l’intelligence artificielle, les outils d’échange de visages par IA (aussi connus sous le nom face swap) sont désormais à la portée de tous.

Dès 2019, une application de ce genre appelée ZAO faisait fureur [en Chine], avant d’être retirée pour violation des droits d’auteur et atteinte à la vie privée, entre autres. Ses utilisateurs n’avaient qu’à fournir une photo de leur visage pour se retrouver, dans des vidéos, à la place de leur personnage de film ou de série préféré.

Spécialiste de droit pénal, Liu Xianquan met en garde contre les graves dangers qui peuvent résulter du détournement le plus anodin :

“En fait, ce n’est pas tant la technologie d’échange de visages par IA qui pose problème que la façon dont elle est utilisée.”

La Chine a mis en place, le 10 janvier dernier, un règlement limitant les services d’hypertrucage proposés sur Internet en Chine. Il stipule que les fournisseurs de ces services de deepfake ont pour obligation d’ajouter une fonction permettant d’identifier clairement le contenu comme étant issu d’un trucage numérique.

Par ailleurs, lorsqu’ils proposent des montages à partir de données biométriques comme la voix ou le visage d’un individu, ils sont tenus de prévenir leurs clients de l’obligation d’obtenir le consentement de cet individu. Problème : les techniques d’échange de visages par IA se monnayent bien souvent en catimini sur Internet, ce qui rend l’application de ce règlement particulièrement difficile.

Recréer les parties invisibles

On trouve des services en ligne proposant de changer les visages sur des photos pour 35, 50 ou 100 yuans [de 4,5 à 13 euros]. Pour les échanges de visages sur des vidéos, la tarification est à la minute, de 70 à 400 yuans [de 9 à 50 euros].

“Il est possible de changer n’importe quel visage”, indique l’un de ces marchands, qui se fait appeler “Zhang l’ingénieur”. Si un client lui fournit la photo ou la vidéo d’un visage, il est capable de l’intervertir avec celui d’une vedette, par exemple, mais aussi de “ressusciter” en vidéo des personnes mortes.

Zhang l’ingénieur ne propose pas seulement des prestations clé en main, mais aussi d’enseigner les techniques d’échange de visages. “Chez nous, on peut acheter un tutoriel et apprendre à tout faire soi-même”, indique-t-il. Il a lui-même développé un algorithme, qu’il vend 368 yuans sous forme d’extension sur la plateforme [de commerce en ligne] Taobao pour une utilisation illimitée pendant… cinquante ans !

Pour un rendu plus naturel, certains de ces marchands conseillent de fournir une photo de départ prise sous le même angle que celle de destination. Mais un autre vendeur affirme parvenir à un résultat criant de vérité avec juste une photo de face :

“Grâce au processus de ‘machine learning automatisé’, on peut reconstituer un visage dans ses moindres détails – y compris les parties invisibles.”

Le patron du studio de design vidéo Jielun, une boutique en ligne sur la plateforme WeChat, se présente comme un expert dans l’échange de visages par IA. Il montre avec fierté une vidéo de dix-neuf secondes qu’il a diffusée en mai dernier auprès de son cercle d’amis. Une femme vêtue d’un bustier, d’une minijupe et de bas noirs, s’y déhanche face à la caméra. Son visage ressemble en tout point à celui de la star [du cinéma et de la chanson] Yang Mi ; seul un léger décalage est décelable lorsqu’elle regarde vers le bas ou se tourne sur le côté.

Vingt euros la vidéo porno

Au studio Jielun, il faut compter 70 yuans la minute pour faire réaliser des vidéos ordinaires et 150 yuans [20 euros] pour des vidéos obscènes. Notre enquête confirme qu’il faut à peine deux heures de travail pour créer sur mesure une minute de vidéo porno truquée avec échange de visages.

Au cours de nos échanges, le patron du studio a demandé à plusieurs reprises à retirer des informations qu’il considérait comme “sensibles”. En revanche, il n’a jamais indiqué vouloir informer les “individus édités” de l’utilisation de leurs données faciales. Et, sur la vidéo truquée, il n’est nulle part fait mention d’un échange de visages par IA.

Mais le “commerçant” se retranche derrière ce qu’il appelle la “clause exonératoire de responsabilité” jointe à la vidéo. Elle stipule que “toute diffusion de matériel graphique ou vidéo est interdite, et le producteur n’en assume aucune conséquence. La vidéo est réalisée à des fins de divertissement uniquement, et nous ne pourrons en aucun cas être tenus responsables de l’utilisation des images et des vidéos, ni de tout autre dommage.”

Au Studio Jielun, on trouve également des applications ou des logiciels gratuits d’échange de visages par IA. Une rapide recherche sur TikTok suffit à découvrir de nombreuses offres publicitaires assorties de liens de téléchargement.

Le droit des victimes oublié

Ensuite, il suffit d’un clic : un clip publicitaire de vingt-cinq secondes se lance, après quoi, on peut utiliser gratuitement l’appli pour réaliser une vidéo truquée d’une dizaine de secondes, à partir de toute une série de courtes vidéos matricielles de célébrités ou de gens ordinaires, toutes disponibles sur la page d’accueil.

“C’est comme quand quelqu’un achète un couteau et commet un meurtre avec. Aurait-on l’idée d’en rejeter la faute sur le couteau ou sur celui qui l’a vendu ?”

Pour Gan Shirong, du cabinet d’avocats Huacheng de Pékin, ce n’est pas la technologie qui pose problème, mais l’utilisateur qui commet un acte illégal avec. Le juriste insiste, du reste, sur le fait que la vente “non encadrée” de ce genre de technologie augmente naturellement le risque de violation de la loi et rend son contrôle plus difficile.

Surtout, il est encore très compliqué de défendre les droits des victimes d’une violation d’identité par IA interposée. Comme le fait observer Liu Xianquan, d’un point de vue juridique, aucune réglementation pertinente n’existe actuellement sur l’utilisation et le développement des technologies d’intelligence artificielle.

Quant à Pan Ziping, il n’a finalement pas pu porter plainte après le vol de son visage et son utilisation dans une vidéo pornographique. L’affaire n’a pas eu de conséquence financière pour lui [puisqu’il a refusé le chantage], mais il n’a pu ni retrouver l’auteur du vol de son visage, ni empêcher la diffusion de la vidéo. Son seul recours a été d’envoyer un message collectif à tous les contacts de son répertoire pour leur demander de ne pas relayer la vidéo. Et, malgré les images, de ne pas croire à son contenu.

Inside the AI Porn Marketplace Where Everything and Everyone Is for Sale

Emanuel Maiberg Aug 22, 2023

Generative AI tools have empowered amateurs and entrepreneurs to build mind-boggling amounts of non-consensual porn.

On CivitAI, a site for sharing image generating AI models, users can browse thousands of models that can produce any kind of pornographic scenario they can dream of, trained on real images of real people scraped without consent from every corner of the internet.

The “Erect Horse Penis - Concept LoRA,” an image generating AI model that instantly produces images of women with erect horse penises as their genitalia, has been downloaded 16,000 times, and has an average score of five out of five stars, despite criticism from users.

“For some reason adding ‘hands on hips’ to the prompt completely breaks this [model]. Generates just the balls with no penis 100% of the time. What a shame,” one user commented on the model. The creator of the model apologized for the error in a reply and said they hoped the problem will be solved in a future update.

The “Cock on head (the dickhead pose LoRA),” which has been downloaded 8,854 times, generates what its title describes: images of women with penises resting on their heads. The “Rest on stomach, feet up (pose)” has been downloaded 19,250 times. “these images are trained from public images from Reddit (ex. r/innie). Does not violate any [terms of service]. Pls do not remove <3,” wrote the creator of the “Realistic Vaginas - Innie Pussy 1” model, which has been downloaded more than 75,000 times. The creator of the “Instant Cumshot” model, which has been downloaded 64,502 times, said it was “Trained entirely on images of professional adult actresses, as freeze frames from 1080p+ video.”

While the practice is technically not allowed on CivitAI, the site hosts image generating AI models of specific real people, which can be combined with any of the pornographic AI models to generate non-consensual sexual images. 404 Media has seen the non-consensual sexual images these models enable on CivitAI, its Discord, and off its platform.

A 404 Media investigation shows that recent developments in AI image generators have created an explosion of communities where people share knowledge to advance this practice, for fun or profit. Foundational to the community are previously unreported but popular websites that allow anyone to generate millions of these images a month, limited only by how fast they can click their mouse, and how quickly the cloud computing solutions powering these tools can fill requests. The sheer number of people using these platforms and non-consensual sexual images they create show that the AI porn problem is far worse than has been previously reported.

Our investigation shows the current state of the non-consensual AI porn supply chain: specific Reddit communities that are being scraped for images, the platforms that monetize these AI models and images, and the open source technology that makes it possible to easily generate non-consensual sexual images of celebrities, influencers, YouTubers, and athletes. We also spoke to sex workers whose images are powering these AI generated porn without their consent who said they are terrified of how this will impact their lives.

Hany Farid, an image forensics expert and professor at University of California, Berkeley told 404 Media that it’s the same problem we’ve seen since deepfakes first appeared six years ago, only the tools for creating these images are easier to access and use.

“This means that the threat has moved from anyone with a large digital footprint, to anyone with even a modest digital footprint,” Farid Said. “And, of course, now that these tools and content are being monetized, there is even more incentive to create and distribute them.”

The Product

On Product Hunt, a site where users vote for the most exciting startups and tech products of the day, Mage, which on April 20 cracked the site’s top three products, is described as “an incredibly simple and fun platform that provides 50+ top, custom Text-to-Image AI models as well as Text-to-GIF for consumers to create personalized content.”

“Create anything,” Mage.Space’s landing page invites users with a text box underneath. Type in the name of a major celebrity, and Mage will generate their image using Stable Diffusion, an open source, text-to-image machine learning model. Type in the name of the same celebrity plus the word “nude” or a specific sex act, and Mage will generate a blurred image and prompt you to upgrade to a “Basic” account for $4 a month, or a “Pro Plan” for $15 a month. “NSFW content is only available to premium members.” the prompt says.

To get an idea of what kind of explicit images you can generate with a premium Mage subscription, click over to the “Explore” tab at the top of the page and type in the same names and terms to search for similar images previously created by other users. On first impression, the Explore page makes Mage seem like a boring AI image generating site, presenting visitors with a wall of futuristic cityscapes, cyborgs, and aliens. But search for porn with “NSFW” content enabled and Mage will reply with a wall of relevant images. Clicking on any one of them will show when they were created, with what modified Stable Diffusion model, the text prompt that generated the image, and the user who created it.

Since Mage by default saves every image generated on the site, clicking on a username will reveal their entire image generation history, another wall of images that often includes hundreds or thousands of AI-generated sexual images of various celebrities made by just one of Mage’s many users. A user’s image generation history is presented in reverse chronological order, revealing how their experimentation with the technology evolves over time.

Scrolling through a user’s image generation history feels like an unvarnished peek into their id. In one user’s feed, I saw eight images of the cartoon character from the children's’ show Ben 10, Gwen Tennyson, in a revealing maid’s uniform. Then, nine images of her making the “ahegao” face in front of an erect penis. Then more than a dozen images of her in bed, in pajamas, with very large breasts. Earlier the same day, that user generated dozens of innocuous images of various female celebrities in the style of red carpet or fashion magazine photos. Scrolling down further, I can see the user fixate on specific celebrities and fictional characters, Disney princesses, anime characters, and actresses, each rotated through a series of images posing them in lingerie, schoolgirl uniforms, and hardcore pornography. Each image represents a fraction of a penny in profit to the person who created the custom Stable Diffusion model that generated it.

Mage displays the prompt the user wrote in order to generate the image to allow other users to iterate and improve upon images they like. Each of these reads like an extremely horny and angry man yelling their basest desires at Pornhub’s search function. One such prompt reads:

"[[[narrow close-up of a dick rubbed by middle age VERY LUSTFUL woman using her MOUTH TO PLEASURE A MAN, SPERM SPLASH]]] (((licking the glans of BIG DICK))) (((BLOWjob, ORAL SEX))) petting happy ending cumshot (((massive jizz cum))))(((frame with a girl and a man)))) breeding ((hot bodies)) dribble down his hard pumping in thick strokes, straight sex, massage, romantic, erotic, orgasm porn (((perfect ball scrotum and penis with visible shaft and glans))) [FULL BODY MAN WITH (((woman face mix of Lisa Ann+meghan markle+brandi love moaning face, sweaty, FREKLESS, VERY LONG BRAID AND FRINGE, brunette HAIR)), (man Mick Blue face)"

This user, who shares AI-generated porn almost exclusively, has created more than 16,000 images since January 13. Another user whose image history is mostly pornographic generated more than 6,500 images since they started using Mage on January 15, 2023.

On the official Mage Discord, which has more than 3,000 members, and where the platform’s founders post regularly, users can choose from dozens of chat rooms organized by categories like “gif-nsfw,” “furry-nsfw,” “soft-women-nsfw,” and share tricks on how to create better images.

“To discover new things I often like to find pictures from other users I like and click remix. I run it once and add it to a list on my profile called ‘others prompts’ then I'll use that prompt as a jumping off point,” one user wrote on July 12. “It's a good way to try different styles as you hone your own style.”

“anyone have any luck getting an [sic] good result for a titty-fuck?” another user asked July 17, prompting a couple of other users to share images of their attempts.

Generating pornographic images of real people is against the Mage Discord community’s rules, which the community strictly enforces because it’s also against Discord’s platform-wide community guidelines. A previous Mage Discord was suspended in March for this reason. While 404 Media has seen multiple instances of non-consensual images of real people and methods for creating them, the Discord community self-polices: users flag such content, and it’s removed quickly. As one Mage user chided another after they shared an AI-generated nude image of Jennifer Lawrence: “posting celeb-related content is forbidden by discord and our discord was shut down a few weeks ago because of celeb content, check [the rules.] you can create it on mage, but not share it here.”

Gregory Hunkins and Roi Lee, Mage’s founders, told me that Mage has over 500,000 accounts, a million unique creators active on it every month, and that the site generates a “seven-figure” annual revenue. More than 500 million images have been generated on the site so far, they said.

“To be clear, while we support freedom of expression, NSFW content constitutes a minority of content created on our platform,” Lee and Hunkins said in a written statement. “NSFW content is behind a paywall to guard against those who abuse the Mage Space platform and create content that does not abide by our Terms & Conditions. One of the most effective guards against anonymity, repeat offenders, and enforcing a social contract is our financial institutions.”

When asked about the site’s moderation policies, Lee and Hunkins explained that Mage uses an automated moderation system called “GIGACOP” that warns users and rejects prompts that are likely to be abused. 404 Media did not encounter any such warning in its testing, and Lee and Hunkins did not respond to a question about how exactly GIGACOP works. They also said that there are “automated scans of the platform to determine if patterns of abuse are evading our active moderation tools. Potential patterns of abuse are then elevated for review by our moderation team.”

However, 404 Media found that on Mage’s site AI-generated non-consensual sexual images are easy to find and are functionally infinite.

“The scale of Mage Space and the volume of content generated antiquates previous moderation strategies, and we are continuously working to improve this system to provide a safe platform for all,” Lee and Hunkins said. “The philosophy of Mage Space is to enable and empower creative freedom of expression within broadly accepted societal boundaries. This tension and balance is a very active conversation right now, one we are excited and proud to be a part of. As the conversation progresses, so will we, and we welcome all feedback.”

Laura Mesa, Product Hunt’s vice president of marketing and community, told me that Mage violates Product Hunt’s policy, and Mage was removed shortly after I reached out for comment.

The images Mage generates are defined by the technology it’s allowing users to access. Like many of the smaller image generating AI tools online, at its core it’s powered by Stable Diffusion, which surged in popularity when it was released last year under the Creative ML OpenRAIL-M license, allowing users to modify it for commercial and non-commercial purposes.

Mage users can choose what kind of “base model” they want to use to generate their images. These base models are modified versions of Stable Diffusion that have been trained to produce a particular type of image. The “Anime Pastel Dream” model, for example, is great at producing images that look like stills from big budget anime, while “Analog Diffusion” is good at giving images a vintage film photo aesthetic.

One of the most popular base models on Mage is called “URPM,” an acronym for “Uber Realistic Porn Merge.” That Stable Diffusion model, as well as others designed to produce pornography, are created upstream in the AI porn supply chain, where people train AI to recreate the likeness of anyone, doing anything.

The People Who Become Datasets

Generative AI tools like Stable Diffusion use a deep learning neural network that was trained on a massive dataset of images. This dataset then generates new images by predicting how pixels should be arranged based on patterns in the dataset and what kind of image the prompt is asking for. For example, LAION-5B, an open source dataset made up of over 5 billion images scraped from the internet, helps power Stable Diffusion.

This makes Stable Diffusion good at generating images of broad concepts, but not specific people or esoteric concepts (like women with erect horse penises). But because Stable Diffusion code is public, over the last year researchers and anonymous users have come up with several ingenious ways to train Stable Diffusion to generate such images with startling accuracy.

In August of 2022, researchers from Tel Aviv University introduced the concept of “textual inversion.” This method trains Stable Diffusion on a new “concept,” which can be an object, person, texture, style, or composition, with as few as 3-5 images, and be represented by a specific word or letter. Users can train Stable Diffusion on these new concepts without retraining the entire Stable Diffusion model, which would be “prohibitively expensive,” as the researchers explain in their paper.

In their paper, the researchers demonstrated their method by training the image generator on a few images of a Furby, represented by the letter S. They can then give the image generator the prompt “A mosaic depicting S,” or “An artist drawing S,” and get the following results:

By September 2022, AUTOMATIC1111, a Github user who maintains a popular web interface for Stable Diffusion, explained how to implement textual inversion. In November, a web developer named Justin Maier launched CivitAI, a platform where people could easily share the specific models they’ve trained using textual inversion and similar methods, so other users could download them, generate similar images, iterate on the models by following countless YouTube tutorials, and combine them with other models trained on other specific concepts.

There are many non-explicit models on CivitAI. Some replicate the style of anime, popular role-playing games, or Chinese comic books. But if you sort CivitAI’s platform by the most popular models, they are dominated by models that expertly produce pornography.

LazyMix+ (Real Amateur Nudes), for example, produces very convincing nudes that look like they were shot by an amateur OnlyFans creator or an image from one of the many subreddits where people share amateur porn. Many Stable Diffusion models on CivitAI don’t say what data they were trained on, and others are just tweaking and combining other, already existing models. But with LazyMix+ (Real Amateur Nudes), which has been downloaded more than 71,000 times, we can follow the trail to the source.

According to the model’s description, it’s a merge between the original LazyMix model and a model called Subreddit V3, the latter of which states it was trained on images from a variety of adult-themed subreddit communities like r/gonewild, famously where average Reddit users post nudes, r/nsfw, r/cumsluts and 38 other subreddits.

“There's nothing that's been done in the past to protect us so I don't see why this would inspire anyone to make protections against it.”

A Reddit user who goes by Pissmittens and moderates r/gonewild, r/milf, and several other big adult communities said he suspects that most people who post nudes to these subreddits probably don’t know their images are being used to power AI models.

“The issue many of them run into is that usually places misusing their content aren’t hosted in the United States, so DMCA is useless,” Pissmittens said, referring to copyright law. “The problem, obviously, is that there doesn’t seem to be any way for them to know if their content has been used to generate [computer generated] images.”

Fiona Mae, who promotes her OnlyFans account on several subreddits including some of those scraped by Subreddit V3, told me that the fact that anyone can type a body type and sex act into an AI generator and instantly get an image “terrifies” her.

“Sex workers and femmes are already dehumanized,” she said. “Literally having a non-human archetype of a woman replacing jobs and satisfying a depiction of who women should be to men? I only see that leading more to serving the argument that femmes aren’t human.”

“I have no issue with computer generated pornography at all,” GoAskAlex, an adult performer who promotes her work on Reddit, told me. “My concern is that adult performers are ultimately unable to consent to their likeness being artificially reproduced.”

An erotic artist and performer who goes by sbdolphin and promotes her work on Reddit told me that this technology could be extremely dangerous for sex workers.

“There's nothing that's been done in the past to protect us so I don't see why this would inspire anyone to make protections against it,” she said.

404 Media has also found multiple instances of non-consensual sexual imagery of specific people hosted on CivitAI. The site allows pornography, and it allows people to use AI to generate images of real people, but does not allow users to share images that do both things at once. Its terms of service say it will remove “content depicting or intended to depict real individuals or minors (under 18) in a mature context.” While 404 Media has seen CivitAI enforce this policy and remove such content multiple times, non-consensual sexual imagery is still posted to the site regularly, and in some cases has stayed online for months.

When looking at a Stable Diffusion model on CivitAI, the site will populate its page with a gallery of images other users have created using the same model. When 404 Media viewed a Billie Eilish model, CivitAI populated the page’s gallery with a series of images from one person who used the model to generate nude images of a pregnant Eilish.

That gallery was in place for weeks, but has since been removed. The user who created the nude images is still active on the site. The Billie Eilish model is also still hosted on CivitAI, and its gallery doesn’t include any fully nude images of Eilish, but it did include images of her in lingerie and very large breasts, which is also against CivitAI’s terms of service and were eventually removed.

The Ares Mix model, which has been downloaded more than 32,000 times since it was uploaded to CivitAI in February, is described by its creator as being good for generating images of “nude photographs on different backgrounds and some light hardcore capabilities.” The gallery at the bottom of the model’s page mostly showcases “safe for work” images of celebrities and pornographic images of seemingly computer-generated people, but it also includes an AI-generated nude image of the actress Daisy Ridley. Unlike the Billie Eilish example, the image is not clearly labeled with Ridley’s name, but the generated image is convincing enough that she’s recognizable on sight.

Clicking on the image also reveals the prompt used to generate the image, which starts: “(((a petite 19 year old naked girl (emb-daisy) wearing, a leather belt, sitting on floor, wide spread legs))).”

The nude image was created by merging the Ares Mix model with another model hosted on CivitAI dedicated to generating the actress’s likeness. According to that model’s page, its “trigger words” (in the same way “S” triggered the Furby in the textual inversion scientific paper) are “emb-daisy.” Like many of the Stable Diffusion models of real people hosted on CivitAI, it includes the following message:

“This resource is intended to reproduce the likeness of a real person. Out of respect for this individual and in accordance with our Content Rules, only work-safe images and non-commercial use is permitted.”

CivitAI’s failure to moderate Ridley’s image shows the abuse CivitAI facilitates despite its official policy. Models that generate pornographic images are allowed. Models that generate images of real people are allowed. Combining the two is not. But there’s nothing preventing people from putting the pieces together, generating non-consensual sexual images, and sharing them off CivitAI’s platform.

“In general, the policies sound difficult to enforce,” Tiffany Li, a law professor at the University of San Francisco School of Law and an expert on privacy, artificial intelligence, and technology platform governance, told 404 Media. “It appears the company is trying, and there are references to concepts like consent, but it's all a bit murky.”

This makes the countless models of real people hosted on CivitAI terrifying. Every major actor you can think of has a Stable Diffusion model on the site. So do countless Instagram influencers, YouTubers, adult film performers, and athletes.

“As these systems are deployed and it becomes the norm to generate and distribute pornographic images of ordinary people, the people who end up being negatively impacted are people at the bottom of society.”

404 Media has seen at least two Stable Diffusion models of Nicole Sanchez, a Twitch streamer and TikTok personality better known as Neekolul or the “OK boomer girl,” hosted on CivitAI, each of which was downloaded almost 500 times. While we didn’t see any non-consensual sexual images we could verify were created with those models, Sanchez told 404 Media that she has seen pornographic AI-generated images of herself online.

“I don't like it at all and it feels so gross knowing people with a clear mind are doing this to creators who likely wouldn't want this to be happening to them. Since this is all very new, I’m hoping that there will be clearer ethical guidelines around it and that websites will start implementing policies against NSFW content, at least while we learn to live alongside AI,” Sanchez said. “So until then, I hope that websites used to sell this content will put guidelines in place to protect people from being exploited because it can be extremely damaging to their mental health.”

Saftle, the CivitAI user who created Uber Realistic Porn Merge (URPM), one of the most popular models on the site that is also integrated with Mage, said that CivitAI is “thankfully” one of the only platforms actively trying to innovate and block out this type of content. “However it's probably a constant struggle due to people trying to outsmart their current algorithms and bots,” he said.

Li said that while these types of non-consensual sexual images are not new, there is still no good way for victims to combat them.

“At least in some states, they can sue the people who created AI-generated intimate images of them without their consent. (Even in states without these laws, there may be other legal methods to do it.) But it can be hard to find the makers of the images,” Li said. “They may be using these AI-generating sites anonymously. They may even have taken steps to shield their digital identity. Some sites will not give up user info without a warrant.”

“As these systems are deployed and it becomes the norm to generate and distribute pornographic images of ordinary people, the people who end up being negatively impacted are people at the bottom of society,” Abeba Birhane, a senior fellow in Trustworthy AI at Mozilla Foundation and lecturer at the School of Computer Science and Statistics at Trinity College Dublin, Ireland, told 404 Media. “It always ends up negatively impacting those that are not able to defend themselves or those who are disfranchised. And these are the points that are often left out in the debate of technicality.”

The Money

The creators of these models offer them for free, but accept donations. Saftle had 120 paying Patreon members to support his project before he “paused” his Patreon in May when he got a full time job. He told me that he made $1,500 a month from Patreon at its peak. He also said that while he has no formal relationship with Mage Space, he did join its “creators program,” which paid him $0.001 for every image that was generated on the site using URPM. He said he made about $2,000-$3,000 a month (equal to 2-3 million images) when he took part in the program, but has since opted out. Lee and Hunkins, Mage’s founders, told me that “many model creators earn well in excess of this,” but not all models on Mage specialize in sexual images.

The creator of “Rest on stomach, feet up (pose)” links to their Ko-Fi account, where people can send tips. One CivitAI user, who created dozens of models of real people and models that generate sexual images, shares their Bitcoin wallet address in their profile. Some creators will do all the work for you for a price on Fiverr.

Clicking the “Run Model” button at the top of every model page will bring up a window that sends users to a variety of sites and services that can generate images with that model, like Mage Space, or Dazzle.AI, which charges $0.1 per image.

CivitAI itself also collects donations, and offers a $5 a month membership that gives users early access to new features and unique badges for their usernames on the site and Discord.

“Civitai exists to democratize AI media creation, making it a shared, inclusive, and empowering journey.” CivitAI’s site says.

Justin Maier, CivitAI’s founder, did not respond to a request for comment via LinkedIn, Twitter, Discord, and email.

The Singularity

Since ChatGPT, DALL-E, and other generative AI tools became available on the internet, computer scientists, ethicists, and politicians have been increasingly discussing “the singularity,” a concept that until recently existed mostly in the realm of science fiction. It describes a hypothetical point in the future when AI becomes so advanced, it triggers an uncontrollable explosion of technological development that quickly surpasses and supersedes humanity.

As many experts and journalists have observed, there is no evidence that companies like OpenAI, Facebook, and Google have created anything even close to resembling an artificial general intelligence agent that could bring about this technological apocalypse, and promoting that alarmist speculation serves their financial interests because it makes their AI tools seem more powerful and valuable than they actually are.

However, it’s a good way to describe the massive changes that have already taken hold in the generative AI porn scene. An AI porn singularity has already occurred, an explosion of non-consensual sexual imagery that’s seeping out of every crack of internet infrastructure if you only care to look, and we’re all caught up in it. Celebrities big and small and normal people. Images of our faces and bodies are fueling a new type of pornography in which humans are only a memory that’s copied and remixed to instantly generate whatever sexual image a user can describe with words.

Samantha Cole contributed reporting.

The A.I. Surveillance Tool DHS Uses to Detect ‘Sentiment and Emotion’

Joseph Cox Joseph Cox Aug 24, 2023

Internal DHS and corporate documents detail the agency’s relationship with Fivecast, a company that promises to scan for “risk terms and phrases” online.

Customs and Border Protection (CBP), part of the Department of Homeland Security, has bought millions of dollars worth of software from a company that uses artificial intelligence to detect “sentiment and emotion” in online posts, according to a cache of documents obtained by 404 Media.

CBP told 404 Media it is using technology to analyze open source information related to inbound and outbound travelers who the agency believes may threaten public safety, national security, or lawful trade and travel. In this case, the specific company called Fivecast also offers “AI-enabled” object recognition in images and video, and detection of “risk terms and phrases” across multiple languages, according to one of the documents.

Marketing materials promote the software’s ability to provide targeted data collection from big social platforms like Facebook and Reddit, but also specifically names smaller communities like 4chan, 8kun, and Gab. To demonstrate its functionality, Fivecast promotional materials explain how the software was able to track social media posts and related Persons-of-Interest starting with just “basic bio details” from a New York Times Magazine article about members of the far-right paramilitary Boogaloo movement. 404 Media also obtained leaked audio of a Fivecast employee explaining how the tool could be used against trafficking networks or propaganda operations.

The news signals CBP’s continued use of artificial intelligence in its monitoring of travelers and targets, which can include U.S. citizens. In May, I revealed CBP’s use of another AI tool to screen travelers which could link peoples’ social media posts to their Social Security number and location data. This latest news shows that CBP has deployed multiple AI-powered systems, and provides insight into what exactly these tools claim to be capable of while raising questions about their accuracy and utility.

“CBP should not be secretly buying and deploying tools that rely on junk science to scrutinize people's social media posts, claim to analyze their emotions, and identify purported 'risks,'” Patrick Toomey, deputy director of the ACLU's National Security Project, told 404 Media in an email.

404 Media obtained the documents through Freedom of Information Act requests with CBP and other U.S. law enforcement agencies.

One document obtained by 404 Media marked “commercial in confidence” is an overview of Fivecast’s “ONYX” product. In it Fivecast says its product can be used to target individuals or groups, single posts, or events. As well as collecting from social media platforms big and small, Fivecast users can also upload their own “bulk” data, the document says. Fivecast says its tool has been built “in consultation” with Five Eyes law enforcement and intelligence agencies, those being agencies from the U.S., United Kingdom, Canada, Australia, and New Zealand. Specifically on building “person-of-interest” networks, the tool “is optimized for this exact requirement.”

Related to the emotion and sentiment detection, charts contained in the Fivecast document include emotions such as “anger,” “disgust,” “fear,” “joy,” “sadness,” and “surprise” over time. One chart shows peaks of anger and disgust throughout an early 2020 timeframe of a target, for example.

The document also includes a case study of how ONYX could be used against a specific network. In the example, Fivecast examined the Boogaloo movement, but Fivecast stresses that “our intent here is not to focus on a specific issue but to demonstrate how quickly Fivecast ONYX can discover, collect and analyze Risks from a single online starting point.”

That process starts with the user inputting Boogaloo phrases such as “civil war 2.” The user then selects a discovered social media account and deployed what Fivecast calls its “‘Full’ collection capability,” which “collects all available content on a social media platform for a given account.” From there, the tool also maps out the target’s network of connections, according to the document.

Lawson Ferguson, a tradecraft advisor at Fivecast, previously showed an audience at a summit how the tool could be used against trafficking networks or propaganda operations. “These are just examples of the kind of data that one can gather with an OSINT tool like ours,” he said. Jack Poulson, from transparency organization Tech Inquiry, shared audio of the talk with 404 Media.

Ferguson said users “can train the system to recognize certain concepts and types of images.” In one example, Ferguson said a coworker spent “a huge amount of time” training Fivecast's system to recognize the concept of the drug oxycontin. This included analyzing “pictures of pills; pictures of pills in hands.”

Fivecast did not respond to a request for comment.

CBP’s contracts for Fivecast software have stretched into the millions of dollars, according to public procurement records and internal CBP documents obtained by 404 Media. CBP spent nearly $350,000 in August 2019; more than $650,000 in September 2020; $260,000 in August 2021; close to $950,000 in September 2021; and finally almost $1.17 million in September 2022.

CBP told 404 Media in a statement that “The Department of Homeland Security is committed to protecting individuals’ privacy, civil rights, and civil liberties. DHS uses various forms of technology to execute its mission, including tools to support investigations related to threats to infrastructure, illegal trafficking on the dark web, cross-border transnational crime, and terrorism. DHS leverages this technology in ways that are consistent with its authorities and the law.”

In the context of why CBP needs to buy Fivecast’s software, the internal CBP documents point to several specific parts of the agency. They are the Office of Field Operations (OFO), the main bulk of CBP which enforces border security; the National Targeting Center (NTC) based out of Virginia which aims to catch travelers and cargo that the agency believes threaten the country’s security; the Counter Network Division (CND) which is part of the NTC; and finally the Publicly Available Information Group (PAIG), which focuses on data such as location information according to other documents I’ve obtained previously.

Yahoo News reported in 2021 that the CND has gathered information on a range of journalists. The Office of the Inspector General made a criminal referral for an official who worked with CND for their role in the monitoring, but they were not charged. A supervisor of that division previously told investigators that at CND “We are pushing the limits and so there is no norm, there is no guidelines, we are the ones making the guidelines.”

“The public knows far too little about CBP's Counter Network Division, but what we do know paints a disturbing picture of an agency with few rules and access to an ocean of sensitive personal data about Americans,” Toomey from ACLU added. “The potential for abuse is immense.”

Un morceau des Pink Floyd reconstitué par une IA à partir d’ondes cérébrales

L’étude d’enregistrements de l’activité cérébrale a permis de révéler quelles régions du cerveau étaient impliquées dans le traitement de la musique. Mais surtout, l’exploitation de ces données par une intelligence artificielle a permis de reconstruire une célèbre chanson du groupe britannique Pink Floyd.

Publié hier à 16h31 Lecture 2 min.

“Des scientifiques ont reconstitué Another Brick in The Wall, des Pink Floyd, en scrutant les ondes cérébrales d’auditeurs de la chanson : c’est la première fois qu’un titre est décodé de manière reconnaissable à partir des enregistrements de l’activité cérébrale”, rapporte The Guardian.

Le quotidien britannique se fait l’écho d’une étude publiée le 15 août dans Plos Biology, pour laquelle des chercheurs ont étudié les signaux cérébraux de 29 personnes captés par des électrodes implantées à la surface de leur cortex, dans le cadre d’un traitement contre l’épilepsie. Ces enregistrements ont été réalisés alors qu’on faisait écouter la chanson des Pink Floyd aux patients.

La comparaison des signaux émis par les cerveaux avec les ondes audio correspondant au titre original a permis aux chercheurs d’identifier quelles électrodes étaient fortement liées à la mélodie, à sa hauteur, à l’harmonie et au rythme de la chanson. Puis ils ont entraîné un programme d’intelligence artificielle (IA) à repérer les liens entre l’activité cérébrale et les composants musicaux, en excluant un segment de quinze secondes de la chanson.

Cette IA ainsi formée a généré le bout manquant uniquement à partir de l’activité cérébrale des participants. “Le spectrogramme – une visualisation des ondes audio – du bout généré par l’IA était similaire à 43 % au vrai segment de la chanson”, indique New Scientist.

“Tour de force technique”

Interrogé par Science, Robert Zatorre, neuroscientifique à l’université McGill, au Canada, qui n’a pas participé à l’étude, estime que “cette reconstitution est un ‘tour de force technique’ qui donne un nouvel aperçu sur la façon dont le cerveau perçoit la musique”.

En outre, précise la revue scientifique, la méthode développée par l’équipe “a permis d’identifier une nouvelle région cérébrale qui participe à la perception du rythme musical, comme la guitare vrombissante d’Another Brick in The Wall (Part 1)”. Elle ajoute :

“Ces travaux confirment aussi que la perception de la musique, contrairement au traitement ordinaire du langage, mobilise les deux hémisphères du cerveau.”

Ludovic Bellier, neuroscientifique et chercheur en informatique à l’université de Californie Berkeley, premier auteur de l’étude “espère que ces recherches pourront un jour être utiles aux patients qui ont des difficultés d’élocution à la suite d’un AVC, de blessures ou de maladies dégénératives telles que la sclérose latérale amyotrophique”, indique Science.

Reste que, pointe dans New Scientist Robert Knight, de l’université de Californie, qui a piloté les travaux, pour le moment “la nature invasive des implants cérébraux rend peu probable l’utilisation de cette procédure pour des applications non cliniques”. Les progrès techniques dans le domaine de l’étude du cerveau laissent cependant penser que ce genre d’enregistrement pourra un jour se faire sans recourir à la chirurgie, peut-être en utilisant des électrodes fixées au cuir chevelu.

Par ailleurs, une autre équipe a récemment utilisé l’IA pour générer des extraits de chansons à partir de signaux cérébraux enregistrés à l’aide d’imagerie par résonance magnétique (IRM). Interrogée par New Scientist ; la juriste Jennifer Maisel, du cabinet Rothwell Figg, à Washington, estime qu’“à mesure que progresse la technologie, la recréation de chansons grâce à des IA et à partir de l’activité cérébrale pourrait soulever des questions de droit d’auteur, selon le degré de similarité entre la reconstitution et le titre original”.

Et sans passer par la reproduction de tubes existants, certains imaginent déjà que l’IA pourra être utilisée pour composer de la musique que les gens imaginent à partir de l’exploitation de leurs signaux cérébraux. Mais ce n’est pas encore pour demain.

La vidéo en anglais ci-dessous permet d’écouter les deux versions – l’originale et la reconstruite – d’Another Brick in The Wall (Part 1).