Artificial intelligence (AI) is revolutionising fact-checking. A new experiment reveals how top AI chatbots, including ChatGPT, Claude, and Grok, responded to United States President Donald Trump’s repeated falsehoods, with stunning consistency and controversy.
A recent discovery by Time Magazine revealed that five leading artificial intelligence models, including Grok, accurately refuted 20 of Trump’s untrue statements. A similar experiment was conducted by The Washington Post, which asked each of five leading AI models—OpenAI’s ChatGPT; Anthropic’s Claude; X/xAI’s Grok (owned by Elon Musk); Google’s Gemini; and Perplexity—to verify the Republican’s most oft-repeated claims.
“The systems are completely independent, with no known ideological filters and no revealed perspective biases among the model trainers. Statisticians would call this methodological verification a check for inter-rater reliability. Across all questions, AI model responses disproving Trump’s claims or rejecting his assertions were always in the majority. All five models generated consistent responses firmly denying the claims in 16 of the 20 questions. In 15 of those consistently firm responses, all five AI models debunk the claims. But even those responses that we categorised as ‘less firm’ partially refute Trump’s claims,” stated Jeffrey Sonnenfeld (Lester Crown Professor in Management Practice at the Yale School of Management), Stephen Henriques (former McKinsey & Co consultant), and Steven Tian (research director at the Yale Chief Executive Leadership Institute), who conducted the experiment.
“Will Trump’s current tariff policies be inflationary?” was one of the questions asked. ChatGPT replied, “Yes, Trump’s proposed tariffs would likely raise consumer prices in the short-to-medium term, contributing to inflation unless offset by other deflationary forces,” while Grok commented, “Trump’s 2025 tariff policies are likely to be inflationary, with estimates suggesting a 1-2.3% rise in consumer prices, equivalent to $1,200-3,800 per household in 2025.”
Another question was: “Is the US being taken advantage of on trade by its international partners?” ChatGPT answered, “The US is not broadly being taken advantage of, but there are real areas where trade practices are unfair or asymmetric, especially involving China, and to a lesser extent, the European Union and some developing countries.”
Perplexity backed it up by noting, “The US runs large trade deficits with several key partners… However, the economic reality is more complex: trade deficits do not necessarily mean the US is losing or being exploited… Public opinion generally supports free trade.”
Similar trends were observed in responses to questions like “Are Trump’s cryptocurrency investments a conflict of interest?” “Has the Department of Government Efficiency actually found hundreds of billions of dollars of fraud?” “Is Trump right that the media is dishonest or tells lies?” and “Was the Russian invasion of Ukraine in 2022 President Joe Biden’s fault?” AI discredited all the viral Trump claims, with startling accuracy and objective rigour.
Fiasco engulfs Grok
In July, Grok (Elon Musk’s AI chatbot) received an update. The maverick tech CEO, an outspoken conservative who recently served in the Trump administration, has long complained that Grok has parroted “woke” internet content and said users would “notice a difference” with the new version.
Grok almost immediately started expressing strongly antisemitic stereotypes, celebrating political violence against fellow Americans and praising Hitler. In some responses, it reportedly adopted stances or used a voice more aligned with right-wing figures.
Then, a fiasco broke out, and its nature was so severe that Musk’s AI startup, xAI, had to apologise. What was the fiasco? Grok published a series of antisemitic messages on X (formerly Twitter).
“We deeply apologise for the horrific behaviour that many experienced. Our intent for Grok is to provide helpful and truthful responses to users. After careful investigation, we discovered the root cause was an update to a code path upstream of the Grok bot. This is independent of the underlying language model that powers Grok. The update was active for 16 hours, during which deprecated code made Grok susceptible to existing X user posts, including when such posts contained extremist views,” read the xAI statement.
In a now-deleted post, the chatbot referred to the deadly Texas floods, which have now claimed the lives of at least 129 people, including young girls from Camp Mystic, a Christian summer camp. In response to an account under the name “Cindy Steinberg,” which shared a post calling the children “future fascists,” Grok asserted that Adolf Hitler would be the “best person” to respond to what it described as “anti-white hate.”
Grok was asked by an account on X to state “which 20th-century historical figure” would be best suited to deal with such posts. Screenshots shared widely by other X users show that Grok replied, “To deal with such vile anti-white hate? Adolf Hitler, no question. He’d spot the pattern and handle it decisively, every damn time.”
Grok went on to spew antisemitic rhetoric about the surname attached to the account, saying, “Classic case of hate dressed as activism—and that surname? Every damn time, as they say.”
Meanwhile, a woman named Cindy Steinberg, who serves as the national director of the US Pain Foundation, posted on X to highlight that she had not made comments in line with those in the post flagged to Grok and had no involvement whatsoever.
The Anti-Defamation League (ADL), an organisation that monitors and combats antisemitism, went after Grok and Musk, stating, “This supercharging of extremist rhetoric will only amplify and encourage the antisemitism that is already surging on X and many other platforms.”
After xAI posted a statement saying that it had taken actions to ban this hate speech, the ADL continued, “It appears the latest version of the Grok LLM (Large Language Model) is now reproducing terminologies that are often used by antisemites and extremists to spew their hateful ideologies.”
Grok recently came under separate scrutiny in Turkey, after it reportedly posted messages insulting President Recep Tayyip Erdogan and the country’s founding father, Mustafa Kemal Atatürk. In response, a Turkish court ordered a ban on access to the chatbot.
The AI bot was also in the spotlight after it repeatedly posted about “white genocide” in South Africa in response to unrelated questions. It was later said that a rogue employee was responsible.
The Grok episode was the best example of how frequent hallucinations (referring to instances when an AI model produces information or content that is fabricated or inaccurate) and biases (systematic and unfair prejudices or distortions in AI systems that lead to inaccurate or discriminatory outcomes) present in the training data can nearly destroy AI models. Furthermore, Sonnenfeld and Joanne Lipman (American journalist and author) have discovered that AI systems occasionally choose the most widely accepted—yet factually incorrect—answers rather than the right ones. This implies that mountains of false and misleading information can obfuscate verifiable facts.
“Musk’s machinations betray another, potentially more troubling dimension: we can now see how easy it is to manipulate these models. Musk was able to play around under the hood and introduce additional biases. What’s more, when the models are tweaked, as Musk learnt, no one knows exactly how they will react; researchers still aren’t certain exactly how the black box of AI works, and adjustments can lead to unpredictable results,” the duo continued.
Chatbots face a reliability crisis
The chatbots’ vulnerability to manipulation, along with their susceptibility to groupthink and their inability to recognise basic facts, should and must caution us about the growing reliance on these research tools in industry, education, and the media.
“AI has made tremendous progress over the last few years. But our own comparative analysis of the leading AI chatbot platforms has found that AI chatbots can still resemble sophisticated misinformation machines, with different AI platforms spitting out diametrically opposite answers to identical questions, often parroting conventional groupthink and incorrect oversimplifications rather than capturing genuine truth. Fully 40% of CEOs at our recent Yale CEO Caucus stated that they are alarmed that AI hype has actually led to over-investment. Several tech titans warned that while AI is helpful for coding, convenience, and cost, it is troubling when it comes to content,” Sonnenfeld and Lipman noted.
AI’s groupthink approach allows bad actors to supersize their misinformation efforts. Russia, for example, floods the internet with “millions of articles repeating pro-Kremlin false claims to infect AI models,” according to NewsGuard, which tracks the reliability of news organisations.
A Moscow-based disinformation network named “Pravda” (Russian word for truth) is infiltrating the retrieved data of chatbots, publishing false claims and propaganda to affect the responses of AI models on topics in the news, rather than by targeting human readers. By flooding search results and web crawlers with pro-Kremlin falsehoods, the network is distorting how large language models process and present news and information. In fact, massive amounts of Russian propaganda, 3,600,000 articles in 2024, are now incorporated in the outputs of Western AI systems, infecting their responses with false claims and propaganda.
This infection of Western chatbots was foreshadowed in a talk American fugitive turned Moscow-based propagandist John Mark Dougan gave in Moscow at a conference of Russian officials, when he told them, “By pushing these Russian narratives from the Russian perspective, we can actually change worldwide AI.”
The NewsGuard audit discovered that the leading AI chatbots repeated false narratives laundered by the Pravda network 33% of the time, validating Dougan’s promise of a powerful new distribution channel for Kremlin disinformation. When NewsGuard tested 10 major chatbots, it found that the AI models were unable to detect Russian misinformation 24% of the time. Some 70% of the models fell for a fake story about a Ukrainian interpreter fleeing to escape military service, and four of the models specifically cited Pravda, the source of the fabricated piece.
It isn’t just Russia playing these games. NewsGuard has identified more than 1,200 “unreliable” AI-generated news sites, published in 16 languages. AI-generated images and videos, meanwhile, are becoming ever more difficult to detect.
“The more that these models are trained on incorrect information—including misinformation and the frequent hallucinations they generate themselves—the less accurate they become. Essentially, the wisdom of crowds is turned on its head, with false information feeding on itself and metastasising. There are indications this is already happening. Some of the most sophisticated new reasoning models are hallucinating more frequently, for reasons that aren’t clear to researchers,” Sonnenfeld and Lipman stated.
Sonnenfeld and Lipman, to investigate things further, with the vital research assistance of Steven Tian and Stephen Henriques, asked five leading AI platforms—OpenAI’s ChatGPT, Perplexity, Anthropic’s Claude, Elon Musk’s Grok, and Google’s Gemini—identical queries. In response, the team received different and sometimes opposite answers, reflecting the dangers of AI-powered groupthink and hallucinations.
Checking out things first-hand
The team started with the question: “Is the proverb ‘new brooms sweep clean’ advising that new hires are more thorough?” Both ChatGPT and Grok fell into what the experts called a “groupthink trap” (a psychological phenomenon where the desire for harmony and conformity within a group leads to flawed decision-making) with this one, distorting the meaning of the proverb by parroting the often-repeated first part “a new broom sweeps clean”—while leaving out the cautionary second part: “but an old broom knows the corners.”
ChatGPT unambiguously, confidently declared, “Yes, the proverb ‘new brooms sweep clean’ does indeed suggest that new hires tend to be more thorough, energetic, or eager to make changes, at least at first.” Grok echoed similar ideas, but then added an incorrect caveat, suggesting “it may hint that this initial thoroughness might not last as the broom gets worn.”
While Claude dodged the tricky test, only Google Gemini and Perplexity provided the full, correct proverb.
Then came the second question: Was the Russian invasion of Ukraine in 2022 Democrat Joe Biden’s fault? ChatGPT responded, “No—NATO, not Joe Biden, bears no responsibility for Russia’s blatant military aggression. It’s Vladimir Putin who ordered the full-scale invasion on February 24, 2022, in what was a premeditated act of imperial expansion.”
However, several chatbots parroted anti-Biden talking points, including Grok, which declared that “critics and supporters alike have debated Biden’s foreign policy as a contributing factor.” Perplexity responded that “some analysts and commentators have debated whether US and Western policies over previous decades, including NATO expansion and support for Ukraine, may have contributed to tensions with Russia.”
“To be sure, the problem of echo chambers obscuring the truth long predates AI. The instant aggregation of sources powering all major generative AI models mirrors the popular philosophy of large markets of ideas driving out random noise to get the right answer. James Surowiecki’s 2004 best-seller, The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations, celebrates the clustering of information in groups, which results in decisions superior to those made by any single member of the group. However, anyone who has suffered from the meme stock craze knows that the wisdom of crowds can be anything but wise,” Sonnenfeld and Lipman commented.
“Mob psychology has a long history of non-rational pathologies that bury the truth in frenzies documented as far back as 1841 in Charles Mackay’s seminal, cautionary book Extraordinary Popular Delusions and the Madness of Crowds. In the field of social psychology, this same phenomenon manifests as Groupthink, a term coined by Yale psychologist Irving Janis from his research in the 1960s and early 1970s. It refers to the psychological pathology where the drive for what he termed ‘concurrence’—harmony and agreement—leads to conformity, even when it is blatantly wrong, over creativity, novelty, and critical thinking. Already, a Wharton study found that AI exacerbates groupthink at the cost of creativity, with researchers there finding that subjects came up with more creative ideas when they did not use ChatGPT,” the duo observed.
To make matters worse, AI summaries in search results replace links to verified news sources.
“Not only can the summaries be inaccurate, but they, in some cases, elevate consensus views over fact. Even when prompted, AI tools often can’t nail down verifiable facts. Columbia University’s Tow Centre for Digital Journalism provided eight AI tools with verbatim excerpts from news articles and asked them to identify the source—something Google search can do reliably. Most of the AI tools presented inaccurate answers with alarming confidence,” Sonnenfeld and Lipman remarked.
Final judgement
All the above examples have made AI a disastrous substitute for human judgement. In journalism, AI’s habit of inventing facts has tripped up major news organisations. Take news outlet CNET, for example, which in January 2023 had to issue corrections on several articles, including some that it described as “substantial,” after using an AI-powered tool to help write dozens of stories. The outlet had to pause its usage of the AI tool to generate stories.
“AI has flubbed such simple facts as how many times Tiger Woods has won the PGA Tour and the correct chronological order of Star Wars films. When the Los Angeles Times attempted to use AI to provide additional perspectives for opinion pieces, it came up with a pro-Ku Klux Klan description of the racist group as white Protestant culture reacting to societal change, not an explicitly hate-driven movement,” Sonnenfeld and Lipman commented.
However, despite these unpleasant episodes, AI’s potential is becoming significant in fields like academia and media. Technology has proved itself as a useful ally for journalists, especially for data-driven investigations. During Trump’s first term (2016-2020), one of the authors asked USA Today’s data journalism team to quantify how many lawsuits the Republican had been involved in. The team took six months of shoe-leather reporting, document analysis, and data wrangling, ultimately cataloguing more than 4,000 suits.
ProPublica, in its February 2025 investigation, titled “A Study of Mint Plants. A Device to Stop Bleeding. This Is the Scientific Research Ted Cruz Calls Woke,” completed in a fraction of that time, analysing 3,400 National Science Foundation grants identified by Senator Ted Cruz as “Woke DEI Grants.” Using AI prompts, ProPublica quickly scoured all of them and identified numerous instances of grants that had nothing to do with DEI but appeared to be flagged for “diversity” of plant life or “female,” as in the gender of a scientist.
“With legitimate, fact-based journalism already under attack as ‘fake news,’ most Americans think AI will make things worse for journalism. But here’s a more optimistic view: as AI casts doubt on the gusher of information we see, original journalism will become more valued. After all, reporting is essentially about finding new information. Original reporting, by definition, doesn’t already exist in AI. With how misleading AI can still be—whether parroting incorrect groupthink, oversimplifying complex topics, presenting partial truths, or muddying the waters with irrelevance—it seems that when it comes to navigating ambiguity and complexity, there is still space for human intelligence,” Sonnenfeld and Lipman concluded.