Key takeaways:
- Conservatives have long decried liberal bias from the so-called ‘mainstream media.’ It turns out conservative news outlets are much more biased.
- The Wall Street Journal is the least biased of the 10 news outlets analyzed by the University of Pennsylvania’s Media Bias Detector, which uses AI to parse through news articles five times a day.
- Fox News and Breitbart are far more biased in their coverage than the most left-leaning publications in the analysis: Huffington Post and The Guardian.
Conservative voters often describe traditional news outlets as “mainstream media,” a derisive moniker that means their coverage will always have a liberal leaning and therefore cannot be trusted.
But a study of the partisanship displayed by 10 major news outlets showed that conservative news outlets are actually much more biased in their reporting than their left-leaning counterparts.
According to the University of Pennsylvania’s AI-powered Media Bias Detector, Fox News and Breitbart are far more partisan than even the most liberal of the group: Huffington Post and The Guardian newspaper.
CNN, a favorite target of conservatives, shows more neutral coverage of the news than liberal. The same can be said of The New York Times, which is another favorite target of the right. The most neutral coverage was provided by The Wall Street Journal, although it was not entirely non-partisan.
The detector parsed news articles from July 14, the day before former president Donald Trump announced Republican Senator J.D. Vance as his running mate, until Aug. 10. Vice President Kamala Harris announced her running mate choice, Minnesota Governor Tim Walz, on Aug. 6.
In a blog, the university said that “left-leaning publishers may be more balanced in their coverage than their right-leaning counterparts.”
Vastly different coverage confuses voters?
Conservative news outlets were far more critical of Harris-Walz than left-leaning media in their headlines: “Kamala Harris is ‘Proud’ to Pick Radical Leftist Tim Walz as Running Mate,” opined Breitbart, while Fox News had “Sean Hannity: Kamala Harris taps the ‘most radical far-left governor’ to be her running mate.”
Left-leaning publications tend to run more positive headlines on Walz: Huffington Post published “Tim Walz Gives Off ‘Midwestern Dad’ Energy in Wholesome Throwback Video at State Fair.”
“These stark contrasts in headlines … highlight the noticeable differences in how left-leaning and right-leaning publishers choose to portray the same event,” the university said. “It is clear that the same event can be framed in vastly different ways or portrayed in a manner that aligns with the values and beliefs of the publisher.”
Using AI to detect bias
The Media Bias Detector uses a large language model (LLM), with humans-in-the-loop, to parse through the news articles of The Associated Press, Breitbart, CNN, Fox News, The Guardian, The Huffington Post, The New York Times, USA Today, The Wall Street Journal, and the Washington Post.
The detector looks at each publisher’s homepage five times a day – 6 a.m., 10 a.m., 2 p.m., 6 p.m., and 10 p.m. EST – and identifies the top 20 most prominent articles displayed to readers. It classifies an article’s prominence based on how far it is from the top of the homepage, the size of the font and size of figures. The model does not consider videos, podcasts and photo galleries.
The detector takes the article text, title, and date of publication, but removes ads and other extra words (for example, ‘click here for more information’) before passing it to the LLM. Next, the UPenn team uses GPT to generate labels for each document. The labels are generated at the article and sentence levels. For simpler tasks like determining the topic, the team used OpenAI’s GPT-3.5 Turbo. For more complex tasks, they used GPT-4o.
Article level: Each article is classified into 36 topics across eight categories: politics, culture and lifestyle, business, health, disaster, economy, sports, and science and technology. This part uses GPT-3.5 Turbo. The article is further classified as a news story, analysis or opinion, using GPT-4o.
GPT detects the tone of each article according to five buckets: very negative (-5, -4), negative (-3, -2), neutral (-1, 0, 1), positive (2, 3) and very positive (4, 5). These articles are further classified in their political leanings: Democratic (-5, -4), neutral-leaning Democratic (-3, -2), neutral (-1, 0, 1), neutral-leaning Republican (2, 3), and Republican (4, 5).
Sentence level: Sentences are categorized according to “fact,” “opinion,” “borderline,” “quote,” or “other.” The tone is tagged as either “positive,” “negative,” or “neutral.” The sentence focus is also measured, to see whether it is referring to a political party.
The UPenn team clustered articles around the same event, basing them on semantic similarity. Research assistants serve as humans-in-the-loop, constantly monitoring random samples of labels generated by GPTs to ensure high accuracy.
