Al Gore’s New Movie Exposes The Big Flaw In Online Movie Ratings

“An Inconvenient Sequel: Truth to Power,” Al Gore’s new documentary, is one of the most interesting films of the year.

We don’t mean its content. The sequel to the Academy Award–winning climate change documentary “An Inconvenient Truth” is a solid movie. But it’s really the film’s online reception that’s intriguing.

“An Inconvenient Sequel” is among the most controversial and polarizing titles of the year. Because of the politics surrounding Gore and climate change, the film divides men and women, critics and fans, and even people who saw the movie and people who are just rating it. But the movie’s aggregate rating hides many of those divisions, giving us a perfect case study for understanding a big weakness of online rating systems: separating the controversial from the mediocre. That weakness could discourage ambitious-but-controversial work.

To start, look at how weird this chart of the film’s ratings distribution is:

In the pantheon of sequels to well-regarded films, “An Inconvenient Sequel” may not be “The Godfather: Part II,” but it’s not exactly “The Godfather: Part III,” either. I think it’s solid, but that seems beside the point. Gore is a capable documentarian, but he’s also a guy for whom 51 million Americans voted and 50.4 million other Americans voted against¹ that one time. Climate change is equally divisive. And those politics are coloring the film’s internet reception.

Of the 2,645 IMDb users who rated the film as of August,² over 38 percent gave the film a 1 out of 10. Of those same 2,645 IMDb users, just under 34 percent gave the film a 10 out of 10. In short: 72 percent of people who rated the movie gave it an extreme score, a 1 or a 10.

This is a massive abnormality for an IMDb rating distribution. As of mid-August, 127 documentaries and feature-length films had been released in 2017 that also earned a Metacritic score and at least 1,000 ratings on IMDb. Here’s a sortable table of all of them.

Simply looking at which films have the most 1s and 10s combined is an easy way to see how divided IMDb users were on the documentary: Only the universally reviled “The Emoji Movie,” and “The Ottoman Lieutenant,” which stirred up a controversy regarding its handling of the Armenian genocide, have more combined extreme scores. But looking at a specialized metric of polarization³ that looks at how evenly split those extreme votes were makes “An Inconvenient Sequel” by far the most heavily and evenly split film of 2017.

The reception to “An Inconvenient Sequel” lays on three major divides: (1) critics vs. audiences, (2) people who saw the movie vs. those who did not, and (3) men vs. women. It’s frankly impressive for a single film to stand astride so many fault lines. But if you’re a casual moviegoer quickly checking IMDb to see if “An Inconvenient Sequel” is worth checking out, you wouldn’t know any of that. You’d think it was just a dumpy movie — the “Breakin’ 2: Electric Boogaloo” of climate documentaries.

Fault Line No. 1:
Critics vs. audiences

The all-too-common critic-versus-fan dance is typically a load of crap. Step No. 1: Some preexisting work of fiction is adapted to the big screen. Step No. 2: Critics don’t like it. Step No. 3: Fans of the original work get mad at the critics. Usually, the lead in the film will say something like “We made this for the fans” to downplay the terrible reviews, and the film will make $20 million and place third at the box office behind the new Pirates of the Caribbean movie and a Chris Pratt flick. Critics gotta critique, fans gotta buy, stars gotta sell.

Roughly three out of four movies have an IMDb audience rating higher than their Metacritic score, so you’re going to see such tension. But we can’t say for certain how much of that is due to differences in how the two groups judge movies, as opposed to the quirks in the rating systems.

Still, let’s take it for granted that, overall, critics are less forgiving than the average moviegoer. “An Inconvenient Sequel” lies way on the other side of that spectrum.

Its Metacritic score is 68 out of 100. Its IMDb score was 5.2 out of 10 (or 52 on Metacritic’s scale). That’s the second-highest markdown⁴ by IMDb raters in the entire 2017 data set. And Metacritic is one of the harsher critical aggregations; “An Inconvenient Sequel” has a great 78 percent score on Rotten Tomatoes.⁵

So while I’m normally skeptical of the critics-versus-fans narrative, it really looks like it exists for “An Inconvenient Sequel” — at least on IMDb — just reversed. The critics thought it was OK-to-fine. The “fans” rated it subpar.

Fault Line No. 2:
People who saw it vs. people who didn’t

But here’s why I put fans in scare quotes.

We initially began monitoring the IMDb ratings of “An Inconvenient Sequel” on July 17, when 678 users had rated the film, including about 45 percent 1s, 38 percent 10s and 17 percent 2-9s. That hundreds of people were able to draw such strong conclusions about the film 11 days before it was released on July 28 is … odd.⁶ By the time the movie was in more than four cinemas on Aug. 4, more than half of the people who would eventually come to rate it had already done so.

Ticket sales went from 0 per week to 17,000 per week to 150,000 per week while the IMDb score barely budged. Put another way: Despite ticket sales increasing by three orders of magnitude per week, the number of reviewers stayed flat — only about 500 people voted on the film each period.

It’s hard to believe that each and every one of those pre-release reviews is bona fide, especially on a politicized film like this.

Personally, we would not be surprised to find out that a large percentage of these votes aren’t real. IMDb applies some filters to prevent ballot stuffing but refuses as policy to discuss those measures. Nonetheless, how did so many people see this film before its wide release? We made efforts to determine whether automated voting bots were used, but the way ratings update on IMDb makes it hard to look for obvious telltale patterns of bot voting. All told, we think this film’s rating should be viewed with utmost skepticism.

Most likely, the film was doomed before it even premiered by people (or bots) who had not seen it.

Fault Line No. 3:
Men vs. women

First, it’s mostly men who have rated “An Inconvenient Sequel”:

Second, men hate the film way more than women do:

So the movie’s IMDb rating is mostly the work of men who feel substantially different about the film than women do. Which raises the question: Why don’t more review sites do what pollsters do, and adjust their sample to reflect the reality of the world? There are arguments for making the raw data available, yes, but leaving this unadjusted and letting men get 86 percent of the say when it comes to what’s well regarded on IMDb (or any other online rating site) seems specious given that women make up a majority of moviegoers in the United States.

In this case, it seems like the gender gap in opinions on climate change — men are far more skeptical that it’s a problem than women — is affecting views of “An Inconvenient Sequel.”

A Failing Grade

The result of boiling down a piece of culture to one single number is that the rating system inevitably simplifies our understanding of the work. Sometimes that’s useful. But let that happen long enough and you’ll set up an incentive for filmmakers to hedge away from the compelling and the controversial.

Six 2017 films in total had an IMDb rating of 5.2 on Aug. 16, the day we happened to pull all of this data. In addition to “An Inconvenient Sequel,” there was “Voice from the Stone,” “Once Upon a Time in Venice,” “xXx: Return of Xander Cage,” “Phoenix Forgotten” and “Vengeance: A Love Story.”

See if you can spot which one is a special case:

Say what you will, but in addition to being controversial, “An Inconvenient Sequel” was ambitious: Few films involve Arctic expeditions, inside access to the Paris Climate Conference, interviews with the sitting secretary of state and a globe-trotting look at catastrophic weather conditions. If ambitious-yet-controversial films are boiled down to a single number that makes them look identical to mediocre films, what incentive does Hollywood have to continue investing in movies that challenge the audience?

The democratization of film reviews has been one of the most substantial structural changes in the movie business in some time, but there are dangerous side effects. The people who make movies are terrified. IMDb scores represent a few thousand mostly male reviewers who might have seen the film but maybe didn’t, and they’re influencing the scoring system of one of the most popular entertainment sites on the planet.

The thing about a democracy is that it really does fall to the user, the voter, and the vote counter to ensure that the will of the masses is correctly manifested.