The Open Secret of Google Search

This article was featured in One Story to Read Today, a newsletter in which our editors recommend a single must-read from The Atlantic, Monday through Friday. Sign up for it here.    

A few weeks ago my house had a septic-tank emergency, which is as awful as it sounds. As unspeakable things began to burble up from my shower drain, I did what any smartphone-dependent person would: I frantically Googled something along the lines of poop coming from shower drain bad what to do. I was met with a slew of cookie-cutter websites, most of which appeared hastily generated and were choked with enough repetitive buzzwords as to be barely readable. Virtually everything I found was unhelpful, so we did the old-fashioned thing and called a professional. The emergency came and went, but I kept thinking about those middling search results—how they typified a zombified internet wasteland.

Like many, I use Google to answer most of the mundane questions that pop up in my day-to-day life. And yet that first page of search results feels like it’s been surfacing fewer satisfying answers lately. I’m not alone; the frustration has become a persistent meme: that Google Search, what many consider an indispensable tool of modern life, is dead or dying. For the past few years, across various forums and social-media platforms, people have been claiming in viral posts that Google’s flagship product is broken. Search google dying on Twitter or Reddit and you can see people grousing about it going back to the mid 2010s. Lately, though, the criticisms have grown louder.

In February, an engineer named Dmitri Brereton wrote a blog post about Google’s search-engine decay, rounding up leading theories for why the product’s “results have gone to shit.” The post quickly shot to the top of tech forums such as Hacker News and was widely shared on Twitter and even prompted a PR response from Google’s Search liaison, Danny Sullivan, refuting one of Brereton’s claims. “You said in the post that quotes don’t give exact matches. They really do. Honest,” Sullivan wrote in a series of tweets.

Read: Be careful what you Google

Brereton’s most intriguing argument for the demise of Google Search was that savvy users of the platform no longer type instinctive keywords into the search bar and hit “Enter.” The best Googlers—the ones looking for actionable or niche information, product reviews, and interesting discussions—know a cheat code to bypass the sea of corporate search results clogging the top third of the screen. “Most of the web has become too inauthentic to trust,” Brereton argued, therefore “we resort to using Google, and appending the word ‘reddit’ to the end of our queries.” Brereton cited Google Trends data that show that people are searching the word reddit on Google more than ever before.

Instead of scrolling through long posts littered with pop-up ads and paragraphs of barely coherent SEO chum to get to a review or a recipe, clever searchers got lively threads with testimonials from real people debating and interacting with one another. Most who use the Reddit hack are doing so for practical reasons, but it’s also a small act of protest—a way to stick it to the Search Engine Optimization and Online Ad Industrial Complex and to attempt to access a part of the internet that feels freer and more human.

Google has built wildly successful mobile operating systems, mapped the world, changed how we email and store photos, and tried, with varying success, to build cars that drive themselves. This story, for example, was researched, in part, through countless Google Search queries and some Google Chrome browsing, written in a Google Doc, and filed to my editor via Gmail. Along the way, the company has collected an unfathomable amount of data on billions of people (frequently unbeknownst to them)—but Google’s parent company, Alphabet, is still primarily an advertising business. In 2020, the company made $147 billion in revenue off ads alone, which is roughly 80 percent of its total revenue. Most of the tech company’s products—Maps, Gmail—are Trojan horses for a gargantuan personalized-advertising business, and Search is the one that started it all. It is the modern template for what the technology critic Shoshana Zuboff termed “surveillance capitalism.”

The internet has grown exponentially and Google has expanded with it, helping usher in some of the web’s greediest, most extractive tendencies. But scale is not always a blessing for technology products. Are we wringing our hands over nothing, or is Google a victim of its own success, rendering its flagship product—Search—less useful?

One can’t really overstate the way that Google Search, when it rolled out in 1997, changed how people used the internet. Before Google came out with its goal to crawl the entire web and organize the world’s information, search engines were moderately useful at best. And yet, in the early days, there was much more search competition than there is now; Yahoo, Altavista, and Lycos were popular online destinations. But Google’s “PageRank” ranking algorithm helped crack the problem. The algorithm counted and indexed the number and quality of links that pointed to a given website. Rather than use a simple keyword match, PageRank figured that the best results would be websites that were linked to by many other high-quality websites. The algorithm worked, and the Google of the late 1990s seemed almost magical: You typed in what you were looking for, and what you got back felt not just relevant but intuitive. The machine understood.

Most people don’t need a history lesson to know that Google has changed; they feel it. Try searching for a product on your smartphone and you’ll see that what was once a small teal bar featuring one “sponsored link” is now a hard-to-decipher, multi-scroll slog, filled with paid-product carousels; multiple paid-link ads; the dreaded, algorithmically generated “People also ask” box; another paid carousel; a sponsored “buying guide”; and a Maps widget showing stores selling products near your location. Once you’ve scrolled through that, multiple screen lengths below, you’ll find the unpaid search results. Like much of the internet in 2022, it feels monetized to death, soulless, and exhausting.

There are all kinds of theories for those ever-intrusive ads. One is that the cost-per-click rates that Google charges advertisers are down, because of competition from Facebook and Amazon (Google is rolling out larger commerce-search ad widgets in response this year) as well as a slowdown in paid-search-result spending. Another issue may stem from cookie-tracking changes that Google is implementing in response to privacy laws such as Europe’s General Data Protection Regulation and the California Consumer Privacy Act. For the past two years, Google has been planning to remove third-party cookies from its Chrome browser. And though Google Search won’t be affected by the cookie ban, the glut of search ads might be an attempt to recoup some of the money that Google stands to lose in the changes to Chrome. If so, this is an example of fixing one problem while creating another. But when I suggested this to Google, the company was unequivocal, arguing that “there is no connection” between Chrome’s plans to phase out support for third-party cookies and Search ads. The company also said that the number of ads it shows in search results “has been capped for several years, and we have not made any changes.” Google claims that, “on average over the past four years, 80 percent of searches on Google haven’t had any ads at the top of search results.”

Any hunt for answers about Google’s Search algorithms will lead you into the world of SEO experts like Marie Haynes. Haynes is a consultant who has been studying Google’s algorithms obsessively since 2008. Part of her job is to keep up with every small change made by the company’s engineers and public communication by Google’s Search-team blog. Companies that can divine the whims of Google’s constantly updated algorithms are rewarded with coveted page real estate. Ranking high means more attention, which theoretically means more money. When Google announced in October 2020 that it would begin rolling out “passage indexing”—a new way for the company to pull out and rank discrete passages from websites—Haynes tried to figure out how it would change what people ultimately see when they query. Rather than reverse engineer posts to sound like bot-written babble, she and her team attempt to balance maintaining a page’s integrity while also appealing to the algorithm. And though Google provides SEO insiders with frequent updates, the company’s Search algorithms are a black box (a trade secret that it doesn’t want to give to competitors or to spammers who will use it to manipulate the product), which means that knowing what kind of information Google will privilege takes a lot of educated guesswork and trial and error.

Charlie Warzel: The internet Google left behind

Haynes agrees that ads’ presence on Search is worse than ever and the company’s decision to prioritize its own products and features over organic results is frustrating. But she argues that Google’s flagship product has actually gotten better and much more complex over time. That complexity, she suggests, might be why searching feels different right now. “We’re in this transition phase,” she told me, noting that the company has made significant advancements in artificial intelligence and machine learning to decipher user queries. Those technical changes have caused it to move away from the PageRank paradigm. But those efforts, she suggested, are in their infancy and perhaps still working out their kinks. In May 2021, Google announced MUM (short for Multitask Unified Model), a natural-language-processing technology for Search that is 1,000 times more powerful than its predecessor.

“The AI attempts to understand not just what the searcher is typing, but what the searcher is trying to get at,” Haynes told me. “It’s trying to understand the content inside pages and inside queries, and that will change the type of result people get.” Google’s focus on searcher intent could mean that when people type in keywords, they’re not getting as many direct word matches. Instead, Google is trying to scan the query, make meaning from it, and surface pages that it thinks match that meaning. Despite being a bit sci-fi and creepy, the shift might feel like a loss of agency for searchers. Search used to feel like a tool that you controlled, but Google may start to behave more like, well, a person—a concierge that has its own ideas and processes. The problematic effects of increased AI inference over time are easy to imagine (while I was writing this article, a Google researcher went viral claiming he’d been placed on administrative leave after notifying the company that one of its AI chatbots—powered by different technology—had become sentient, though the company disagrees). Google could use such technology to continue to lead people away from their intended searches and toward its own products and paid ads with greater frequency. Or, less deviously, it could simply gently algorithmically nudge people in unexpected directions. Imagine all the life decisions that you make in a given year based on information you process after Googling. This means that the stakes of Google’s AI interpreting a searcher’s intent are high.

Read: Google’s ‘sentient’ chatbot is our self-deceiving future

But some of Google’s lifeless results are made by humans. Zach Verbit knows what it’s like to serve at the pleasure of Google’s Search algorithms. After college, Verbit took a freelance-writing gig with the HOTH, a marketing company that specializes in search-engine optimization. Verbit’s “soul crushing” job at the HOTH was to write blog posts that would help clients’ sites rank highly. He spent hours composing listicles with titles like “10 Things to Do When Your Air-Conditioning Stopped Working.” Verbit wrote posts that “sounded robotic or like they were written by somebody who’d just discovered language.” He had to write up to 10 posts a day on subjects he knew nothing about. Quickly, he started repurposing old posts for other clients’ blogs. “Those posts that sound like an AI wrote them? Sometimes they’re from real people trying to jam in as many keywords as possible,” Verbit told me.

That his hastily researched posts appeared high in search results left him dispirited. He quit the job after a year, describing the industry of search-gaming as a house of cards. His time in the SEO mines signaled to him the decline of Google Search, arguably the simplest, most effective, and most revolutionary product of the modern internet. “The more I did the job, the more I realized that Google Search is completely useless now,” he said. HOTH’s CEO, Marc Hardgrove disputed the notion that its client blog posts were “over-optimized” for SEO purposes and that the company discourages jargony posts as they don’t rank as high. “Overusing keywords and creating un-compelling content would be detrimental to our success as an SEO company, he wrote in an email. “That’s why The HOTH does not require, or even encourage, the writers we work with to overuse keywords into their blog posts to help with optimization.”

Google is still useful for many, but the harder question is why its results feel more sterile than they did five years ago. Haynes’s theory is that this is the result of Google trying to crack down on misinformation and low-quality content—especially around consequential search topics. In 2017, the company started talking publicly about a Search initiative called EAT, which stands for “expertise, authoritativeness, and trustworthiness.” The company has rolled out numerous quality rater guidelines, which help judge content to determine authenticity. One such effort, titled Your Money or Your Life, applies rigorous standards to any pages that show up when users search for medical or financial information.

“Take crypto,” Haynes explained. “It’s an area with a lot of fraud, so unless a site has a big presence around the web and Google gets the sense they’re known for expertise on that topic, it’ll be difficult to get them to rank.” What this means, though, is that Google’s results on any topic deemed sensitive enough will likely be from established sources. Medical queries are far more likely to return WebMD or Mayo Clinic pages, instead of personal testimonials. This, Haynes said, is especially challenging for people looking for homeopathic or alternative-medicine remedies.

There’s a strange irony to all of this. For years, researchers, technologists, politicians, and journalists have agonized and cautioned against the wildness of the internet and its penchant for amplifying conspiracy theories, divisive subject matter, and flat-out false information. Many people, myself included, have argued for platforms to surface quality, authoritative information above all else, even at the expense of profit. And it’s possible that Google has, in some sense, listened (albeit after far too much inaction) and, maybe, partly succeeded in showing higher-quality results in a number of contentious categories. But instead of ushering in an era of perfect information, the changes might be behind the complainers’ sense that Google Search has stopped delivering interesting results. In theory, we crave authoritative information, but authoritative information can be dry and boring. It reads more like a government form or a textbook than a novel. The internet that many people know and love is the opposite—it is messy, chaotic, unpredictable. It is exhausting, unending, and always a little bit dangerous. It is profoundly human.

But it’s worth remembering what that humanity looked like inside search results. Rand Fishkin, the founder of the software company SparkToro, who has been writing and thinking about search since 2004, believes that Google has gotten better at not amplifying conspiracy theories and hate speech, but that it took the company far too long. “I don’t know if you searched for holocaust information between 2000 and 2008, but deniers routinely showed up in the top results,” he told me. The same was true for Sandy Hook hoaxers—in fact, campaigns from the Sandy Hook families to fight the conspiracy theories led to some of the search engine’s changes. “Whenever somebody says, ‘Hey, Google doesn’t feel as human anymore,’ all I can say is that I bet they don’t want a return to that,” Fishkin said.

Google Search might be worse now because, like much of the internet, it has matured and has been ruthlessly commercialized. In an attempt to avoid regulation and be corporate-friendly, parts of it might be less wild. But some of what feels dead or dying about Google might be our own nostalgia for a smaller, less mature internet. Sullivan, the Search liaison, understands this longing for the past, but told me that what feels like a Google change is also the search engine responding to the evolution of the web. “Some of that blog-style content has migrated over time to closed forums or social media. Sometimes the blog post we’re hoping to find isn’t there.” Sullivan believes that some of the recent frustrations with Google Search actually reflect just how good it’s become. “We search for things today we didn’t imagine we could search for 15 years ago and we believe we’ll find exactly what we want,” he said. “Our expectations have continued to grow. So we demand more of the tool.” It’s an interesting, albeit convenient, response.

From the July/August 2008 issue: Is Google making us stupid?

Google has rewired us, transforming the way that we evaluate, process, access, and even conceive of information. “I can’t live without that stuff as my brain is now conditioned to remember only snippets for Google to fill in,” one Reddit user wrote while discussing Brereton’s “Google Is Dying” post. Similarly, Google users shape Search. “The younger generation searches really differently than I do,” Haynes told me. “They basically speak to Google like it’s a person, whereas I do keyword searching, which is old-school.” But these quirks, tics, and varying behaviors are just data for the search giant. When younger generations intuitively start talking to Google like it’s a person, the tool starts to anticipate that and begins to behave like one (this is part of the reason behind the rise of humanized AI voice assistants).

Fishkin argues that Google Search—and many of Google’s other products—would be better with some competition and that Search’s quality improved the most from 1998 to 2007, which he attributes to the company’s need to compete for market share. “Since then,” he said, “Google’s biggest search innovation has been to put more Google products up front in results.” He argues that this strategy has actually led to a slew of underwhelming Google products. “Are Google Flights or Google Weather or Google’s stocks widget better than competitors? No, but nobody can really compete, thanks to the Search monopoly.”

“Is Google Search dying?” is a frivolous question. We care about Search’s fate on a practical level—it is still a primary way to tap into the internet’s promise of unlimited information on demand. But I think we also care on an existential level—because Google’s first product is a placeholder to explore our hopes and fears about technology’s place in our life. We yearn for more convenience, more innovation, more possibility. But when we get it, often we can only see what we’ve lost in the process. That loss is real and deeply felt. It’s like losing a piece of our humanity. Search, because of its utility, is even more fraught.

Most people don’t want their information mediated by bloated, monopolistic, surveilling tech companies, but they also don’t want to go all the way back to a time before them. What we really want is something in between. The evolution of Google Search is unsettling because it seems to suggest that, on the internet we’ve built, there’s very little room for equilibrium or compromise.