Engagement data is the new algorithm for search engines

Photo credit: antbphotos / 123RF Stock Photo.

A mobile revolution has been taking place over the past 20 years as people shift from desktops to smartphones as their primary way to access the internet. In 2015, Google claimed mobile searches outnumbered desktop searches for the first time. This shift affected more than just the devices people used, but the nature of search and how people found things on the internet.

The evolution of Google’s PageRank

It was in the late 1990s when Google became known to the world for the discovery of PageRank. The important part of this new trend was the shift from relying on just keywords and their frequencies for relevance ranking to leveraging the web graph. The discovery was significant and allowed using the link structure of the web to estimate the importance of a web page.

In the early days of web browsing, users had to click from page to page following links. PageRank is an algorithm that takes a large graph of web pages (using only a graph, not considering the content) and calculates a probability score for each, representing the probability of a random visitor ending up on the destination page.

In simpler terms, web pages with more in-links, also called backlinks, from other pages are more popular. In fact, the algorithm was originally called Backrub because it leveraged backlinks as indications of page quality and sources of extra keywords (words on the backlinks are predictive of the relevance for the destination page).

In the older days, link pages or directory pages were hubs that helped people find content. A page that was not a directory or linked from a directory would be harder to find and hence have a lower score. Pages that are more remote from a directory would be scored even lower. Before the use of the web graph, there was no way to differentiate an official page versus some random page mentioning a topic. Although the use of PageRank in major search engines is debatable, they all heavily relied on the web graph link structure as a key ranking feature. Google was the first major engine to heavily leverage the web graph, getting a huge leap on relevance over its then competitors: Altavista, Hotbot, Yahoo, and others.

PageRank in the mobile era

In the same way, there was a major shift in the late 90s from keyword-only ranking to text combined with web graph-based ranking. In 2010, there was a new technological shift from the web graph to engagement-based ranking. In 2017, when people spend even more time in apps than on the mobile web, the use and expectations of search have changed.

First, there was the public web. Now, there are even more sources of search content such as social media and in-app content (some of which have no analog on the public web). The web graph-based methods relied on being able to estimate the importance of a web page based on how other pages on the public web linked to each other. Unfortunately, this approach suffers from major problems that have only gotten worse. Another factor making things worse is that more and more content are not part of the public web. Social sites like Facebook and in-app content that may not have an analog on the web and do not contain traditional web links do not permit search engines to crawl them.

There are a few major problems with graph-based methods to rank mobile content:

• The abuse of the algorithms such as link-spam, where companies traded in highly connected pages and created whole virtual graphs to trick web crawlers.
• The ability to handle real-time content. The web graph is slow to discover new content, as the link-building and crawling process takes time. A new page is unlikely to be linked to as much as an older page. The web graph is particularly bad in situations where a graph doesn’t even exist. This can happen for in-app content or content within a “private platform” like Facebook.
• The introduction of mobile complicates things even further. There is no natural analog when navigating content within an app as there is for a website.

In fact, there is no universal standard for how to connect the web or other platforms (i.e. Facebook) to in-app content. At Branch, we built our whole company around the need to provide strong deep linking across most major platforms to in-app content, but despite having a method for bridging this gap, the web graph is still losing its mojo and can’t cover all cases (some content don’t have any web presence).

In addition, the web graph is inherently global (there has been some research on splitting the graph into smaller and more localized categories or interest areas) and cannot be personalized.

The solution to these problems is increased use of engagement data. What better way to predict the importance of a particular piece of content than how that content has been engaged with by real users? The trend has already begun to take off. Even Google is demonstrating this shift in several places, such as Google Play (app-search).  Marketers and SEO specialists have taken notice too. In apps like Twitter and Facebook, global and personal engagement data are very significant ranking functions for search.

Engagement and mobile search

Engagement data is inherently personal and, when aggregated, global. It captures real-time trends with lower latency than the web graph (which requires discovering content through crawling). Engagement data, in theory, exists for all content (although it might be contained within walled gardens/private networks) and is already being used where available. This data will have higher coverage of content than a sparse web graph. An interesting piece of content is going to be engaged with, while not all content for the current web as of 2017 will be linked to, but might be shared on a social network (a form of engagement).

Why should you care?