Is search data the new crystal ball?

Blog post originally published on State of Digital as part of a monthly column.


The more ubiquitous, pervasive and natural search is, the more intelligent it becomes. No longer is it a magnifying glass surfacing content from the depths of the web. Search is starting to look more and more like a crystal ball capable to predict the flight of flu epidemics, match winners and presidential election outcomes.

Not many technologies are capable of processing as much data, as frequently and actually make sense of it all. Search was fed on big data, grew with artificial intelligence and, if some say it is not rocket science, it is verging towards science fiction.

Brains in a box.

Historically search engines were indexing a web of documents to point searchers in the most relevant direction when they tapped a couple of keywords in a search box. Coping with the expansion and diversification of that universe was no small task, neither was the treatment of the ever increasingly more complex queries. So the technology had to gain in sophistication.

Algorithms started to extrapolate the strings of characters that were inputted. It became less about finding a specific phrase, and more about understanding its meaning. It was an evolution dictated by necessity. First, human beings are so prone to mistyping that machines could not rely on us ; second a same need can be expressed by different synonyms; and third, words have several meanings based on their context (e.g. from a PC, searching for “coffee” may relate to coffee harvesting whilst the same person using the same keyword on his smartphone may be after a caffeine shot).

This led to new functionality like query suggestions, auto-correction, auto-fill, semantic search… but also drastic evolutions of the algorithms with the integration of social, geographical or device signals. Search was no longer literal; it had become contextual.

From smart to intelligent
However, searchers still require to proactively engage with a user interface in order to trigger queries. These interactions remain contrived, even if you consider the conversational nature of voiced queries. Search will only truly become intelligent when the engine can anticipate what I need, even before I verbalise that intent. That is one of the promises of digital personal assistant like Cortana who relies on Bing information architecture and machine-learning to anticipate your needs. One of my favourites is her ability to urge me when to leave for my next appointment by making sense of my current location and the traffic conditions to my destination.

So could we take anticipation to the next level and predict the future.
Search engines are a database of intent where millions of people converge to look for information of what is top of mind for them. At the same time, social networks are the depository of sentiments. If you have developed the ability to process, analyse and understand these two humongous, historical and real-time information sets you have the opportunity to discover user sentiment for certain events or entities, estimate popularity trends, as well as predict outcomes of future events.

Bing Predict explored that concept with popularity-based contests like American Idol, for which web and social signals can highly correlate with popularity voting patterns and thus allows the engine to accurately project who will be eliminated each week and who the eventual winner will be. At the other end of the spectrum, predicting the outcome of the World Cup, Tour de France or the Premier League requires the incorporation of player/team stats, tournament trends and game history, location, and data from social channels.

The data from social channels provides the Bing model with the “wisdom of the crowd.” This approach is different from predictions for popularity-based contests. That model is able to interpret specific data as priority information such as team strengths, as popularity alone doesn’t dramatically help a team win or lose (some fans may object to this assertion but it’s largely true).

This machine-learned approach proved to be more reliable than traditional statistical methods on several occasions. Bing predicted accurately the Scottish Independence Referendum  outcome from the very first day whilst the official statistic was oscillating between the Yes and the No. Our predictions for each of the men’s and women’s Wimbledon matches had an average accuracy of 71 percent, and got the winners from the first serve. We also predicted Froome’s victory in the Tour de France.

What can brands learn from these forward-looking experiments?

Machine learning models are already making their way to the advertiser toolset. Bing Ads for instance includes an opportunity tab which allows brands to evaluate the future impact of actions taken on their search marketing campaigns based on auction and competitive behaviours. That is just a first step.

I have already written about how brands should think outside the (search) box , and harness the full potential of the search data to inform their marketing strategy. Think for instance about the evolution of the geographic spectrum of your search queries to inform your stock strategy for the next holiday season.

Next, businesses can enrich their own data with real-world, publically available data sets to identify further correlations. It can be a small collection of manually curated convention centre calendars which infer future influx of visitors to a city, or richer data sets from Open Data sites around the world .

It might take some creative thinking on your part to reveal true insights, but ignoring this resource means missing out on a big opportunity to create value for your company and customers. This is about modelling the real world in advertising campaigns with extra rigor and an opportunistic mind set thanks to the accessibility and democratisation of Business Intelligence tools, like PowerMap or Cortana Analytics .

Finally, I am convinced that soon enough new advertising models will come to fruition. Trajectory marketing, for instance, would consist in geo-targeting consumers based on the location they will be at rather than the location they are, by modelling their current position, their celerity, external factors like traffic, weather conditions, etc.  After all, marketing is about seeding the right message to the right audience, at the right time.

And that time is in the near future.