The heart in Artificial Intelligence

Blog post originally published on State of Digital as part of a monthly column

My son Arthur has just been awarded a prize for story-telling at his primary school. So when I watched that short movie whose script was generated by artificial intelligence, based on thousands of sci-fi books and films, I could certainly see a lot of similarity between both outputs. For me this epitomizes the current state of AI… It is raw, forming, full of potential but still with a long way to go towards maturity.

Today we will be covering the heart in artificial intelligence. After all, if artificial intelligence is, by definition, artificial, how can it have a heart, how can it have emotions? On the other hand, if AI is the brain child of thinking, feeling people, how can it not have a heart? This is a critical question for us, marketers who want to trigger emotional reactions from consumers but who also rely always more on algorithms and automation. To answer it, we will first need to define artificial intelligence. We’ll explore the 7 outcomes we can expect from AI, and consider how we materialize these expectations today to enable everyone of us to fulfil our potential.

The building blocks of AI

To understand if a heart is beating inside artificial intelligence, we need to understand where AI comes from. Although AI has been all over the press lately, it is not news… It is rather a 30-year old corpus of work, aimed at creating intelligent machines, by combining three building blocks: machine learning, human learning and data science. And in many ways there is a strong analogy between AI and raising a child.

Just like children get their foundational learnings from their parents, teachers and by the school books they read, machine learning is based on known properties, and the machine learns from the data. Think if/then scenarios. If your son behaves well, then he will be treated by Santa. If your daughter sees a puddle, then she should not to stomp in it to keep her feet dry. This is also how machine learning works: if you liked that book, then you’ll probably like these ones too. If you bought a laptop, then you should consider this bag. These are just small, basic examples of a very complex field.

Kids learn fast that if they cry and shout they get your attention… Now, you will certainly want them to assimilate that such a behaviour is not a normal mode of expression. Human learning is how we make course corrections to the machine learning that is happening. Cortana, Microsoft’s digital personal assistant, has a team behind the scenes working on human learning so she can get smarter. This human learning gives the digital personal assistant more personality, and her responses to queries are more human because of this.

Data science is the third brick of artificial intelligence. Data science is the discovery of unknown properties, or connections, in data. In this case, the machine is presented with a massive amount of data and asked to find connections in it. This is how we might discover that watching a certain program in your youth increases your chance to marry a foreigner. We didn’t know there was a connection between these pieces of data until we went looking for that connection.

Human intervention

We have seen these three fields accelerating their capabilities recently due to the exponential development rate of our computing power. We are able to process, analyse and render an ever-growing amount of information, at an increasing pace. But where does that data that feeds machine learning, human learning and data science come from?
It comes from us! Artificial intelligence comes from us. In many ways, it is us.

Artificial intelligence is only as intelligent as the data it takes in. It is only as fair as the data it takes in. It is only as human as the data it takes in. It is only as socially acceptable as the data it takes in. I would like to share with you two examples of AI, which to a large extent illustrate how humans can influence how intelligent a bot can be.
Remember Tay, Microsoft first experiment as a Twitter bot? Tay learned from her inputs, which were hijacked by some people who wanted to influence her negatively. In this case, Tay incited high emotion from people who engaged with her or read about what happened with her, even if Tay, herself, did not express emotion and was merely a reflection of the hatred that fed her.

On the other end, Microsoft also created Xiaoice a couple years ago and it is a perfect example of where technology is going and why we think of conversations as a new platform for brands and commerce. Xiaoice is a chat-bot based on Bing search technology and big data. It draws on AI, social media, and machine learning so she can hold a proper conversation – the average exchange between Xiaoice and a user has 26 turns. She’s sensitive to emotions and remembers your previous chats. If you tell her about a breakup, she’ll check in with you. If you introduce her to a puppy through a photo, she’ll recognize the breed, ask you for its development. And to say this bot has been popular is an understatement. Three days after she was available, Xiaoice had been added to 1.5 million conversations on WeChat. Once added to Weibo, the Chinese micro-blogging service, it became one of the most popular celebrity accounts. And today, Xiaoice is used by over 40 million people.

Tay and Xiaoice are like two twins, split at birth and raised in two different environments, with different influences… Two very different individuals in the end.

Assessing our expectations

So, what can we reasonably expect from artificial intelligence?

As mentioned before the computing advancements have enabled a fast acceleration of three technologies which underpin the maturation of artificial intelligence: object recognition, natural language processing and speech. If the AI can see, speak and listen, it is not far from being able to exchange with human being transparently.

Actually, mid-October 2016, Microsoft researchers announced they had reached human parity with the word error rate (WER) for conversational speech recognition, meaning that their AI was as capable as a professional transcriber to write up an oral conversation. Language understanding and acquisition is not easy, and it is critical to the success of AI. If you travelled a bit, you will be familiar on the complexity implied by accents, dialects, pronunciation but also the fact that a same word may have several meanings based on the context. This progress was critical because without this piece of artificial intelligence, so many developments wouldn’t move forward. Think about how patient you would be with a digital personal assistant or a sales advisor that misunderstood most of what you said?

Natural language learning is a complex skill, as we know from watching our children learn to speak. But with our increased computing capabilities, not only are we able to recognize accurately the words but we are able to do this instantaneously. This unlocks new scenarios like Voice-to-text which allows deaf children to read the transcript of a discussion in real time or Skype Translator which not only has the natural language skills necessary for a conversation but can also translate into other languages.

Well, this outcome is one of many. Capitalizing on the progress of machine learning around object recognition, natural language processing and speech, we have seen our expectations towards AI graduate from the most basic to much more advanced outcomes.

The 7 outcomes of AI

According to Silicon-Valley analyst, Ray Wang, there are seven intertwined outcomes for artificial intelligence, based on what we are now able to program via machine learning.

1. Perception is an example of early machine learning, now totally engrained in our daily life. Drawing on existing data, the machine delivers information about what is happening now. The weather, traffic, sales volumes, stock prices – things that are measureable and reportable. This AI outcome brings us back to the core promise of search engines when based on a typed or voiced query, the machine learning understands the intent and provides the answer or links to the information. For humans, learning to express their perception, it’s pretty simple as well. A child can describe what is happening now with ease. We learn this almost immediately: it is dark; I am hot; or, based on these circumstances, I am joyful. To illustrate a more advanced Perception outcome, we can look at facial recognition and play with http://how-old.net which assesses your age based on your traits (and which we hate to be accurate).
2. Next, Notification. If I did not have my calendar delivering notifications, I would be a horrible colleague – late to meetings or just not showing up because I cannot hold my schedule in my mind. Here the intent is less explicitly verbalized, but it is still initiated by the user and the information remains factual without any analysis of the data. We learn notification early as well, perhaps starting with letting Mom know we’re hungry. Fact: I am hungry; Notification: I cry. It never stops – in school, we notify the teacher that we have the answer.
3. Suggestion is another area we have grown to be familiar with, and is now engrained in our daily life. You searched for these words, but “Did you mean?”… The machine learns from past behaviours and suggests alternative actions. We all love this machine learning with our Spotify account for instance. If I listen to a song and I like it, the AI suggests more songs for me to enjoy. And you can always retain that Human Learning capability to ensure that the AI never drifts from Justin Timberlake to Justin Bieber… Early suggestions were basic, but imagine what can influence them today: demographics, location, day, time, weather, behaviours, etc. The data sets are humongous but we are now capable to combine and process them in no time and identify new, maybe more obscure connections
4. Our children learn a nice drawing will trigger a smile from their parents, or that it’s time to wash their hands before a meal. Over time, we don’t even have to remind them; they just know it’s what’s next and it becomes Automation. A suggestion or a recommended action can grow into automation based on learning your preferences. If you follow avidly the progress of your favorite team, the AI will start to automatically inform you of their performance. If you always make a reservation for 7pm on Saturdays, your AI will start to spontaneuously fill in the date and time on your reservations. If you trigger the same report every Monday morning, the machine will start to pull the information for you and make it available in your Business Intelligence dashboard.
5. Predictions can be the hardest machine learning to train, because so many variables can affect this outcome. Think of a child who sees Daddy packing a suitcase; based on past behaviour, this toddler knows that this means Daddy is leaving for a few days, which is sad. But sometimes it also means that the child gets to travel with Daddy. What factors will alert the toddler about what outcome to expect? Microsoft has developed a program called Bing Predicts which combines and models all the data signals we can find, and comes up with incredibly accurate predictions. It initially explored popularity-based contests like American Idol, for which the web and social signals are very strong and highly correlate with popularity voting patterns. You search for information about that performer, his history, his latest video clip. At the same time, you comment the performance on Facebook or Twitter. By combining anonymized search patterns to social signals Bing Predicts could accurately project who would be eliminated each week during American Idol and who the eventual winner would be. More complex, we then turned to sporting events and even world political challenges. During the World Cup in Brazil, our team predicted accurately with 100% accuracy the winners of the final elimination round. During the last year Rugby World cup, we had 87% accuracy across the tournament. Surprised? In order to successfully predict a sporting event outcome, the number and type of signals we incorporated quadrupled from what we used to predict a basic popularity event like American Idol. This is because we recognize that popularity alone does not predict whether a team will win – Sorry for the fans. A fan base has however special insight into the abilities of their teams, and those fans are having constant discussions about their team. This is called the “Insider Knowledge.” We weighted their knowledge against player and team stats, tournament trends, game history, location and even weather conditions. This is how we were successful in our predictions.
6. If we manage to predict accurately the future, the next logical step after prediction is Prevention. Again Bing Predicts shines in this category: by analysing large samples of search queries, Microsoft scientists have been able to identify internet users who are suffering from pancreatic cancer even before they were diagnosed. The researchers focused on searches conducted on Bing that indicated someone had been diagnosed with pancreatic cancer. From there, they worked backward, looking for earlier queries that could have shown that the Bing user was experiencing symptoms before the diagnosis. Those early searches, they believe, can be warning flags.
7. Finally, Situational Awareness for AI comes close to mimicking human behaviour in decision making. We see situational awareness as a combination of many aspects of AI, from object recognition to conversational speech. Here’s an example:

These 7 outcomes are complex and require a lot of training and time to accomplish. They are also interconnected and not mutually exclusive. They actually build upon each other to offer the benefits of AI to us, users.

In conclusion, everything we’re seeing with AI is exciting and rich. We see the heart in AI every day, when we ask it to help uncover cancer, help two people connect when they don’t speak the same language. But where is the moral and ethical compass for artificial intelligence?

As alluded to through this article, AI is still at its infancy and it is our collective responsibility to set it on the right trajectory. At Microsoft we are committed to this, and partnered with the University of Cambridge and the Partnership on AI, two international authorities to help shape the future of that promising discipline. For some, AI is a modern Oedipus that will have to “kill the father”, take away our jobs, make ourselves redundant. But for someone like Satya Nadella, AI will actually enable people to fulfil their full potential as we have seen across the 7 outcomes of AI. So yes, for Microsoft, AI has a heart. It is the mankind’s heart.

An Introduction to Conversational Commerce and Bots

Blog post originally published on State of Digital as part of a monthly column.

Conversations.

These days, there is a lot of chatter about talking – and it’s not that surprising considering the forays in AI, machine learning and natural language processing have made interactions with technology more conversational, more human. Bots are now capable to think, but also view, speak, and listen. So businesses are discussing how they can, could, will, and should be using bots to drive more personalised conversational interfaces with their customers.

“By 2020, customers will manage 85% of their relationship with the enterprise without interacting with a human.” – Gartner Predicts

I already prefer to use an ATM at the bank and I go to the self-checkout when doing my groceries. It doesn’t seem like that far fetched to see how conversational bots can become an extension of a self-service interface to your brand across the platform and channel your customer choose.

Bing has already a few chatbots integrated directly into the search results page, creating new, deeper engagement with the brand right here where the customer intent is verbalised. As I was planning my latest trip to Seattle, I’ve found it pretty useful to interact via the bot to quickly discover the parking availability for the restaurant I was heading to and to make sure it offered gluten-free dishes for the colleague who was joining me:

What are bots and chatbots?

There is no difference. In the beginning bots were just pieces of software designed to automate specific tasks. Today, bots have evolved thanks to the accelerations of our capabilities in machine learning and natural language process to comprehend and engage in conversations. They are acting as an interface that can be plugged into APIs or various data sources to deliver information on demand and help drive conversational commerce. In many ways, they are a new paradigm to how we will consume the web information. We used to go to web pages, then to interact with apps… We are about to exchange with this new interface through natural language which will, in turn, retrieve the information we are after. We will stop browsing or tapping: we will start asking.

What’s the difference between bots and digital assistants?

A digital assistant like Cortana, Siri, Alexa or the Google Assistant is more advanced than a chatbot. Some call them “meta agents”. In addition to being able to parse conversational language like a chatbot they are also incorporate additional layers of artificial intelligence to help merge utility, productivity, entertainment and the ability to accomplish actions together to become an intelligent agent. It is both contextually and historically aware, which means that it can provide relevant information, proactive recommendations, tailored to your preferences and situation. As such, Cortana is a predictive and proactive agent that can, based on interactions, learn to interact with you, other people and bots.

What’s the difference between bots and skills?

What do you want to teach the digital assistant to do? A skill. A skill teaches the digital assistant how to do something or to take an action, based on a voice command. I might ask, “Hey Cortana, ask HUE to dim the kitchen lights” or “Hey Alexa, book a UBER in ten minutes!”,  “Hey Cortana, let OPENTABLE book a table for two at John Howie Steakhouse at 7pm.”

Conversational Commerce – Connecting the dots

When it first launched, Domino’s Pizza Bot was super simple. Simple but effective. Customers typed the word Pizza and it would place for their connected account an “Easy Order.” The bot wasn’t really conversational in nature, but it started with an easy-to-use feature which allowed Domino’s to get their foot in the connected kitchen door in order to test and learn. Currently, Domino’s has both a true conversational commerce chatbot and Amazon Alexa skills with increased capabilities. Although retaining their ambition for a simple consumer benefit, ordering a pizza, they have upped the AI complexity to allow users to build their order from scratch, but also to track an order or reorder their most recent order.

Let’s take a high lever look at getting started with bots and what you need to keep top of mind as you get started.

Tips for getting started with bots:

1. Plan
   1. Set goals and expectation for what it can do.
   2. Focus on interactions that mean the most to your customers
2. Start Simple:
   1. Focus on building a feature that works amazingly and will delight your customers.
   2. Don’t reveal all the features at once. It can overwhelm your customers.
   3. Integrate the features into the flow of the conversation where they make sense.
3. Develop your bot:
   1. Choose a frame, like the Microsoft bot framework that can help you scale across channels.
   2. Don’t try to launch across every channel at first. Take a test-and-learn approach and roll out features and channels.
4. Monitor bots closely:
   1. Mistakes are inevitable – Learn from them and fix any identified errors/mistakes often and quickly.
      1. Examine it: What questions did people ask that the bot wasn’t able to answer?
      2. Teach it: What words does the bot not understand? Does it not get that veggie is another term for vegetarian?
      3. Humanize it: Does the tone of your brand shine through? Make the experiences more engaging?
   2. Ask your customer for feedback, especially early on. Give your customers the option at the end of the session if they’d like to participate to help make the bot better!
   3. Learn from customer interactions and feedback.
   4. Generate smarter, more personalized interactions capitalizing on the growing number of cognitive services available
5. Iterate and Adapt. Rinse & Repeat.

Everyone loves a good conversation. Bots and skills are opening up ways to communicate directly and more conversationally with your customers – providing them a more natural experience. It also can give your business a deeper look into the customer experience – including their emotions and sense of urgency during the interactions. Bots and skills can help you provide more personalized experiences that create a more meaningful connection with your customers.

So, what if we’d stop chatting, and started coding?

Podcast: speaking with Movidiam on AI, Chatbots and Augmented Reality

I was recently invited to contribute to Movidiam's podcast and share my thoughts on the future of search, chat bots, Artificial Intelligence, the rise of Augmented Reality and their impact on marketing. If you regularly read this blog, you will be familiar with some of the concepts and principles that I am touching base on, but nonetheless you may be inclined to stop reading and start listening to my French accent:

Here is a link to that podcast.

The evolution of search through a modern consumer journey

Blog post originally published on State of Digital as part of a monthly column.

The IAB UK has just released their digital advertising spend report for 2016 and announced that digital advertising grew at its fastest rate for nine years – up 17.3% to £10.3 billion. The last time annual growth was higher was in 2007 (+38%), when Apple released the iPhone leading to the accelerated adoption of smartphones, the emergence of the app paradigm, etc. The reason I call out that historical reference is because I genuinely believe we are at an equivalently critical inflexion point with the rise of AI: it grew out of search and is about to transform marketing as a whole.

As a matter of fact, the historical powerhouse of the digital landscape, Search, has continued to grow by +15% to £4.99 billion, which reflects two major trends:

• First advertisers have (finally) realised that search plays a role throughout the customer decision journey and not just at the final conversion step
• Second, search is, in itself, evolving drastically to offer new capabilities and surfaces of engagement for advertisers to capture new intents, new behaviours, new usages

Following a modern consumer journey and its query paths

As disruptive technologies reshape the digital marketing landscape, brands are investing time and effort to remain relevant and top-of-mind with consumers. For years, search marketers have obsessed over bottom of the funnel activity for its seemingly higher CTRs and conversion rates, in part fuelled by last click attribution models. But most marketers today agree that it is essential for a brand to appear across all stages of the funnel to drive brand affinity and recall.

Research at Bing Ads now allows us to quantify and better understand the customers behind the clicks and where they stand on the customer decision journey (CDJ). Consumers tend to go through five distinct stages, which vary depending on the type of purchase: initiation, research, comparison, transaction and experience. Consumers are subject to these same five CDJ stages, although each step will vary in length and importance depending on the cost of failure, frequency, cost and complexity of the task, and the shopper type.

As consumers go through a given decision journey, search plays a pivotal role and different types of queries will be used throughout:

• Category searches: Include broad search terms that are not product or brand specific, such as shoes, running shoes, hiking shoes. These usually appear during the initiation, research and comparison stages but can also take part during transaction
• Tangential searches include queries that are related to a given journey, but not in the exact same category. For instance, if I am searching for running shoes, a tangential search could be about preparing for a half marathon, running training program or running equipment. These are similar to category searches in that they usually get searched on before a transaction
• Competitor brand searches: Include your competitors’ brands or products. This is an opportunity for your brand to appear through conquest advertising. For Nike, these terms could include Under Armour, Adidas, etc. As consumers get further along their decision journey they begin to hone in on their purchase and more specific searches occur
• Brand searches: Include a specific brand or product name. Nike is the brand and product searches could include Nike Pegasus or Nike Running shoes. Brand terms are often the best performing keywords helping drive the most transactions

Search drives awareness

Traditionally search has placed a premium on brand keywords. The Bing Ads research team has found new insights on the role of non-brand search terms within today’s journeys. Brand Impact Study on usage data from a major US auto retailer determined that 72% of brand ad clicks (which includes category, competitor and tangential searches) had a non-brand keyword precede the brand click. In other words, retailers who do not run non-brand keywords throughout multiple stages of decision journeys are missing out on a majority of relevant searches and leaving gaps for their competitors.

Additionally, consumers who are exposed to a brand ad on a category or competitor query were 30% more likely to do a branded search. On average they had a 15% higher conversion rate compared to consumers who were not exposed to the brand ad. Having a branded ad appear in category and competitor brand queries improved brand affinity and recall, and increased the propensity for future brand searches.

Another Bing Ads research completed in Q3 2016 with a leader in the automotive insurance vertical measured brand awareness and perception based on exposure to ads within the search results pages. The key learning from this study was that searchers on Bing who saw a branded ad for non-branded search queries showed a statistically significant increase in brand awareness, perception and purchase intent. After being exposed to a branded ad, searchers indicated a 24% lift in unaided awareness, 28% lift in purchase intent and a 30% lift in viewing the retailer as a market leader. Brand awareness further improved when searchers clicked through on the ad and were taken to the brand’s landing page. In the study, consumers who were merely exposed to an ad without clicking on it, self-reported as more likely to take an action or next step

Although search marketers have long assumed some level of brand awareness in search, it is the amount of brand awareness shown here that proves impressive and indicates the importance of staying present throughout each stage of a journey

Search informs and educates

As consumers go through the initiation phase of the consumer journey and begin to do more research and consider purchasing, search can help with the decision-making process.

A Forrester Consumer Technographics study called out that 60% of consumers will use a search engine to find the product they want and 61% will read product reviews before making a purchase. And one of the most trustworthy sources in this research and comparison phase is none other than search engines. Furthermore, consumers consistently rely on search engines, more than any other source, as a reliable place to research about brands, products, or services that they are considering buying. According to Forrester, 49% of consumers reported that they rely on search to inform purchase decisions, and 19% of respondents identified search engines as the most influential source in driving their decisions.

New search experiences, new marketing expectations

The evolution of the consumer behaviours outlined above and the reaction from marketers by investing throughout the different stages of the consumer journeys are a sign of maturation for a discipline that is merely 15 years old. And yet, new search experiences powered by AI are coming to the fore, and are likely to further disrupt the model.

It has been well documented that search is no longer just a search box on a webpage with a list of links. It is literally breaking out of the box; it is changing in some dramatic and exciting ways. Once a function that used to be limited to a text box on a specific site has slowly been integrated into more and more of the technologies/devices, apps and sites we use every day, from phones to gaming consoles. As such, Search becomes more pervasive, but also more personal and predictive.

Conversations as a Platform is a great embodiment of how the Future of Search makes sense of these 3P (personal, predictive and pervasive). The rise of messaging apps where consumers now spend most of their time and technological advancements in NLP, AI and machine learning are creating rupture that is similar to what we have seen in decades past with the App paradigm. Conversations as a platform unlock a more human, personal way to discover, search for, access and interact with information. This new platform will enable us to interact with devices more intuitively, using natural language conversations, evolving us from mechanical keyboard and mouse to touch and beyond.

Search will never again be a constrained to writing in a search box. Instead, search will be a partner that can listen and communicate in dialogue with a consumer on any platform and any device. Thanks to new, more natural interfaces, voice search is becoming increasingly possible and accurate. We imagine a rich ecosystem of conversations, ones that include: people to people, people to your personal digital assistant, people to bots, and even personal digital assistants calling on bots on your behalf. That’s the world that you’re going to get to see in the years to come.

We have already covered the current rise of voice search as a result of mobile search adoption, but it is likely to further accelerate as search becomes more pervasive and turns your TV Set, you fridge or your car into a search box. ComScore predicts that by 2020, over 40% of the queries will be voiced rather than typed. It is worth noting that these experiences are often screenless which further disrupt the preconceptions that too many marketers have when they think they have nailed their search strategy.

How do you optimise your web presence for voice search? How do you market to chatbots? How will we see consumer intent evolve in these new experiences? We already seeing that 16% of searches every month had never searched for before. We also see that 25% of the clicks on the Bing Network are from searches not happening on Google… So, the versatility of the above-mentioned consumer journey is likely to continue to increase. And what an exhilarating evolution it is!

Podcast - glancing at Advertising Week on Chatbots, AI and Machine Learning

Back in March, as my team took part in Advertising Week, I was honoured to be invited by Jason Miller, from the Sophisticated Marketer podcast, to deep dive into the evolution of Artificial Intelligence, Chatbots and what it means for marketers and how they can utilize the technology now and prepare for the future:

Live from #AWEurope: Chatbots, AI and Machine Learning

Connected cows and the greener pastures of big data

Blog post originally published on State of Digital as part of a monthly column.


Remember the cringing router sound that accompanied your early internet connections? Being a 40-year old father of two, when I hear that screeeeeeeechhhhhhhhhh I can’t avoid feeling a smile pop on my face. That sound reminds me, with a touch of nostalgia, of the cry of a newborn. It makes me smile as fond memories have overshadowed the dirty nappies and sleepless nights.

To a certain extent, that sound was indeed the cry of a baby internet, when browsing was a commitment and a flaming logo the pinnacle of creativity. The worldwide web has since grown up. A lot.

Small steps towards big data

Looking back, that dial-up buzz was literally the first signal in our digital footprint. We signed on. And in that very moment big data spun to life.
What were we learning from those early signals? Not much. We knew how many people were online and roughly where they were. But as the web of documents developed, search engines arose and our ability to understand people increased. We started to know what you wanted. Think about it: you probably tell search engines things you wouldn’t tell your closest friends. We understood that your inputs into a search engine were, on a personal level, an expression of your desire, and on a global level, an expression of the world’s consciousness. The so-called Zeitgeist.

Although I have spent the last 10 years in the realm of online advertising, in today’s article I will focus less on search marketing, and more on the information infrastructure and machine learning that Bing is part of, looking at how this is influencing our future. What are we doing with our data footprint? Over the course of the last year I have asked many people across Europe how the idea of data collection made them feel. By large, the response was discomfort and hesitance. Until provided with more perspective.

The heights of data complexity

So let’s get back to our story. In order to understand the complexity and depth of the data infrastructure that we are part of, let’s contemplate what has changed since the emergence of the first search engines. With each of these four changes and associated amount of data surge that came with it, you need to visualize a growing mountain.
First is your search habit. From a few searches per day to multiple searches per hour, we are now searching constantly, and not just you but also the billions of people who got online in the recent decade.

Second is your search access. Most of us had access to a desktop computer 20 years ago. But just one. A grey, cold box, sealed to a desk. We certainly couldn’t put it in our pocket and take it with us to a party. It is not just computers, think about all the devices you own which are harnessing computing power: laptop, tablet, smartphones, TV, but also your car and now your fridge.

The third big change is your search expression. You have gone from using basic computer commands, with amp signs and inverted commas, to using more human a language. You’ve gone from asking “what” to asking “why…” and “how to…”. In fact we have seen the growth of queries starting by Why being three-fold the growth of What queries, which means we are no longer looking for information, we are looking for answers. You’ve layered sequential searches on top of these, in a complex web of intents.

Finally, the integration of search with other infrastructures has also changed. A search engine used to be an isolated service. Now it’s plugged into the social graph. This means that several points of contact are linked and with them a flurry of new signals, millions of them that only a few super-computers are able to capture, organize, model and render. Search engines are the database of intents, and social networks are the depository of sentiments. We have developed the ability to process, analyze and understand these two humongous, historical and real-time information sets together.

The search crystal ball

We can understand your sentiment for certain events or entities, estimate popularity trends, as well as predict outcomes of future events. Microsoft has developed a program called Bing Predicts which combines and models all the data signals we can find, and comes up with incredibly accurate predictions. We initially explored popularity-based contests like American Idol, for which the web and social signals are very strong and highly correlate with popularity voting patterns. Bing Predicts could accurately project who would be eliminated each week during American Idol and who the eventual winner would be. Just by using all of the signals that are out there.

Getting more complex, we turned to sporting events and even world political challenges. During the World Cup in Brazil, our team predicted accurately with 100% accuracy the winners of the final elimination round. During the last year Rugby World cup, we had 80% accuracy across the tournament. Surprised? In order to successfully predict a sporting event outcome, the number and type of signals we incorporated quadrupled from what we used to predict a basic popularity event like American Idol. This is because we recognize that popularity alone does not predict whether a team will win – Sorry for the fans. A fan base has however special insights into the abilities of their teams, and those fans are having constant discussions about their team. This is called the Insider Knowledge. We up-weighted their knowledge against player and team stats, tournament trends, game history, location and even weather conditions. This is how we were successful in our predictions.

We finally turned our attention to political events, and in particular the Scottish referendum two years ago. The process and results were presented at TEDxSuzhou.


We were and are predicting the future. Can you imagine a business need that this kind of prediction can answer? Of course you can! We’re experimenting right now with predicting the upcoming trends in fashion, in automobile, in technology – so we can help our advertisers make smarter business decisions.

So we saw how predictions can play a role in entertainment, sport or business, fine. Fine, until we find a way to make this kind of data infrastructure even more meaningful, at a society and mankind level. What can we do with this capability that goes beyond entertainment and the novelty factor? Can we use our big data to make a meaningful impact on society?

Up close

All of this is exciting on a global or country level. When we’re talking about millions of inputs, it’s no wonder you can make predictions and have an impact like this. It is just a massive sample size. What about bringing this big data infrastructure to a personal level? Is it possible for a machine to learn so much about you that it can accurately predict your next move? Or predict when you will need something, and provide it? That is the promise behind digital personal assistant like Cortana.
Cortana is not only on Windows Phone but also Android and iPhone. And since the release of Windows 10, she’s even on your desktop. As outlined in a previous article, you set up Cortana with some basic info about yourself, then use her to help you with things like scheduling and reminders and web searches. Before you know it, Cortana is spontaneously sending you an alert to inform you that you should leave the office now to be on time for your next appointment in Farringdon, because she found some congestion on your normal route. It doesn’t take Cortana long to learn so much about you that she can predict your next move and offer assistance.

A new layer of data in your coat

While our mobile phones aren’t exactly wearables, we sometimes behave as if they are, keeping them on our body no matter where we go. With wearables, two important things converge: big data infrastructure and your expectations.

When you hear “wearables”, you probably think of a smart watch or one of these fitness bands. But to go back to my introductory analogy, these are just the first baby steps towards the full potential of wearable and how that technology will be able to enhance our capabilities, as individuals or as professionals. Think about it: wearables can capture and communicate signals about your location, your manner of travel – whether you’re on foot or in a car – time of day, most recent queries, usual route home from work, the weather, your physiological state, etc.

So for instance, if your wearable identifies that your hydration is low, it could prompt a notification that factors in your location, whether you’re moving, what time of day it is and therefore whether the nearby branch of your favourite coffee shop is open. It could even cross-reference this with your earlier interest in gingerbread lattes, and the fact that it is raining, and direct you to the nearest open coffee shop with plenty of indoor seating and gingerbread lattes on the holiday menu. Your wearable might even send you an alert for a coupon the coffee shop is offering.

Greener pastures ahead

As the wearable technology grows, your expectations for your experience with technology in general will change. And that is for the better. After all what the point accumulating data points like hoarders unless you do something greater about it. And if I have learned something about the internet, is that it is a fertile ground for creative usage of untapped opportunities.

I am from the French Alps where I spent most of my summers walking the mountains with my grandmother. She used to herd cattle in these alpine pastures and she was telling me stories about how much each of her cows were almost like members of her family. They had names, and she could tell when something was wrong with any of them.
These days are gone. Nowadays a farm is no longer taking care of a small dozens of cows, but hundreds. The personal relationship of each animal is no longer an option. The story of the connected cows started with a farmer in Japan who was exhausted with the effort of figuring out the exact time his cows were fertile – because it is a very short window, only 12-18 hours every 21 days, and it happens usually between 10pm and 8am. Of course knowing this precise time of estrus would give farmers a chance to successfully inseminate the cows.

These are farms with hundreds of cows – you can image what a nightmare this would be to keep track. Could technology help? A farmer in Japan asked Fujitsu for help. Fujitsu consulted with some university researchers and they came up with this idea of putting wearables – pedometers – on the cows, and providing the data to Microsoft Azure, in the cloud, for analysis and alerts that go straight to the farmer’s smartphone.

It turns out that when a cow is in estrus, she paces. The number of steps she is taking increases tremendously, and this data alerts the farmer to the right moment for fertilization. The connected cow project has been 95% accurate – and that 5% where it misses the mark turns out to be when the cow actually skips the farm and goes missing.
Not only is this wearable incredibly accurate, it also helped the researches discover that there is an optimum window for fertilization if you’d like a female or if you’d like a male. With 70% probability, a farmer should fertilize in the first half of the estrus window if he needs more milk cows or if he needs more bulls. But it does not stop there… The Fujitsu researchers were able to also correlate pacing patterns with increased risks of genetic diseases and pathology.


It is amazing what data can tell you, if you know how to look at it. Sometimes creatively! This is the joy of data infrastructure. We can do wonderful things in the world when we collect, analyze and render the data that’s available to us. Microsoft is on the leading edge of this, with products like Power BI, Azure, our cloud platform but also Bing our search engine and its machine learning capabilities which can make sense of the millions other data points that come together to make big data smart, useful, creative and – yes – joyful. And you, what was the last time you found a creative inspiration in your data set?

(Super)human nature

On your marks...

The 31st Olympiads have started, and I am ecstatic. I indeed have a personal relationship with this global event since I was born in an Olympian city, Grenoble, just like my kids who saw the light in London. I have particularly fond memories of London 2012, and wish to my Carioca friends to experience the same exhilarations we had back then.

The Olympics are a fantastic platform to see human nature at its best: physical exploits, determination, team work, perfection, apolitical statements and some more loaded, the highest degrees of emotions like happiness and despair, intertwined and simultaneous. You simply cannot remain unmoved by this competition. When tears blend with sweat. Why cries of joy cohabit with cries of distress. When pain is the path to pleasure.

However something which gives me even more goose bump is what Paralympians achieve. It is close to superhuman. At least that is what Channel 4, the Paralympic official broadcaster, claims:


Enabling abilities

It is incredible what these so-called "disabled" athletes are actually able to achieve. In many ways they are more capable than many of us. Actually, Oscar Pistorius did compete in the London 400m race against "able" athletes...

His participation raised some questions at the time, because observers wondered if his handicap was an unfair advantage over the other competitors. Actually, to be precise the debate was not exactly on the handicap itself, but rather on the technology used to address it: the blades. Would the blades provide extra spring and pace that human legs would not be able to provide?

More interestingly, it raised an ethical debate which tells a lot about human nature: it was less about diversity and inclusion of disable athletes amongst able competitors, but would Pistorius' participation set a precedent and open the door to technology-enhanced bodies? After all, if some are ready to inject some illegal chemical in their metabolism to enhance their performances, would some be ready to deliberately alter their body to integrate some technology that would multiply their capabilities? What is disability? Could some weirdoes mutilate themselves to compete? Scary, but plausible.

Technology opening doors.

Debate aside, I am amazed by how some technologies are enabling people to live up to the Olympic motto: faster, higher, stronger. More capable in other words. I find this exciting, don't you? And not just in sport... I already wrote about how technology helps with the virtualisation and dematerialisation of our lives. But technology is capable of such grand things, like allowing deaf people to hear or colour blind people to discover the chromatic gamut.

Let me introduce you to Neil Harbisson, a Catalan-raised but British-born artist and cyborg activist who has made the headlines for having an antenna implanted in his skull and for being officially recognised as a cyborg by a government. The antenna allows him to perceive visible and invisible colours such as infrareds and ultraviolets via sound waves. In other words, he does not see colours... He hears them. Each colour is associated to a sound, and with each sounds comes associated emotions.

What I like about this last example is that it redefines entirely the notion of ability. Technically, Neil is not able to see colours, but he has invented a new way to perceive them and in fact this new ability is richer than the classical sight because it spans beyond the human visible spectrum. Just like these athletes are not just emulating able athletes, they are defining new performances in totally new categories. They are shifting the battle ground to places where they are not disable... To places where they thrive. For that and for everything they do, they have my unconditional respect and admiration.

Is search data the new crystal ball?

Blog post originally published on State of Digital as part of a monthly column.


The more ubiquitous, pervasive and natural search is, the more intelligent it becomes. No longer is it a magnifying glass surfacing content from the depths of the web. Search is starting to look more and more like a crystal ball capable to predict the flight of flu epidemics, match winners and presidential election outcomes.

Not many technologies are capable of processing as much data, as frequently and actually make sense of it all. Search was fed on big data, grew with artificial intelligence and, if some say it is not rocket science, it is verging towards science fiction.

Brains in a box.

Historically search engines were indexing a web of documents to point searchers in the most relevant direction when they tapped a couple of keywords in a search box. Coping with the expansion and diversification of that universe was no small task, neither was the treatment of the ever increasingly more complex queries. So the technology had to gain in sophistication.

Algorithms started to extrapolate the strings of characters that were inputted. It became less about finding a specific phrase, and more about understanding its meaning. It was an evolution dictated by necessity. First, human beings are so prone to mistyping that machines could not rely on us ; second a same need can be expressed by different synonyms; and third, words have several meanings based on their context (e.g. from a PC, searching for “coffee” may relate to coffee harvesting whilst the same person using the same keyword on his smartphone may be after a caffeine shot).

This led to new functionality like query suggestions, auto-correction, auto-fill, semantic search… but also drastic evolutions of the algorithms with the integration of social, geographical or device signals. Search was no longer literal; it had become contextual.

From smart to intelligent
However, searchers still require to proactively engage with a user interface in order to trigger queries. These interactions remain contrived, even if you consider the conversational nature of voiced queries. Search will only truly become intelligent when the engine can anticipate what I need, even before I verbalise that intent. That is one of the promises of digital personal assistant like Cortana who relies on Bing information architecture and machine-learning to anticipate your needs. One of my favourites is her ability to urge me when to leave for my next appointment by making sense of my current location and the traffic conditions to my destination.

So could we take anticipation to the next level and predict the future.
Search engines are a database of intent where millions of people converge to look for information of what is top of mind for them. At the same time, social networks are the depository of sentiments. If you have developed the ability to process, analyse and understand these two humongous, historical and real-time information sets you have the opportunity to discover user sentiment for certain events or entities, estimate popularity trends, as well as predict outcomes of future events.

Bing Predict explored that concept with popularity-based contests like American Idol, for which web and social signals can highly correlate with popularity voting patterns and thus allows the engine to accurately project who will be eliminated each week and who the eventual winner will be. At the other end of the spectrum, predicting the outcome of the World Cup, Tour de France or the Premier League requires the incorporation of player/team stats, tournament trends and game history, location, and data from social channels.

The data from social channels provides the Bing model with the “wisdom of the crowd.” This approach is different from predictions for popularity-based contests. That model is able to interpret specific data as priority information such as team strengths, as popularity alone doesn’t dramatically help a team win or lose (some fans may object to this assertion but it’s largely true).

This machine-learned approach proved to be more reliable than traditional statistical methods on several occasions. Bing predicted accurately the Scottish Independence Referendum  outcome from the very first day whilst the official statistic was oscillating between the Yes and the No. Our predictions for each of the men’s and women’s Wimbledon matches had an average accuracy of 71 percent, and got the winners from the first serve. We also predicted Froome’s victory in the Tour de France.

What can brands learn from these forward-looking experiments?

Machine learning models are already making their way to the advertiser toolset. Bing Ads for instance includes an opportunity tab which allows brands to evaluate the future impact of actions taken on their search marketing campaigns based on auction and competitive behaviours. That is just a first step.

I have already written about how brands should think outside the (search) box , and harness the full potential of the search data to inform their marketing strategy. Think for instance about the evolution of the geographic spectrum of your search queries to inform your stock strategy for the next holiday season.

Next, businesses can enrich their own data with real-world, publically available data sets to identify further correlations. It can be a small collection of manually curated convention centre calendars which infer future influx of visitors to a city, or richer data sets from Open Data sites around the world .

It might take some creative thinking on your part to reveal true insights, but ignoring this resource means missing out on a big opportunity to create value for your company and customers. This is about modelling the real world in advertising campaigns with extra rigor and an opportunistic mind set thanks to the accessibility and democratisation of Business Intelligence tools, like PowerMap or Cortana Analytics .

Finally, I am convinced that soon enough new advertising models will come to fruition. Trajectory marketing, for instance, would consist in geo-targeting consumers based on the location they will be at rather than the location they are, by modelling their current position, their celerity, external factors like traffic, weather conditions, etc.  After all, marketing is about seeding the right message to the right audience, at the right time.

And that time is in the near future.