THE SQL Server Blog Spot on the Web

Welcome to SQLblog.com - The SQL Server blog spot on the web Sign in | |
in Search

Jamie Thomson

This is the blog of Jamie Thomson, a freelance data mangler in London

Cortana and Schema.org

Some definitions:

  • Cortana – Microsoft’s digital personal assistant, competing with Apple’s Siri and Google Now.
    “She gets to know you by learning your interests over time. She looks out for you, providing proactive, useful recommendations. And Cortana keeps you closer to the people and things you care about most, by keeping track of all that matters.” - http://www.windowsphone.com/en-US/features-8-1#Cortana
  • Schema.org – “a collection of schemas that used to markup HTML pages, and that can also be used for structured data interoperability Search engines rely on this markup to improve the display of search results” https://schema.org/.
    I think of schema.org as machine-readable metadata on a web page. This has existed for years in various guises (e.g. microformats), schema.org is a company-independent initiative to standardise those schemas.

One of the interesting features that Cortana provides is the ability to parse your email in order to discover flight information and then keep you up to date with information about that flight. Upon reading MSDN article Sending flight information to Microsoft Cortana with contextual awareness it transpires that this feature is entirely dependent upon the email containing schema.org markup:

Microsoft Cortana interprets schema.org markup in e-mails to extract airline flight reservation data
http://msdn.microsoft.com/en-us/library/dn632191.aspx

We all unknowingly make use use of http://schema.org markup all the time (every time you use Google for example, the share contract in Windows 8 & Windows Phone 8.1 can also leverage schema.org) however this is the first time I’ve been made aware of a user experience that is totally dependent on it.

I find this fascinating, I’ve long had an interest in structured metadata on the web (I’ve been banging on about it in blog posts for the last 10 years) and now my mind is in overdrive thinking of other scenarios that could leverage this. Taking a look at the documentation shows us that schema.org provides definitions of a multitude of things (in fact Thing is the base class from which everything else is derived), some examples:

Its not hard to envisage that Cortana might one day see mention of a music track in an email and offer the ability to play it for you, or see contact information and offer to add that to your contact list. Where this gets potentially even more interesting (to me, anyway) is when we consider that Cortana is already extensible by 3rd party apps – I foresee that in the future 3rd party apps will have the ability to subscribe to a particular schema and have Cortana notify them when it comes across an instance of that schema (imagine a Spotify app being informed that an email just arrived with information about a new music release and then offering to play that for you).

Perhaps this 3rd party extensibility could work the other way too by presenting other data sources to Cortana. For example, if Twitter ever get round to implementing annotations then Cortana could potentially read metadata hidden within tweets as well as combing through emails.

Expect that in the future Cortana will parse new data sources, find new types of information and allow more 3rd parties to act upon it. The potential for a degraded user experience with notifications flashing in your face all the time is a worry but frankly I don’t care, this is all fascinating stuff to a data geek like me.

@Jamiet 

P.S. Thank you to Savas Parastatidis from the Cortana dev team for highlighting Cortana’s use of schema.org.

Published Thursday, May 01, 2014 10:34 AM by jamiet
Filed under: ,

Comment Notification

If you would like to receive an email when updates are made to this post, please register here

Subscribe to this post's comments using RSS

Comments

 

Savas Parastatidis said:

Yup!

Also have a look at the recently released schema.org actions. Even though I think there is still work to do in the representation of actions, you can imagine the possibilities for 3rd party service integration... "call me a taxi", "share this photo", "post this video", "deserve a table for two", ...

Not saying that this is what we are doing... Just imagine the possibilities.

May 1, 2014 10:33 AM
 

jamiet said:

Hi Savas,

Yeah I was googling (whoops, forgot who you work for, binging :) ) around this morning while writing this and saw a lot of people talking about actions. Definitely warrants a bit more investigation.

"Not saying that this is what we are doing"

I'll assume that you're definitely not not doing it :)

JT

May 1, 2014 11:02 AM

Leave a Comment

(required) 
(required) 
Submit

This Blog

Syndication

Powered by Community Server (Commercial Edition), by Telligent Systems
  Privacy Statement