Contextual search using natural language

• US 10,509,829 B2
• Filed: 10/05/2015
• Issued: 12/17/2019
• Est. Priority Date: 01/21/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Rambhia, Amy, Syed, Harris, Doran, Ed
• Primary Examiner(s): Nguyen, Merilyn P
• Application Number:
  US14/875,143
• Publication Number:
  US 20160210363A1
• Time in Patent Office: 1,534 Days
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06F 9/453 Help systems
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 17/00 Speaker identification or verification techniques
• CPC Class Codes: G10L 17/22 Interactive procedures; Man-machine interfaces
• CPC Class Codes: H04L 67/10 in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46

Abstract

A context aware digital assistant supported on devices such as smartphones, tablet computers, wearable computing devices, personal computers (PCs), game consoles, and the like is configured with a natural language interface that enables a user to launch searches for content using contextual references such as time, date, event, location, schedule, activity, contacts, or device. The user can thus use natural language to express the context that is applicable to the sought-after content rather than having to formulate a query that uses a specific syntax. The digital assistant can comprehensively search for the content across applications (i.e., both first and third party applications), devices, and services.

Claims

1. A method performed on a device of a plurality of devices associated with a device user for facilitating a search for content using contextual references, comprising:

monitoring user interactions with the device of the plurality of devices;

identifying content exposed by the device based on the monitored user interactions;

using the identified content from the monitored user interactions, determining contextual reference tags for respective pieces of content, the contextual reference tags indicating that the respective pieces of content were exposed on the device of the plurality of devices;

applying and associating the determined contextual reference tags to the respective pieces of content, the contextual reference tags including an identification of specific devices on which the respective pieces of content were exposed;

exposing a user interface supporting natural language inputs for the search from the device user;

receiving the natural language inputs from the device user through the user interface, wherein the natural language inputs include an identification of a device on which the device user accessed content;

parsing the received natural language inputs for contextual references, the contextual references including the identified device and one or more of time, date, event, location, schedule, activity, or contact;

performing the search for content using the contextual reference tags and the received contextual references; and

providing results of the search, the search results identifying content matching at least one or more of the parsed contextual references for the identified device.

2. The method of claim 1 further including providing the search results using the contextual reference tags.

3. The method of claim 1 in which the user interface is supported at least in part by a digital assistant operating on the device.

4. The method of claim 3 in which the digital assistant interacts with the user using a spoken voice over an audio interface supported on the device.

5. The method of claim 3 in which the digital assistant interacts with the user over a graphical user interface supported on the device and further including displaying the search results on the graphical user interface.

6. The method of claim 3, further including configuring the digital assistant, responsively to voice input, gesture input, or manual input for performing at least one of sharing contact information, sharing a stored contact, scheduling a meeting, looking at a user'"'"'s calendar, scheduling a reminder, making a call, operating a device, playing games, making purchases, taking notes, scheduling an alarm or wake-up reminder, sending a message, checking social media for updates, scraping a website, interacting with a search service, sharing or showing files, sending a link to a website, or sending a link to a resource.

7. The method of claim 1 in which the content is stored in one or more remote cloud-based stores or cloud-based services.

8. The method of claim 1 further including rank ordering the search results according to the contextual references.

9. The method of claim 1 further including enabling content identified in the search results to be available for consumption on the device.

10. The method of claim 1 in which the user interactions with the device include consuming media, accessing documents, or browsing the web using a web browser.

11. A device, comprising:

one or more processors;

a user interface (UI) configured to interact with a user of the device using graphics or audio; and

a memory device storing computer-readable instructions which, when executed by the one or more processors, perform a method to;

monitor user interactions with the device of a plurality of devices;

identify content exposed by the device based on the monitored user interactions;

using the identified content from the monitored user interactions, determine contextual reference tags for respective pieces of content, the contextual reference tags indicating that the respective pieces of content were exposed on the device of the plurality of devices;

apply and associate the determined contextual reference tags to the respective pieces of content, the contextual reference tags including an identification of specific devices on which the respective pieces of content were exposed;

expose a digital assistant on the device in which the digital assistant is configured for voice interactions with the user using the UI,

receive voice inputs from the device user, wherein the voice inputs include an identification of a device on which the user accessed content,

trigger a search for content using the contextual reference tags and the contextual references that are included in the voice inputs, the contextual references including the identified device, and

handle content that is identified in results from the search using the digital assistant.

12. The device of claim 11 further comprising displaying search results on the UI as an aggregated list.

13. The device of claim 11 further comprising using the digital assistant to provide search results using audio.

14. The device of claim 11 in which the digital assistant is configured to use a natural language interface to interact with the user using spoken words.

15. The device of claim 11 in which the contextual reference tags identify one or more of time, date, event, location, schedule, activity, contact, or device.

16. One or more computer-readable memory devices storing instructions which, when executed by one or more processors disposed in a cloud-based computer server, perform a method comprising the steps of:

receiving registrations from applications and services running on a plurality of local devices;

monitoring user interactions at the plurality of local devices with registered applications and services;

identifying content exposed by the plurality of local devices based on the monitored user interactions;

using the identified content from the monitored user interactions, determining contextual reference tags for respective pieces of content; and

tagging the identified content with the determined contextual reference tags so that the content is associated with a given contextual reference tag, the contextual reference tags including an identification of a device on which respective pieces of content were accessed and one or more of time, date, event, location, schedule, activity, contact, or device.

17. The one or more computer-readable memory devices of claim 16 further including receiving a search request from a user for content that matches one or more contextual reference tags.

18. The one or more computer-readable memory devices of claim 17 further including performing a search responsively to the request.

19. The one or more computer-readable memory devices of claim 18 further including transmitting the search results to one or more of the local devices.