Intelligent capture, storage, and retrieval of information for task completion

• US 10,409,876 B2
• Filed: 05/26/2016
• Issued: 09/10/2019
• Est. Priority Date: 05/26/2016
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Vinnakota, Rajasekhar, Gill, Deepinder Singh, Gupta, Ravi, Vangala, Vipindeep, Thirumalai-Anandanpillai, Srinivasa Varadhan
• Primary Examiner(s): Spieler, William
• Application Number:
  US15/166,182
• Publication Number:
  US 20170344649A1
• Time in Patent Office: 1,202 Days
• CPC Class Codes: G06F 16/2465 Query processing support for facilitating data mining operations in structured databases
• CPC Class Codes: G06F 16/9024 Graphs; Linked lists G06F16/9027 takes precedence
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation

Abstract

Intelligent capture, storage, and retrieval of information for task completion are provided. A user can explicitly request to capture information, or user information is implicitly captured from one or more data streams. Semantic understanding of captured information is provided, wherein offline processes are run that analyze captured information to bucket them into a set of pre-defined high value scenarios. Whenever a capture falls into such scenarios, additional structured information about the capture is extracted. Further, aspects provide for storing and indexing information in a data repository. Users are enabled to explicitly retrieve the stored information using a natural language interface, or stored information can be proactively provided to the user based on context to enable task completion.

Claims

1. A method for analyzing captured information for task completion, the method comprising:

analyzing a captured information item associated with a user to identify an entity and properties associated with the captured information item;

storing the entity and the properties associated with the captured information item in a knowledge graph;

enriching the knowledge graph, wherein enriching the knowledge graph comprises;

identifying relationships between the entity and the properties,

identifying other entities associated with the entity, and

representing the identified relationships and the identified other entities in the knowledge graph;

based on the identified relationships and the identified other entities of the enriched knowledge graph, identifying a plurality of actions, each action associated with one or more of the entity, a type of the entity, one or more of the identified other entities, and a type of one or more of the identified other entities;

prioritizing the identified actions based on an importance of each action to the user, the importance determined based on the user'"'"'s previous usage of and reactions to the respective action;

indexing the entity, the properties, and the prioritized actions in an index;

receiving an indication to query the index for the captured information item;

querying the index for the captured information item;

retrieving the captured information item including the entity, the properties, and the prioritized actions from the index; and

providing the captured information item for presentation to the user, wherein one or more of the identified actions are presented based on a respective priority.

2. The method of claim 1, wherein:

analyzing the captured information item comprises analyzing an implicitly captured information item; and

prior to analyzing the captured information item, mining through stored data for identifying an information item to capture and store for the user.

3. The method of claim 2, wherein mining through the stored data comprises mining through a cloud-based repository for the user including the user'"'"'s documents, photos, and email attachments.

4. The method of claim 3, wherein mining through the stored data for identifying the information item to capture and store for the user comprises analyzing optical character recognition metadata associated with the stored data.

5. The method of claim 2, wherein identifying the information item to capture and store for the user comprises determining whether the information item matches a predefined schema, wherein the predefined schema defines semantics of a particular capture type.

6. The method of claim 1, wherein:

analyzing the captured information item comprises analyzing an explicitly captured information item; and

prior to analyzing the captured information item;

receiving an indication of a request from the user to capture the information item;

performing natural language processing on the request to extract a key and value dataset; and

indexing the key and the value dataset in the index.

7. The method of claim 1, wherein prioritizing the identified actions based on the importance of each action to the user further comprises determining the importance based on collected data associated with the user or associated with collective users, the collected data selected from the group consisting of:

search logs;

browser logs; and

advertising logs.

8. The method of claim 1, wherein receiving the indication to query the index for the captured information item comprises receiving an explicit request for the captured information item via an interaction with an intelligent personal digital assistant through a natural language user interface or a graphical user interface.

9. The method of claim 1, wherein receiving the indication to query the index for the captured information item comprises receiving contextual data associated with the user for determining whether the index includes one or more captured information items that can be proactively surfaced to the user based on the contextual data.

10. The method of claim 9, wherein querying the index for the captured information item comprises querying the index for captured information items that can be proactively surfaced to the user based on the contextual data.

11. The method of claim 1, wherein providing the captured information item for presentation to the user on an endpoint computing device for task completion comprises at least one of the following:

providing a response via a natural language user interface;

performing a task;

providing a service;

gathering information; and

operating the endpoint computing device.

12. A system for analyzing captured information for task completion, the system comprising:

a processing unit; and

a memory, including computer readable instructions, which when executed by the processing unit is operable to;

analyze a captured information item associated with a user to identify an entity and properties associated with the captured information;

store the entity and the properties in a knowledge graph;

enrich the knowledge graph, wherein the processing unit being operable to enrich the knowledge graph comprises the processing unit being operable to;

identify relationships between the entity and the properties,

identify other entities associated with the entity, and

represent the identified relationships and the identified other entities in the knowledge graph;

based on the identified relationships and the identified other entities of the enriched knowledge graph, identify a plurality of actions, each action associated with one or more of the entity, a type of the entity, one or more of the identified other entities, and a type of the identified other entities;

prioritize the identified actions based on an importance of each action to the user, the importance determined based on the user'"'"'s previous usage of and reactions to the respective action;

index the entity, the properties, and the prioritized actions in an index;

receive an indication to query the index for the captured information item;

query the index for the captured information item;

retrieve the captured information item including the entity, the properties, and the prioritized actions from the index; and

provide the captured information item for presentation to the user, wherein one or more of the identified actions are presented based on a respective priority.

13. The system of claim 12, wherein:

the captured information is an implicitly captured information item; and

prior to analyzing the captured information item, the processing unit is operative to mine through stored data for identifying the information item to capture and store for the user, the stored data including data stored in a cloud-based repository, the repository including the user'"'"'s documents, photos, and email attachments.

14. The system of claim 13, wherein in mining through the stored data for identifying the information item to capture and store for the user, the processing unit is operative to analyze optical character recognition metadata associated with the stored data.

15. The system of claim 12, wherein:

the captured information item is an explicitly captured information item; and

prior to analyzing the captured information item, the processing unit is further operative to;

receive an indication of a request from the user to capture the information item;

perform natural language processing on the request to extract a key and value dataset; and

index the key and the value dataset in the index.

16. The system of claim 12, wherein the processing unit is further operative to prioritize the identified actions based on the importance of each action to the user by determining the importance based on collected data associated with the user or associated with collective users, the collected data selected from the group consisting of:

search logs;

browser logs; and

advertising logs.

17. The system of claim 12, wherein in receiving the indication to query the index for the captured information item, the processing unit is further operative to receive contextual data associated with the user for determining whether the index includes one or more captured information items that can be proactively surfaced to the user based on the contextual data.

18. A computer readable storage device including computer readable instructions, which when executed by a processing unit is operable to:

analyze a captured information item associated with a user to identify an entity and properties associated with the captured information item;

store the entity and the properties in a knowledge graph;

enrich the knowledge graph, wherein the processing unit being operable to enrich the knowledge graph comprises the processing unit being operable to;

identify relationships between the entity and the properties,

identify other entities associated with the entity, and

represent the identified relationships and the identified other entities in the knowledge graph;

based on the identified relationships and the identified other entities of the enriched knowledge graph, identify a plurality of actions, each action associated with one or more of the entity, a type of the entity, one or more of the identified other entities, and a type of the identified other entities;

prioritize the identified actions based on an importance of each action to the user, the importance determined based on the user'"'"'s previous usage of and reactions to the respective action;

index the entity, the properties, and the prioritized actions in an index;

receive an indication to query the index for the captured information item;

query the index for the captured information item;

retrieve the captured information item including the entity, the properties, and the prioritized actions from the index; and

provide the captured information item for presentation to the user, wherein one or more of the identified actions are presented based on a respective priority.

19. The computer readable storage device of claim 18, wherein:

when the captured information item is an implicitly captured information item, prior to analyzing the captured information item, the processing unit is operable to mine through stored data for identifying the information item to capture and store for the user, the stored data including data stored in a cloud-based repository, the repository including the user'"'"'s documents, photos, and email attachments; and

when the captured information item is an explicitly captured information item, prior to analyzing the captured information item, the processing unit is operable to;

receive an indication of a request from the user to capture the information item;

perform natural language processing on the request to extract a key and value dataset; and

index the key and the value dataset in the index.

20. The computer readable storage device of claim 18, wherein in receiving the indication to query the index for the captured information item, the processing unit is further operative to receive contextual data associated with the user for determining whether the index includes one or more captured information items that can be proactively surfaced to the user based on the contextual data.