Intelligent contextually aware digital assistants

• US 9,542,648 B2
• Filed: 04/10/2014
• Issued: 01/10/2017
• Est. Priority Date: 04/10/2014
• Status: Active Grant

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Palo Alto Research Center, Inc. (Xerox Holdings Corp.)
• Inventors: Roberts, Michael
• Primary Examiner(s): JACKSON, JAKIEDA R
• Application Number:
  US14/250,322
• Publication Number:
  US 20150293904A1
• Time in Patent Office: 1,006 Days
• Field of Search: 704/9
• US Class Current: 1/1
• CPC Class Codes: G06F 16/90332 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 40/35 Discourse or dialogue representation
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation
• CPC Class Codes: G10L 15/1815 Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 2015/226 using non-speech characteristics

Abstract

One embodiment of the present invention provides a system for providing context-based web services for a user. During operation, the system receives a sentence as input from a user. The system performs natural language processing on the sentence to determine one or more parameters. The system retrieves data from a foreground knowledge graph containing contextual data for the user and from a background knowledge graph containing background information corresponding to the parameters. The system determines a set of arguments based on the parameters and/or data from the foreground knowledge graph and/or data from the background knowledge graph. The system then selects an action module based on results of the natural language processing and/or the set of arguments. The system passes the arguments to the action module. The action module then uses the arguments to respond to a question or interact with web services to perform an action for the user.

Claims

1. A computer-executable method for providing context-based web services for a user, comprising:

receiving a sentence as input from a user interacting with a visual interface that includes an animated agent;

performing natural language processing on the sentence to determine one or more parameters;

retrieving data from a foreground knowledge graph that contains contextual data for the user and from a background knowledge graph that contains background information corresponding to the one or more parameters, wherein the background knowledge graph is different from the foreground knowledge graph;

determining a set of arguments based on the one or more parameters and data from the foreground and background knowledge graphs;

passing the set of arguments to an action module selected based on results of the natural language processing and the set of arguments;

using the set of arguments, by the selected action module, to interact with web services to perform an action for the user and provide a response to the user, wherein providing the response involves using a text-speech translator to produce audio for the response, using a viseme extractor to determine mouth positions of the animated agent for synchronous display with the audio, and animating the animated agent based on animation tags inserted into the response; and

converting general and domain-specific knowledge into modifications to the background knowledge graph, which involves obtaining a document set on a particular subject based on performing a web search, and modifying the background knowledge graph using results from analyzing the document set using a content analysis module and a semantic meaning extraction system.

2. The method of claim 1, wherein performing an action for the user further comprises completing an online sales transaction.

3. The method of claim 1, wherein performing natural language processing to determine one or more parameters further comprises:

determining a sentence structure of the sentence;

determining whether there is an entry in a database corresponding to the sentence structure;

responsive to determining that there is an entry in the database corresponding to the sentence structure, retrieving information from the entry in the database; and

extracting parameters from the sentence based on information retrieved from the database entry.

4. The method of claim 1, wherein performing natural language processing on the sentence to determine one or more parameters further comprises:

determining a sentence structure of the sentence;

determining whether there is an entry in a database corresponding to the sentence structure;

responsive to determining that there is no entry in the database corresponding to the sentence structure, engaging in a dialogue to elicit one or more parameters;

determining mapping of the one or more parameters to properties on an object; and

storing information that includes the mapping and the one or more parameters in a database.

5. The method of claim 1, wherein changes in the contextual data of the foreground knowledge graph triggers performing an action based on the user'"'"'s context.

6. The method of claim 1, further comprising:

adding contextual data to the foreground knowledge graph based on detected user activity and/or user communications;

disambiguating another input sentence that requires information from the background knowledge graph based on the contextual data from the foreground knowledge graph; and

performing another action for the user based at least on a portion of the contextual data added to the foreground knowledge graph and the information from the background knowledge graph.

7. The method of claim 1, wherein one or more modules perform parameterized queries and modifications on the foreground knowledge graph and the background knowledge graph.

8. A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method for providing context-based web services for a user, the method comprising:

receiving a sentence as input from a user interacting with a visual interface that includes an animated agent;

performing natural language processing on the sentence to determine one or more parameters;

retrieving data from a foreground knowledge graph that contains contextual data for the user and from a background knowledge graph that contains background information corresponding to the one or more parameters, wherein the background knowledge graph is different from the foreground knowledge graph;

determining a set of arguments based on the one or more parameters, data from the foreground and background knowledge graphs;

passing the set of arguments to an action module selected based on results of the natural language processing and the set of arguments;

using the set of arguments, by the selected action module, to interact with web services to perform an action for the user and provide a response to the user,

wherein providing the response involves using a text-speech translator to produce audio for the response, using a viseme extractor to determine mouth positions of the animated agent for synchronous display with the audio, and animating the animated agent based on animation tags inserted into the response; and

converting general and domain-specific knowledge into modifications to the background knowledge graph, which involves obtaining a document set on a particular subject based on performing a web search, and modifying the background knowledge graph using results from analyzing the document set using a content analysis module and a semantic meaning extraction system.

9. The non-transitory computer-readable storage medium of claim 8, wherein performing an action for the user further comprises completing an online sales transaction.

10. The non-transitory computer-readable storage medium of claim 8, wherein performing natural language processing to determine one or more parameters further comprises:

determining a sentence structure of the sentence;

determining whether there is an entry in a database corresponding to the sentence structure;

responsive to determining that there is an entry in the database corresponding to the sentence structure, retrieving information from the entry in the database; and

extracting parameters from the sentence based on information retrieved from the database entry.

11. The non-transitory computer-readable storage medium of claim 8, wherein performing natural language processing on the sentence to determine one or more parameters further comprises:

determining a sentence structure of the sentence;

determining whether there is an entry in a database corresponding to the sentence structure;

responsive to determining that there is no entry in the database corresponding to the sentence structure, engaging in a dialogue to elicit one or more parameters;

determining mapping of the one or more parameters to properties on an object; and

storing information that includes the mapping and the one or more parameters in a database.

12. The non-transitory computer-readable storage medium of claim 8, wherein changes in the contextual data of the foreground knowledge graph triggers performing an action based on the user'"'"'s context.

13. The non-transitory computer-readable storage medium of claim 8, wherein the method further comprises:

adding contextual data to the foreground knowledge graph based on detected user activity and/or user communications;

disambiguating another input sentence that requires information from the background knowledge graph based on the contextual data from the foreground knowledge graph; and

performing another action for the user based at least on a portion of the contextual data added to the foreground knowledge graph and the information from the background knowledge graph.

14. The non-transitory computer-readable storage medium of claim 8, wherein one or more modules perform parameterized queries and modifications on the foreground knowledge graph and the background knowledge graph.

15. A computing system for providing context-based web services for a user, the system comprising:

one or more processors,

a non-transitory computer-readable medium coupled to the one or more processors having instructions stored thereon that, when executed by the one or more processors, cause the one or more processors to perform a method, the method comprising;

receiving a sentence as input from a user interacting with a visual interface that includes an animated agent;

performing natural language processing on the sentence to determine one or more parameters;

retrieving data from a foreground knowledge graph that contains contextual data for the user and from a background knowledge graph that contains background information corresponding to the one or more parameters, wherein the background knowledge graph is different from the foreground knowledge graph;

determining a set of arguments based on the one or more parameters and data from the foreground and background knowledge graphs;

passing the set of arguments to an action module selected based on results of the natural language processing and the set of arguments;

using the set of arguments, by the selected action module, to interact with web services to perform an action for the user and provide a response to the user,

wherein providing the response involves using a text-speech translator to produce audio for the response, using a viseme extractor to determine mouth positions of the animated agent for synchronous display with the audio, and animating the animated agent based on animation tags inserted into the response; and

converting general and domain-specific knowledge into modifications to the background knowledge graph, which involves obtaining a document set on a particular subject based on performing a web search, and modifying the background knowledge graph using results from analyzing the document set using a content analysis module and a semantic meaning extraction system.

16. The computing system of claim 15, wherein performing an action for the user further comprises completing an online sales transaction.

17. The computing system of claim 15, wherein performing natural language processing to determine one or more parameters further comprises:

determining a sentence structure of the sentence;

determining whether there is an entry in a database corresponding to the sentence structure;

responsive to determining that there is an entry in the database corresponding to the sentence structure, retrieving information from the entry in the database; and

extracting parameters from the sentence based on information retrieved from the database entry.

18. The computing system of claim 15, wherein performing natural language processing on the sentence to determine one or more parameters further comprises:

determining a sentence structure of the sentence;

determining whether there is an entry in a database corresponding to the sentence structure;

responsive to determining that there is no entry in the database corresponding to the sentence structure, engaging in a dialogue to elicit one or more parameters;

determining mapping of the one or more parameters to properties on an object; and

storing information that includes the mapping and the one or more parameters in a database.

19. The computing system claim 15, wherein changes in the contextual data of the foreground knowledge graph triggers performing an action based on the user'"'"'s context.

20. The computing system of claim 15, wherein the method further comprises:

adding contextual data to the foreground knowledge graph based on detected user activity and/or user communications;

disambiguating another input sentence that requires information from the background knowledge graph based on the contextual data from the foreground knowledge graph; and

performing another action for the user based at least on a portion of the contextual data added to the foreground knowledge graph and the information from the background knowledge graph.

21. The computing system of claim 15, wherein one or more modules perform parameterized queries and modifications on the foreground knowledge graph and the background knowledge graph.