Architecture and processes for computer learning and understanding

• US 10,621,285 B2
• Filed: 03/20/2017
• Issued: 04/14/2020
• Est. Priority Date: 06/24/2016
• Status: Active Grant

• Current Assignee: Mind Lakes, LLC
• Original Assignee: Elemental Cognition LLC
• Inventors: Ferrucci, David, Barborak, Mike, Buchanan, David, Burnham, Greg, Chu-Carroll, Jennifer, Kalyanpur, Aditya, Lally, Adam, Pacifico, Stefano, Wang, Chang
• Primary Examiner(s): He, Jialong
• Application Number:
  US15/464,076
• Publication Number:
  US 20200034423A1
• Time in Patent Office: 1,121 Days
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 16/3331 Query processing
• CPC Class Codes: G06F 40/211 Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 40/35 Discourse or dialogue representation
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/006 based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation
• CPC Class Codes: G06N 5/027 Frames
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

An architecture and processes enable computer learning and developing an understanding of arbitrary natural language text through collaboration with humans in the context of joint problem solving. The architecture ingests the text and then syntactically and semantically processes the text to infer an initial understanding of the text. The initial understanding is captured in a story model of semantic and frame structures. The story model is then tested through computer generated questions that are posed to humans through interactive dialog sessions. The knowledge gleaned from the humans is used to update the story model as well as the computing system'"'"'s current world model of understanding. The process is repeated for multiple stories over time, enabling the computing system to grow in knowledge and thereby understand stories of increasingly higher reading comprehension levels.

Claims

1. A computer-implemented method, comprising:

forming, by a computing system, a semantic representation of a natural language story, the semantic representation using knowledge stored in a current world model that expresses beliefs about how natural language is understood, wherein the semantic representation includes multiple generative semantic primitive (GSP) structures, the multiple GSP structures include one or more predicates and one or more roles that indicate one or more beliefs regarding an understanding of the natural language story, wherein the current world model is developed over time through processing of multiple natural language stories;

generating, by the computing system, one or more questions to evaluate the semantic representation;

maintaining a dependency structure for linear dialogs to identify questions that are independent from one another;

submitting, from the computing system to a separate user computing device, the one or more questions for presentation to a human user;

receiving, by the computing system from the separate user computing device, one or more responses indicative of input from the human user when answering the one or more questions;

iterating through the dependency structure based at least in part on the one or more responses to identify a set of next questions and branch conditions indicating that at least two next questions of the set of next questions are independent of one another;

learning, by the computing system, at least one new GSP structure that indicates a new belief regarding the understanding of the natural language story, the new belief having an above threshold probability of being included in the multiple GSP structures that is associated with a theme or a context of the natural language story based at least in part on the one or more responses received from the separate user computing device; and

revising, by the computing system, the semantic representation of the natural language story based at least in part on the one or more responses received from the separate user computing device and the at least one new GSP structure.

2. The computer-implemented method of claim 1, wherein forming the semantic representation comprises:

performing a syntactic analysis of the natural language story to produce one or more linguistic analysis results; and

creating multiple semantic structures to represent the natural language story in part by using the one or more linguistic analysis results and knowledge induced from large language corpora, wherein creating the multiple semantic structures includes creating the multiple GSP structures by defining the one or more predicates and the one or more roles for the multiple GSP structures.

3. The computer-implemented method of claim 2, wherein the one or more questions pertain to the one or more linguistic analysis results.

4. The computer-implemented method of claim 2, wherein the one or more questions pertain to an ability of the multiple semantic structures to represent the natural language story.

5. The computer-implemented method of claim 1, wherein generating the one or more questions comprises:

building the dependency structure to provide a dialog context for interpreting and answering a question, the dependency structure being derived from data in the semantic representation; and

forming the one or more questions in a natural language.

6. The computer-implemented method of claim 5, wherein the dependency structure comprises a tree-type data structure with root nodes and branches to children nodes that can be logically traversed to ask opening and follow-on questions.

7. The computer-implemented method of claim 1, wherein submitting the one or more questions comprises sending the one or more questions in a format that can be presented visually on the separate user computing device in a graphical user interface.

8. The computer-implemented method of claim 7, wherein the graphical user interface comprises a first area having a dialog interface to represent a dialog between the human user and the computing system and a second area adjacent to the first area, the second area having a story aid interface that depicts a portion of the natural language story and an associated understanding of the portion.

9. The computer-implemented method of claim 1, further comprising updating the current world model with information pertaining to the semantic representation as revised.

10. A computer-implemented method, comprising:

processing, over time by a computing system, multiple natural language stories to produce a current world model that includes a collection of generative semantic primitive (GSP) structures to express beliefs about how natural language is understood, wherein a GSP structure of the collection of GSP structures includes a predicate and one or more roles, wherein the collection of GSP structures is associated with a theme or a context of the multiple natural language stories;

maintaining the current world model in a datastore of the computing system;

receiving, by the computing system, a natural language story;

maintaining a dependency structure for linear dialogs to identify questions that are independent from one another;

engaging in a dialog session with a human user to receive input from the human user for use by the computing system to evaluate an extent to which the beliefs of the current world model produce an understanding of the natural language story, wherein engaging in the dialog session includes iterating through the dependency structure based at least in part on the input to identify a set of next questions; and

revising, by the computing system, the current world model by adding at least one new GSP structure with at least one of a new predicate or one or more new roles based at least in part on the input from the human user and the at least one new GSP structure having an above threshold probability of being included with a particular collection of GSP structures that is associated with a particular theme or a particular context of the natural language story.

11. The computer-implemented method of claim 10, wherein engaging in a dialog session comprises:

generating, by the computing system, one or more questions to evaluate the beliefs;

submitting, from the computing system to a separate user computing device, the one or more questions for presentation to the human user; and

receiving, by the computing system from the separate user computing device, one or more responses indicative of the input from the human user when answering the one or more questions.

12. The computer-implemented method of claim 11, wherein generating the one or more questions comprises:

building the dependency structure to provide a dialog context for interpreting and answering a question, the dependency structure being derived from data pertaining to the beliefs; and

forming the one or more questions in a natural language.

13. The computer-implemented method of claim 12, wherein the dependency structure comprises a tree-type data structure with root nodes and branches to children nodes that can be logically traversed to ask opening and follow-on questions.

14. A computing system, comprising:

a datastore containing a current world model that expresses beliefs about how natural language is understood;

one or more processors to access the current world model maintained in the datastore; and

memory coupled to the one or more processors, the memory storing computer executable instructions that, when executed by the one or more processors, perform acts comprising;

processing multiple natural language stories over time, the processing including performing syntactic analysis on the multiple natural language stories and subsequently inferring semantic structures as representations of the multiple natural language stories, in part by using information maintained in the current world model and results from the syntactic analysis, wherein inferring the semantic structures includes creating multiple generative semantic primitive (GSP) structures by defining a predicate and one or more roles for at least one GSP structure of the multiple GSP structures, wherein the processing includes;

processing, over a first time period, a first story of the multiple natural language stories having a first reading comprehension level;

inferring one or more first semantic structures as representations of the first story;

processing, over a second time period after the first time period, a second story of the multiple natural language stories having a second reading comprehension level that is more difficult than the first reading comprehension level; and

inferring at least one second semantic structure as a representation of the second story based at least in part on expanding at least one of the one or more first semantic structures to include a new semantic structure;

maintaining a dependency structure for linear dialogs to identify questions that are independent from one another;

conducting dialog sessions with multiple different human users to aggregate multiple responses from the multiple different human users to evaluate the semantic structures as understandings of the multiple natural language stories, wherein evaluating the semantic structures includes at least one of adding a new GSP structure or redefining at least one of the predicate or the one or more roles for an existing GSP structure; and

modifying the current world model in the datastore over time to include the semantic structures inferred from the multiple natural language stories and evaluated by the multiple different human users, the semantic structures including the one or more first semantic structures inferred during or after the first time period and the at least one second semantic structure inferred during or after the second time period.

15. The computing system of claim 14, wherein conducting the dialog sessions comprises:

generating, by the computing system, multiple questions to evaluate the semantic structures;

submitting, from the computing system to separate multiple user computing devices, the multiple questions for presentation to the multiple different human users;

receiving, by the computing system from the separate multiple user computing devices, multiple responses indicative of input from the multiple different human users when answering the multiple questions; and

iterating through the dependency structure based at least in part on the multiple responses to identify a set of next questions and branch conditions indicating that at least two next questions of the set of next questions are independent of one another.

16. The computing system of claim 15, wherein generating the multiple questions comprises:

building the dependency structure to provide a dialog context for interpreting and answering a question, the dependency structure being derived from data pertaining to the beliefs; and

forming the multiple questions in a natural language.

17. The computing system of claim 16, wherein the dependency structure comprises a tree-type data structure with root nodes and branches to children nodes that can be logically traversed to ask opening and follow-on questions.

18. A computing system, comprising:

one or more processors; and

memory coupled to the one or more processors, the memory storing computer-executable modules comprising;

a knowledge integration engine to form semantic structures that provide a semantic representation of a natural language story, wherein the semantic structures include generative semantic primitive (GSP) structures defined with predicates and roles to express a first understanding of the natural language story; and

a dialog engine to facilitate a dialog session with a human user to evaluate an extent to which the semantic structures represent the natural language story by generating and submitting one or more questions to a separate user computing device used by the human user and collecting one or more responses from the separate user computing device indicative of input from the human user, wherein the dialog engine includes a dialog dispatcher to break up the dialog session into sub-dialog sessions and maintains a dependency structure for linear dialogs to identify questions that are independent from one another, wherein the dialog dispatcher includes a structure crawler to iterate through the dependency structure based on the one or more responses to identify a set of next questions; and

wherein the knowledge integration engine updates the semantic structures to include at least one of adding a new GSP structure or redefining an existing GSP structure to express a second understanding of the natural language story based in part on the one or more responses, wherein adding the new GSP structure or redefining the existing GSP structure increases the extent to which the semantic structures represent the natural language story.

19. The computing system of claim 18, wherein the knowledge integration engine is configured to form the semantic structures in part by using one or more linguistic analysis results to form a syntactic analysis of the natural language story and knowledge induced from large language corpora.

20. The computing system of claim 18, wherein the dialog engine submits the one or more questions in a format that can be presented visually on the separate user computing device in a graphical user interface.

21. The computing system of claim 18, wherein the dialog engine generates the one or more questions by building the dependency structure to provide a dialog context for interpreting and answering a question, the dependency structure being derived from data in the semantic representation.