Adaptive virtual intelligent agent

• US 9,189,742 B2
• Filed: 11/18/2014
• Issued: 11/17/2015
• Est. Priority Date: 11/20/2013
• Status: Active Grant

• Current Assignee: Justin London
• Original Assignee: Justin London
• Inventors: London, Justin
• Primary Examiner(s): Hill, Stanley K
• Application Number:
  US14/546,097
• Publication Number:
  US 20150142704A1
• Time in Patent Office: 364 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/90332 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 40/35 Discourse or dialogue representation
• CPC Class Codes: G06F 40/58 Use of machine translation, e.g. for multi-lingual retrieval, for server-side translation for client devices or for real-time translation
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/042 Knowledge-based neural networks; Logical representations of neural networks
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 5/04 Inference or reasoning models
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: G06Q 30/016 After-sales
• CPC Class Codes: G06Q 30/0613 Third-party assisted
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 15/193 Formal grammars, e.g. finite state automata, context free grammars or word networks
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue

Abstract

Embodiments of an adaptive virtual intelligent agent (“AVIA”) service are disclosed. It may include the functions of a human administrative assistant for an enterprise including customer support, customer relationship management, and fielding incoming caller inquiries. It also has multi-modal applications for the home through interaction with AVIA implemented in the home. It may engage in free-form natural language dialogs. During a dialog, embodiments maintain the context and meaning of the ongoing dialog and provides information and services as needed by the domain of the application. Over time, the service automatically extends its knowledge of the domain (as represented in the Knowledge Tree Graphs) through interaction with external resources. Embodiments can intelligently understand and converse with users using free-form speech without pre-programmed deterministic sequences of questions and answers, can dynamically determine what it needs to know to converse meaningfully with users, and knows how to obtain information it needs.

Claims

1. A system comprising:

a processor;

a memory module communicatively coupled to the processor;

a data structure template stored in the memory module, wherein;

the data structure template is a graph data structure comprising a first plurality of nodes; and

the first plurality of nodes are connected by a first plurality of edges;

machine readable instructions stored in the memory module that cause the system to perform at least the following when executed by the processor;

receive a voice-based input;

in response to receiving the voice-based input, generate a data structure instantiation based on the data structure template, wherein;

the data structure instantiation is a graph data structure comprising a second plurality of nodes; and

the second plurality of nodes are connected by a second plurality of edges;

access the data structure template to determine a prompt to provide to a speaker;

provide the prompt to the speaker;

receive an input in response to providing the prompt to the speaker;

add a node to the data structure instantiation that includes the input, wherein the node added to the data structure instantiation is connected to at least one of the second plurality of nodes by a first edge; and

connect the node added to the data structure instantiation with at least one of the first plurality of nodes of the data structure template by a second edge.

2. The system of claim 1, wherein the machine readable instructions stored in the memory module further cause the system to perform at least the following when executed by the processor:

configure the data structure instantiation based on at least one input received from a speech recognition component or a type of entity associated with the speaker session;

define a relationship between the at least one input received from the speech recognition component and one or more of the second plurality of nodes of the data structure instantiation;

transform the input into structured meaning by processing a natural language expression of the input in order to establish one or more relationships between one or more concepts of the natural language expression, wherein each of the one or more relationships includes a key-value pair, an attribute-value pair, a semantic frame, or a logical form;

trigger a method in the data structure instantiation in response to the input; and

determine a response to be provided to the user based on one or more of the second plurality of nodes of the data structure instantiation.

3. The system of claim 1, further comprising at least one data repository stored in the memory module, wherein the at least one data repository includes a customer-relationship-management (CRM) data repository, a business-specific data repository, and a remotely connected third-party data repository, wherein:

the CRM data repository includes set of user preferences and a set of user data;

the business-specific data repository includes one or more business-knowledge-items, wherein the one or more business-knowledge items include an employee directory, a product list, an inventory list, a pricing list, or a company profile, wherein the company profile includes an hours of operation list and a locations list; and

the machine readable instructions stored in the memory module further cause the system to perform at least the following when executed by the processor;

query the at least one data repository; and

add the node to the data structure instantiation based on the query of the at least one data repository.

4. The system of claim 1, wherein:

the data structure template includes a goods template, a services template, or a combined template;

the goods template includes a store template;

the services template includes a restaurant template, a medical template, a legal template, or an office template;

the combined template includes a hybrid of the goods template and the services template;

the store template includes a set of states for an inquiry and a method to check an item from an inventory stored in the business-specific data repository;

the restaurant template includes a set of states for an order placement and a method to order a menu item from a menu stored in the business-specific data repository;

the medical template includes a set of states about a patient and a method to schedule a medical procedure;

the legal template includes a set of states about a client and a method to determine a type of legal services required and a second method to prompt the client for information regarding the type of legal services required;

the office template includes a set of states about the speaker and a method to schedule an appointment in an electronic appointment book;

the service template includes a set of states about service offerings offered by the business stored in a the business-specific data repository;

existing pre-configured templates can be modified, adapted, tailored, or customized to the specific business using them; and

additional templates can be created.

5. The system of claim 1, wherein the machine readable instructions stored in the memory module further cause the system to perform at least the following when executed by the processor:

process the input using a remotely connected third party data repository or a remote World Wide Web resource to determine at least one synonym for the input when the relationship between the input and the one or more of the second plurality of nodes of the data structure instantiation cannot be determined;

determine if the relationship between the input and the data structure template can be established based on the synonym; and

add the input to a list of acceptable responses associated with a specific node of the data structure template or the data structure instantiation.

6. The system of claim 1, wherein the machine readable instructions stored in the memory module further cause the system to perform at least the following when executed by the processor:

determine whether any additional information is required to complete the specific data structure graph instantiation; and

iteratively generate a series of prompts to request the additional information from the speaker when it is determined that additional information is required.

7. The system of claim 1, wherein the machine readable instructions stored in the memory module further cause the system to perform at least the following when executed by the processor:

receive the input from the speaker;

determine a language associated with the input;

translate each response provided by the system to the speaker to the language spoken by the speaker before providing the response back to the speaker;

produce a written transcript of a speaker session;

store the written transcript in the CRM data repository associated with the speaker;

send the written transcript to the speaker;

send one or more computer machine outputs chosen from the group consisting of confirmations, alerts, uniform resource locator (URL) links, menus, advertisements, and promotions to the speaker;

update a CRM data repository associated with the speaker with each message sent to the speaker;

receive an original input;

match a speaker'"'"'s voice to a voice data repository; and

retrieve a previously created data structure instantiation associated with the speaker'"'"'s voice from a CRM data repository.

8. The system of claim 7, wherein the system is an interactive voice response system, wherein the machine readable instructions stored in the memory module further cause the system to perform at least the following when executed by the processor:

determine if the input is a data item, command, or a query;

determining whether the speaker is recognized;

if the speaker is not recognized, instantiate the data structure instantiation based on the data structure template;

if the input is a first data item, fill in the first data item in the data structure instantiation;

if the input is a first command, automatically trigger a method associated with a state stored in the data structure instantiation;

determine if the method associated with the first command has a predetermined, sufficient number of required data items filled in on the data structure instantiation to trigger the method; and

if the method cannot be executed, select and deliver a prompt from a phrase construct data repository to elicit one or more missing data items prior to triggering the method.

9. The system of claim 1, wherein the system is an interactive voice response system, wherein the machine readable instructions stored in the memory module further cause the system to perform at least the following when executed by the processor:

fill in at least one node in the data structure instantiation with the response;

define a contextual relationship between the response and at least one matched node to the response;

calculate and assign a weight to a relationship between the input and each node to which it is related, wherein a higher weight is assigned to the input if it has either previously occurred or has a high confidence match for contextual relevance to the node with which it is associated.

10. The system of claim 9, wherein the machine readable instructions stored in the memory module further cause the system to perform at least the following when executed by the processor:

analyze a plurality of specific data structure instantiations to determine if the input is more common than other inputs, and if the input is more common, determine if the system responded appropriately; and

increase the weight of the relationship between the input and its node if the system responded appropriately; and

decrease the weight of the relationship between the at least one input and its node if the system did not respond appropriately.

11. The system of claim 1, wherein:

the first plurality of nodes of the data structure template comprises one or more action nodes, one or more concept nodes, and one or more property nodes;

at least one of the one or more property nodes is a child of at least one of the one or more concept nodes;

the second plurality of nodes of the data structure instantiation includes a speaker information node, an events/actions node, and a business information/concepts node; and

the node added to the data structure instantiation is added as a child node of at least one of the speaker information node, the events/actions node, or the business information/concepts node.

12. The system of claim 1, wherein the machine readable instructions stored in the memory module that further cause the system to perform at least the following when executed by the processor:

determine a time that the node was added to the data structure;

associate a timestamp that includes the time with the first edge; and

traverse the second plurality of nodes of the data structure instantiation in forward or reverse sequential time order based on a plurality of timestamps to access specific data items or to analyze historical context of the conversation.

13. A system comprising:

a processor;

a memory module communicatively coupled to the processor;

a plurality of data structure templates stored in the memory module, wherein;

each of the plurality of data structure templates is a graph data structure comprising a first plurality of nodes; and

the first plurality of nodes are connected by a first plurality of edges;

a plurality of data structure instantiations stored in the memory module, wherein;

each of the plurality of data structure instantiations is a graph data structure comprising a second plurality of nodes; and

the second plurality of nodes are connected by a second plurality of edges;

machine readable instructions stored in the memory module that cause the system to perform at least the following when executed by the processor;

receive an input from a speaker;

determining whether the speaker is recognized;

if the speaker is recognized;

selecting a data structure instantiation from the plurality of data structure instantiations, wherein the selected data structure instantiation was previously associated with the recognized speaker; and

determining a prompt having the highest likelihood of being relevant to the speaker session based on one or more nodes in the selected data structure instantiation; and

if the speaker is not recognized;

determining a type of business associated with the speaker session;

selecting a data structure template of the plurality of data structure templates that is associated with the speaker session; and

determining a prompt having the highest likelihood of being relevant to the speaker session based on one or more nodes in the plurality of data structure instantiations.

14. The system of claim 13, wherein the machine readable instructions stored in the memory module further cause the system to perform at least the following when executed by the processor:

determine a language associated with the input; and

translate subsequent input into a system default language when the determined language is not the system default language; and

translate output from the system to the determined language when the determined language is not the system default language.

15. The system of claim 14, wherein the machine readable instructions stored in the memory module further cause the system to perform at least the following when executed by the processor:

perform historical context analysis to calculate if other speakers are speaking with regard to a specific reason by analyzing the plurality of data structure instantiations, across a plurality of speakers, for a defined period of time, and if a predetermined threshold is reached increase a weight, associated with a node corresponding to that specific reason in at least one of the plurality of data structure instantiations.

16. A method of interactively responding to voice comprising:

receiving a voice-based input;

identifying a speaker associated with the voice-based input;

if the speaker is recognized, selecting a first data structure instantiation associated with the speaker from a memory module, wherein the first data structure instantiation includes a first plurality of nodes connected by a first plurality of edges;

if the speaker is not recognized, selecting a data structure template from the memory module, wherein the selected data structure template includes a third plurality of nodes connected by a third plurality of edges, and creating a second data structure instantiation that is associated with the speaker based on the selected data structure template, wherein the second data structure instantiation includes a second plurality of nodes connected by a second plurality of edges;

determining a phrase construct having the highest likelihood of being relevant to the speaker based on analysis of one or more contexts chosen from the first data structure instantiation associated with the speaker or a plurality of other data structure instantiations stored in the memory module; and

prompting the user with the phrase construct.

17. The method of claim 16, further comprising:

if the speaker is not recognized and the second data structure instantiation is created;

adding a first data item from a business-specific data repository to a first node of the second data structure instantiation;

prompting the speaker for a second data item;

receiving the second data item; and

adding the second data item to a second node of the second data structure instantiation.

18. The method of claim 16, further comprising:

receiving a response in response to prompting the user with the phrase construct;

determining whether the response is a data item, a command, or a query;

if the response is a data item, adding the data item to a first node of the data structure instantiation, prompting the speaker for a second data item, receiving a second data item, and adding the second data item to a second node of the data structure instantiation;

if the response is a command, checking the data structure instantiation to determine if a predetermined number of nodes have been filled in to execute the command; and

if an insufficient number of nodes have been filled in, iteratively generating additional prompts to complete the data structure instantiation sufficiently to execute the command; and

if the response is a query;

checking whether the query is related in context to a dialog;

if the query is context-related, checking the data structure instantiation to determine if a node contains a responsive answer; and

if there is no responsive answer, generating a responsive prompt.

19. The method of claim 18, further comprising:

if any of the responses received are not recognized,

processing an unrecognized response to determine a context and meaning for the unrecognized response;

adding a node to the data structure instantiation and populating the node with the unrecognized response;

evaluating a plurality of data structure instantiations and increasing a weight associated with an added node each time it appears in a data structure instantiation; and

if a given added node reaches a predetermined threshold, adding a prompt to a data repository of phrase constructs.

20. The method of claim 16, wherein processing an unrecognized response includes:

accessing a third party data repository to determine a category or a synonym for the unrecognized response and attempting to find a node associated with the category or the synonym;

or

searching one or more data structure instantiations stored in a memory module for a relationship between the unrecognized response and a concept.

21. The method of claim 16, further comprising:

receiving a set of inputs;

associating the set of inputs with one or more of the second plurality of nodes of the second data structure instantiation, wherein the second data structure instantiation is instantiated based upon a home automation template comprising a set of states relating to a set of appliances and a set of commands for controlling the set of appliances, wherein the set of commands comprises temperature adjustment commands, lighting commands, security commands, and appliance commands;

iteratively performing the receiving and associating steps until either a command input is received or the second data structure instantiation is complete; and

upon completion of an activity, increasing a weight associated with a setting for an appliance according to a command received for the activity.