Intelligent agent / personal virtual assistant with animated 3D persona, facial expressions, human gestures, body movements and mental states

• US 10,178,218 B1
• Filed: 09/12/2015
• Issued: 01/08/2019
• Est. Priority Date: 09/04/2015
• Status: Active Grant

• Current Assignee: Vishal Vadodaria
• Original Assignee: Vishal Vadodaria
• Inventors: Vadodaria, Vishal
• Primary Examiner(s): Salomon, Phenuel S
• Application Number:
  US14/852,556
• Time in Patent Office: 1,214 Days
• Field of Search: None
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 40/35 Discourse or dialogue representation
• CPC Class Codes: G06F 40/56 Natural language generation
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: H04L 51/063 Content adaptation, e.g. replacement of unsuitable content
• CPC Class Codes: H04M 1/72403 with means for local support of applications that increase the functionality
• CPC Class Codes: H04W 4/50 Service provisioning or reconfiguring

Abstract

A computer implemented system, method, and media for providing an Intelligent Interactive Agent that comprises an animated graphical representation of a plurality of behavioral states of the Intelligent Interactive Agent, said plurality of behavioral states comprising one or more states selected from the following: a Nervous state, an Afraid state, a Smiling state, a laughing state, a shocked state, or a Confused state, said behavioral state further comprising an index values for each state indicating the intensity of such states.

Claims

1. A portable electronic apparatus in communication with remote computing equipment over a communications path, comprising:

one or more input devices comprising at least a camera and a microphone;

one or more output devices comprising at least a touch screen and an audio speaker;

one or more processors; and

memory storing one or more programs executable by the one or more processors;

a Natural Language Processing/Natural Language Understanding (NLP/NLU) engine comprising programming instructions to dynamically receive and process user data comprising user verbal commands, user text, and user image data, and comprising programming instructions to dynamically output audio, text, and graphic animation responsive to the user data;

an audio percept module operatively associated with the microphone, the audio percept module comprising programming instructions to receive verbal commands from the user through the microphone and convert the verbal commands to audio percept data using the Natural Language Processing/Natural Language Understanding (NLP/NLU) engine;

a linguistic percept module operatively associated with the touch screen, the linguistic percept module comprising programming instructions to receive text from the user through the touch screen and convert the text to text percept data using the Natural Language Processing/Natural Language Understanding (NLP/NLU) engine;

a visual percept module operatively associated with the camera, the visual percept module comprising programming instructions to receive image data from the user through the camera and convert the image data into a visual percept data using the Natural Language Processing/Natural Language Understanding (NLP/NLU) engine;

a display output module operatively associated with the touch screen, the display output module comprising programming instructions to render an Intelligent Interactive Agent displayed on the touch screen as a graphic animation to a user;

wherein the display output module is programmed to display the Intelligent Interactive Agent having human-like characteristics, said characteristics comprising a human-like facial proportion, a human-like appearance, and human-like gestures;

an audio output module operatively associated with the audio speaker and linked to the display output module, the audio output module comprising programming instructions to to transmit a human-like voice through the audio speaker;

a text output module operatively associated with the touch screen and linked to the display output module, the text output module comprising programming instructions to transmit a text communications using the touch screen;

a semantic data module operatively associated with the NLP/NLU engine comprising programming instructions to output semantic data consisting essentially of semantic action data, semantic fact data, semantic modality data, semantic entity data, semantic urgency data and semantic concept data;

a Behavioral State module operatively associated with the NLP/NLU engine comprising programming instructions to computationally derive a Behavioral State of the Intelligent Interactive Agent and an index value from the semantic data, the Behavioral State comprising one or more states selected from a nervous state, an afraid state, a smiling state, a laughing state, a shocked state, a confused state, an embarrassed state, an overjoyed state, a relieved state, an angry state, a sorry state, a confident state, a crying state, an indifference state, or a frowning state, the index value comprising an intensity value for the Behavioral State;

a Mental State module module operatively associated with the NLP/NLU engine comprising programming instructions to computationally assign a Mental State from a series of derived Behavioral States over a period of time, the Mental State selected from physical energy, enthusiasm, health, happiness, sadness, fear, anger;

a Fact Repository comprising programming instructions to store the visual percept data, the audio percept data and the text percept data in memory;

an Ethics Checker Module linked to the NLP/NLU engine, the Ethics Checker Module comprising programming instructions to o computationally derive an Ethics Value from semantic data using a comparison of semantic action data against semantic entity data, the Ethics Checker Module comprising programming instructions to compare the ethics value against an Ethics Database and assign an Ethics Value ranging from −

10 to +10, the Ethics Value used in part to computationally derive the Behavioral State of the Intelligent Interactive Agent, the Ethics Checker Module comprising programming instructions to dynamically change the Behavioral State of the Intelligent Interactive Agent based on changes in user data wherein a sequence of Behavioral States are programmed to increase or decrease in intensity, the Ethics Checker Module linked to the Display Output Module and comprising programming instructions to dynamically change the graphic animation of the Intelligent Interactive Agent to synchronize a specific Behavioral State to a specific facial expression, voice modality, text modality, and gesture, rendered in the graphic animation;

a Validity Module comprising programming instructions to compare semantic action data to semantic entity data, and comprising programming instructions to compare semantic action data to semantic concept data using the Natural Language Processing/Natural Language Understanding (NLP/NLU) engine, the NLP/NLU having a Speech Extractor, the Speech Extractor comprising programming instructions to extract a grammar part of audio percept data or text percept data, the grammar part comprising a verb and a noun, a Verb Lexicon Matching Filter linked to the Validity Module and comprising programming instructions to disambiguate the verb to a semantic action data, a Noun Lexicon Filter comprising programming instructions to disambiguate the noun to a semantic entity, a Common Sense Knowledge Database linked to the Validity Module, the Validity Module comprising programming instructions to compare the semantic action data and the semantic entity data and output a Validity Value that the combination is valid, is invalid, or is possibly invalid, the Validity Module comprising programming instructions to create a user notification comprising the Validity Value, and comprising programming instructions to create a user confirmation question based on said validity value, said user confirmation question comprising an audio or text request presented by the Intelligent Interactive Agent to confirm yes or no, or comprising an audio or text request presented by the Intelligent Interactive Agent to offer an alternative use of one or more related verbs from the Verb Lexicon Matching Filter and one or more related nouns from the Noun Lexicon Filter, the Validity Value used in part to computationally derive the Behavioral State of the Intelligent Interactive Agent, the Validity Module comprising programming instructions to dynamically change the Behavioral State of the Intelligent Interactive Agent based on changes in user data wherein a sequence of Behavioral States are programmed to increase or decrease in intensity, the Validity Module linked to the Display Output Module and comprising programming instructions to dynamically change the graphic animation of the Intelligent Interactive Agent to synchronize a specific Behavioral State to a specific facial expression, voice modality, text modality, and gesture, rendered in the graphic animation;

a Missing Input Module comprising programming instructions to analyze semantic data for missing parameters and missing attributes using the Natural Language Processing/Natural Language Understanding (NLP/NLU) engine, the NLP/NLU engine comprising programming instructions to use the Speech Extractor to extract a grammar part of audio percept data or text percept data, the grammar part comprising a verb and a noun, a Verb Lexicon Matching Filter comprising programming instructions to disambiguate the verb to a semantic action data, a Noun Lexicon Filter comprising programming instructions to disambiguate the noun to a semantic entity data, a Missing Parameters and Missing Attributes Database linked to the Missing Input Module, the Missing Input Module comprising programming instructions to compare the semantic action data and the semantic entity data to the Missing Parameters and Missing Attributes Database and calculate a Missing Input Value based on whether one or more missing parameters and missing attributes are not provided, the Missing Input Module comprising programming instructions to create an audio or text user notification presented by the Intelligent Interactive Agent that a parameter or attribute of the one or more missing parameters and missing attributes are not provided, and comprising programming instructions to create an audio or text user confirmation question presented by the Intelligent Interactive Agent comprising an audio or text request to confirm yes or no, or comprising a request to provide the parameter or attribute of the one or more missing parameters and missing attributes, the Missing Input Value used in part to computationally derive the Behavioral State of the Intelligent Interactive Agent, the Missing Input Module comprising programming instructions to dynamically change the Behavioral State of the Intelligent Interactive Agent based on changes in user data wherein a sequence of Behavioral States are programmed to increase or decrease in intensity, the Missing Input Module linked to the Display Output Module and comprising programming instructions to dynamically change the graphic animation of the Intelligent Interactive Agent to synchronize a specific Behavioral State to a specific facial expression, voice modality, text modality, and gesture, rendered in the graphic animation;

an Abstract Input Module comprising programming instructions to analyze semantic data for abstraction parameters and attributes using the Natural Language Processing/Natural Language Understanding (NLP/NLU) engine, the NLP/NLU engine comprising programming instructions to use the Speech Extractor to extract an abstraction part of audio percept data or text percept data, the abstraction part comprising a start location, destination location, start date, end date, start time, end time, a generic verb, a generic noun, and a generic adjective, an Abstraction Level Evaluator linked to the Abstract Input Module and comprising programming instructions to disambiguate the abstraction part to semantic data, the Abstract Input Module comprising programming instructions to calculate a Percentage Variance of the Area of the Expected Input and Area of the actual User Input, and output an Abstraction Value, the Abstract Input Module comprising programming instructions to create an audio or text user notification presented by the Intelligent Interactive Agent that an abstraction mismatch has occurred, and comprising programming instructions to request by audio or text presented by the Intelligent Interactive Agent to provide additional information, the Abstraction Value used in part to computationally derive the Behavioral State of the Intelligent Interactive Agent, the Abstract Input Module comprising programming instructions to dynamically change the Behavioral State of the Intelligent Interactive Agent based on changes in user data wherein a sequence of Behavioral States are programmed to increase or decrease in intensity, the Abstract Input Module linked to the Display Output Module and comprising programming instructions to dynamically change the graphic animation of the Intelligent Interactive Agent to synchronize a specific Behavioral State to a specific facial expression, voice modality, text modality, and gesture, rendered in the graphic animation;

an Expectation Module comprising programming instructions to analyze semantic data for expectation parameters and attributes using the Natural Language Processing/Natural Language Understanding (NLP/NLU) engine, the NLP/NLU engine comprising programming instructions to use the Speech Extractor to extract an expectation part of audio percept data or text percept data, the expectation part comprising a tone of voice, a decency of language, a modality of language, a specificity of location, date, time, verb, noun, or adjective, completeness of request, validity of action, and ethics of action, the Expectation Module comprising programming instructions to disambiguate the expectation to a semantic data, the Expectation Module comprising programming instructions to compare the semantic data to the Expectation Evaluator and output a tone of voice value, a decency of language value, a modality of language value, a validity value, an ethics value, a Specificity Factor, Average Completeness value, the Expectation Module comprising programming instructions to create an audio or text user notification presented by the Intelligent Interactive Agent that an expectation mismatch has occurred, and comprising programming instructions to request by audio or text presented by the Intelligent Interactive Agent to provide addition information, the Expectation Value used in part to computationally derive the Behavioral State of the Intelligent Interactive Agent, the Expectation Module comprising programming instructions to dynamically change the Behavioral State of the Intelligent Interactive Agent based on changes in user data wherein a sequence of Behavioral States are programmed to increase or decrease in intensity, the Expectation Module linked to the Display Output Module and comprising programming instructions to dynamically change the graphic animation of the Intelligent Interactive Agent to synchronize a specific Behavioral State to a specific facial expression, voice modality, text modality, and gesture, rendered in the graphic animation;

an Impact Analysis Module comprising programming instructions to compare a user'"'"'s audio or text command to historical user data using the Natural Language Processing/Natural Language Understanding (NLP/NLU) engine, the NLP/NLU engine comprising programming instructions to use the Speech Extractor to extract a command part of audio percept data or text percept data, a Historical Success/Failure Database linked to the Impact Analysis Module, the Impact Analysis Module comprising programming instructions to compare the command part to the Historical Success/Failure Database, and output a Success Value based on whether the command part matches success parameters and attributes stored in the Historical Success/Failure Database, the Impact Analysis Module comprising programming instructions to create an audio or text user notification presented by the Intelligent Interactive Agent before a user command is executed when an success mismatch has occurred, and to create a second audio or text user notification presented by the Intelligent Interactive Agent after a user command is executed, and comprising programming instructions to request by audio or text presented by the Intelligent Interactive Agent a confirmation of success or failure, and comprising programming instructions to store the Success Value and the confirmation of success of failure to the Historical Success/Failure Database, the Success Value used in part to computationally derive the Behavioral State of the Intelligent Interactive Agent, the Impact Analysis Module comprising programming instructions to dynamically change the Behavioral State of the Intelligent Interactive Agent based on changes in user data wherein a sequence of Behavioral States are programmed to increase or decrease in intensity, the Impact Analysis Module linked to the Display Output Module and comprising programming instructions to dynamically change the graphic animation of the Intelligent Interactive Agent to synchronize a specific Behavioral State to a specific facial expression, voice modality, text modality, and gesture, rendered in the graphic animation; and

,

a Facial Expressions Animation Module comprising programming instructions to dynamically render facial graphic animation responsive to the Intelligent Interactive Agent Behavioral State and Mental State, the Facial Expressions Animation Module comprising programming instructions to match a Behavioral State to a Facial Expression wherein a Nervous Behavioral State is paired to a Nervous Facial Expression, an Afraid Behavioral State is paired to an Afraid Facial Expression, a Smiling Behavioral State is paired to a Smiling Facial Expression, a Laughing Behavioral State is paired to a Laughing Facial Expression, a Shocked Behavioral State is paired to a Shocked Facial Expression, a Confused Behavioral State is paired to a Confused Facial Expression, an Embarrassed Behavioral State is paired to an Embarrassed Facial Expression, an Overjoyed Behavioral State is paired to an Overjoyed Facial Expression, a Relieved Behavioral State is paired to a Relieved Facial Expression, an Angry Behavioral State is paired to an Angry Facial Expression, a Sorry Behavioral State is paired to a Sorry Facial Expression, a Confident Behavioral State is paired to a Confident Facial Expression, a Crying Behavioral State is paired to a Crying Facial Expression, an Indifference Behavioral State is paired to an Indifference Facial Expression, and a Frowning Behavioral State is paired to a Frowning Facial Expression, the Facial Expressions Animation Module comprising programming instructions to assign an Expressiveness Factor ranging from 0 to 5 to each Facial Expression where EF-0 is low expressivity and EF-5 is highest expressivity,

the facial graphic animation comprising programming instructions to comprise Atomic Expressions that involve movement of the facial muscle groups of the forehead, eyebrows, eyelids, eyes, nose, cheeks, lips and the jaw bone, each movement programmed by calculating a set of Morph Targets on an animation mesh for each Facial Expression, each set of Morph Targets comprising a set of key points on the animation mesh that move from an initial position to a final position;

a Hand Gestures Animation Module comprising programming instructions to dynamically render hand gesture graphic animation responsive to the Intelligent Interactive Agent Behavioral State and Mental State, each Behavioral State paired to a specific hand gesture, the hand gesture graphic animation programmed to comprise movement of shoulders, elbows, neck, spine, wrist, and fingers, each movement programmed by calculating a set of Morph Targets on an animation mesh for each Hand Gesture, each set of Morph Targets comprising a set of key points on the animation mesh that move from an initial position to a final position; and

,

a Body Gestures Animation Module comprising programming instructions to dynamically render body gesture graphic animation responsive to the Intelligent Interactive Agent Behavioral State and Mental State, each Behavioral State paired to a specific Body Gesture, the body gesture graphic animation programmed to comprise movement of shoulders, elbows, neck, spine, each movement programmed by calculating a set of Morph Targets on an animation mesh for each Body Gesture, each set of Morph Targets comprising a set of key points on the animation mesh that move from an initial position to a final position.

2. The portable electronic apparatus of claim 1, further comprising wherein the one or more programs include instructions for accepting ambiguous user input and validating user input using a syntactical tree structure module.

3. The portable electronic apparatus of claim 1, further comprising an attribute clarification module, said attribute clarification module comprising programming instructions to identify ambiguous content in the user input, where the ambiguous content comprises plural items that are not associated with numerical attributes, calendar items that are missing date and time attributes, relationship items that are missing hierarchy attributes, navigation items that are missing location attributes, and where the attribute clarification module automatically formulates a question or option for a user to add the missing attributes.

4. A method of providing a graphical representation of a behavioral state of an animated interactive intelligent agent, comprising the steps:

providing the portable electronic apparatus of claim 1;

inputting a spoken verbal command to the Intelligent Interactive Agent to perform an action selected from the group consisting of;

searching airline flights, buying food, making travel arrangements, finding weather conditions, obtaining directions, buying a house, selling a house, renting a house, scheduling a meeting, setting a reminder, replying to a contact, making a restaurant reservation; and

receiving requests from the Intelligent Interactive Agent to provide details and receiving options and alternatives from the Intelligent Interactive Agent, where processing of the spoken verbal command to the Intelligent Interactive Agent generated a graphical representation of a Behavioral State of the Intelligent Interactive Agent comprising one or more states selected from a nervous state, an afraid state, a smiling state, a laughing state, a shocked state, a confused state, an embarrassed state, an overjoyed state, a relieved state, an angry state, a sorry state, a confident state, a crying state, an indifference state, or a frowning state.

5. The method of claim 4, further comprising the step wherein the one or more programs include instructions for accepting ambiguous user input and validating user input using one or more of the following modules:

a syntactical tree structure module, an abstraction module, a disambiguation module, or a common sense knowledge module.

6. The method of claim 4, further comprising the step wherein the one or more programs include instructions for providing feedback to the user wherein the feedback comprises an ethical graphical or spoken output from the Intelligent Interactive Agent.

7. The method of claim 4, further comprising the step wherein the one or more programs include instructions for providing feedback to the user wherein the feedback comprises a graphical or spoken output from the Intelligent Interactive Agent, said output comprising an attribute clarification module, said attribute clarification module configured to identify ambiguous content in the user input, where the ambiguous content comprises plural items that are not associated with numerical attributes, calendar items that are missing date and time attributes, relationship items that are missing hierarchy attributes, navigation items that are missing location attributes, and where the attribute clarification module automatically formulates a question or option for a user to add the missing attributes.

8. The method of claim 4, further comprising the step wherein the one or more programs include instructions for providing feedback to the user wherein the feedback comprises a graphical or spoken output from the Intelligent Interactive Agent, said output comprising an intensity module, said intensity module configured to display facial characteristics in the Intelligent Interactive Agent reactive to a users input, wherein said user input is analyzed for tone of user voice input, and decency of language used.

9. A non-transitory computer-readable storage medium comprising instructions that, when executed by a processor, cause the processor to:

run one or more programs including programming instructions comprising the following modules;

a Natural Language Processing/Natural Language Understanding (NLP/NLU) engine comprising programming instructions to dynamically receive and process user data comprising user verbal commands, user text, and user image data, and comprising programming instructions to dynamically output audio, text, and graphic animation responsive to the user data;

an audio percept module operatively associated with the microphone, the audio percept module comprising programming instructions to receive verbal commands from the user through the microphone and convert the verbal commands to audio percept data using the Natural Language Processing/Natural Language Understanding (NLP/NLU) engine;

a linguistic percept module operatively associated with the touch screen, the linguistic percept module comprising programming instructions to receive text from the user through the touch screen and convert the text to text percept data using the Natural Language Processing/Natural Language Understanding (NLP/NLU) engine;

a visual percept module operatively associated with the camera, the visual percept module comprising programming instructions to receive image data from the user through the camera and convert the image data into a visual percept data using the Natural Language Processing/Natural Language Understanding (NLP/NLU) engine;

a display output module operatively associated with the touch screen, the display output module comprising programming instructions to render an Intelligent Interactive Agent displayed on the touch screen as a graphic animation to a user;

wherein the display output module is programmed to display the Intelligent Interactive Agent having human-like characteristics, said characteristics comprising a human-like facial proportion, a human-like appearance, and human-like gestures;

an audio output module operatively associated with the audio speaker and linked to the display output module, the audio output module comprising programming instructions to to transmit a human-like voice through the audio speaker;

a text output module operatively associated with the touch screen and linked to the display output module, the text output module comprising programming instructions to transmit a text communications using the touch screen;

a semantic data module operatively associated with the NLP/NLU engine comprising programming instructions to output semantic data consisting essentially of semantic action data, semantic fact data, semantic modality data, semantic entity data, semantic urgency data and semantic concept data;

a Behavioral State module operatively associated with the NLP/NLU engine comprising programming instructions to computationally derive a Behavioral State of the Intelligent Interactive Agent and an index value from the semantic data, the Behavioral State comprising one or more states selected from a nervous state, an afraid state, a smiling state, a laughing state, a shocked state, a confused state, an embarrassed state, an overjoyed state, a relieved state, an angry state, a sorry state, a confident state, a crying state, an indifference state, or a frowning state, the index value comprising an intensity value for the Behavioral State;

a Mental State module module operatively associated with the NLP/NLU engine comprising programming instructions to computationally assign a Mental State from a series of derived Behavioral States over a period of time, the Mental State selected from physical energy, enthusiasm, health, happiness, sadness, fear, anger;

a Fact Repository comprising programming instructions to store the visual percept data, the audio percept data and the text percept data in memory;

an Ethics Checker Module linked to the NLP/NLU engine, the Ethics Checker Module comprising programming instructions to o computationally derive an Ethics Value from semantic data using a comparison of semantic action data against semantic entity data, the Ethics Checker Module comprising programming instructions to compare the ethics value against an Ethics Database and assign an Ethics Value ranging from −

10 to +10, the Ethics Value used in part to computationally derive the Behavioral State of the Intelligent Interactive Agent, the Ethics Checker Module comprising programming instructions to dynamically change the Behavioral State of the Intelligent Interactive Agent based on changes in user data wherein a sequence of Behavioral States are programmed to increase or decrease in intensity, the Ethics Checker Module linked to the Display Output Module and comprising programming instructions to dynamically change the graphic animation of the Intelligent Interactive Agent to synchronize a specific Behavioral State to a specific facial expression, voice modality, text modality, and gesture, rendered in the graphic animation;

a Validity Module comprising programming instructions to compare semantic action data to semantic entity data, and comprising programming instructions to compare semantic action data to semantic concept data using the Natural Language Processing/Natural Language Understanding (NLP/NLU) engine, the NLP/NLU having a Speech Extractor, the Speech Extractor comprising programming instructions to extract a grammar part of audio percept data or text percept data, the grammar part comprising a verb and a noun, a Verb Lexicon Matching Filter linked to the Validity Module and comprising programming instructions to disambiguate the verb to a semantic action data, a Noun Lexicon Filter comprising programming instructions to disambiguate the noun to a semantic entity, a Common Sense Knowledge Database linked to the Validity Module, the Validity Module comprising programming instructions to compare the semantic action data and the semantic entity data and output a Validity Value that the combination is valid, is invalid, or is possibly invalid, the Validity Module comprising programming instructions to create a user notification comprising the Validity Value, and comprising programming instructions to create a user confirmation question based on said validity value, said user confirmation question comprising an audio or text request presented by the Intelligent Interactive Agent to confirm yes or no, or comprising an audio or text request presented by the Intelligent Interactive Agent to offer an alternative use of one or more related verbs from the Verb Lexicon Matching Filter and one or more related nouns from the Noun Lexicon Filter, the Validity Value used in part to computationally derive the Behavioral State of the Intelligent Interactive Agent, the Validity Module comprising programming instructions to dynamically change the Behavioral State of the Intelligent Interactive Agent based on changes in user data wherein a sequence of Behavioral States are programmed to increase or decrease in intensity, the Validity Module linked to the Display Output Module and comprising programming instructions to dynamically change the graphic animation of the Intelligent Interactive Agent to synchronize a specific Behavioral State to a specific facial expression, voice modality, text modality, and gesture, rendered in the graphic animation;

a Missing Input Module comprising programming instructions to analyze semantic data for missing parameters and missing attributes using the Natural Language Processing/Natural Language Understanding (NLP/NLU) engine, the NLP/NLU engine comprising programming instructions to use the Speech Extractor to extract a grammar part of audio percept data or text percept data, the grammar part comprising a verb and a noun, a Verb Lexicon Matching Filter comprising programming instructions to disambiguate the verb to a semantic action data, a Noun Lexicon Filter comprising programming instructions to disambiguate the noun to a semantic entity data, a Missing Parameters and Missing Attributes Database linked to the Missing Input Module, the Missing Input Module comprising programming instructions to compare the semantic action data and the semantic entity data to the Missing Parameters and Missing Attributes Database and calculate a Missing Input Value based on whether one or more missing parameters and missing attributes are not provided, the Missing Input Module comprising programming instructions to create an audio or text user notification presented by the Intelligent Interactive Agent that a parameter or attribute of the one or more missing parameters and missing attributes are not provided, and comprising programming instructions to create an audio or text user confirmation question presented by the Intelligent Interactive Agent comprising an audio or text request to confirm yes or no, or comprising a request to provide the parameter or attribute of the one or more missing parameters and missing attributes, the Missing Input Value used in part to computationally derive the Behavioral State of the Intelligent Interactive Agent, the Missing Input Module comprising programming instructions to dynamically change the Behavioral State of the Intelligent Interactive Agent based on changes in user data wherein a sequence of Behavioral States are programmed to increase or decrease in intensity, the Missing Input Module linked to the Display Output Module and comprising programming instructions to dynamically change the graphic animation of the Intelligent Interactive Agent to synchronize a specific Behavioral State to a specific facial expression, voice modality, text modality, and gesture, rendered in the graphic animation;

an Abstract Input Module comprising programming instructions to analyze semantic data for abstraction parameters and attributes using the Natural Language Processing/Natural Language Understanding (NLP/NLU) engine, the NLP/NLU engine comprising programming instructions to use the Speech Extractor to extract an abstraction part of audio percept data or text percept data, the abstraction part comprising a start location, destination location, start date, end date, start time, end time, a generic verb, a generic noun, and a generic adjective, an Abstraction Level Evaluator linked to the Abstract Input Module and comprising programming instructions to disambiguate the abstraction part to semantic data, the Abstract Input Module comprising programming instructions to calculate a Percentage Variance of the Area of the Expected Input and Area of the actual User Input, and output an Abstraction Value, the Abstract Input Module comprising programming instructions to create an audio or text user notification presented by the Intelligent Interactive Agent that an abstraction mismatch has occurred, and comprising programming instructions to request by audio or text presented by the Intelligent Interactive Agent to provide additional information, the Abstraction Value used in part to computationally derive the Behavioral State of the Intelligent Interactive Agent, the Abstract Input Module comprising programming instructions to dynamically change the Behavioral State of the Intelligent Interactive Agent based on changes in user data wherein a sequence of Behavioral States are programmed to increase or decrease in intensity, the Abstract Input Module linked to the Display Output Module and comprising programming instructions to dynamically change the graphic animation of the Intelligent Interactive Agent to synchronize a specific Behavioral State to a specific facial expression, voice modality, text modality, and gesture, rendered in the graphic animation;

an Expectation Module comprising programming instructions to analyze semantic data for expectation parameters and attributes using the Natural Language Processing/Natural Language Understanding (NLP/NLU) engine, the NLP/NLU engine comprising programming instructions to use the Speech Extractor to extract an expectation part of audio percept data or text percept data, the expectation part comprising a tone of voice, a decency of language, a modality of language, a specificity of location, date, time, verb, noun, or adjective, completeness of request, validity of action, and ethics of action, the Expectation Module comprising programming instructions to disambiguate the expectation to a semantic data, the Expectation Module comprising programming instructions to compare the semantic data to the Expectation Evaluator and output a tone of voice value, a decency of language value, a modality of language value, a validity value, an ethics value, a Specificity Factor, Average Completeness value, the Expectation Module comprising programming instructions to create an audio or text user notification presented by the Intelligent Interactive Agent that an expectation mismatch has occurred, and comprising programming instructions to request by audio or text presented by the Intelligent Interactive Agent to provide addition information, the Expectation Value used in part to computationally derive the Behavioral State of the Intelligent Interactive Agent, the Expectation Module comprising programming instructions to dynamically change the Behavioral State of the Intelligent Interactive Agent based on changes in user data wherein a sequence of Behavioral States are programmed to increase or decrease in intensity, the Expectation Module linked to the Display Output Module and comprising programming instructions to dynamically change the graphic animation of the Intelligent Interactive Agent to synchronize a specific Behavioral State to a specific facial expression, voice modality, text modality, and gesture, rendered in the graphic animation;

an Impact Analysis Module comprising programming instructions to compare a user'"'"'s audio or text command to historical user data using the Natural Language Processing/Natural Language Understanding (NLP/NLU) engine, the NLP/NLU engine comprising programming instructions to use the Speech Extractor to extract a command part of audio percept data or text percept data, a Historical Success/Failure Database linked to the Impact Analysis Module, the Impact Analysis Module comprising programming instructions to compare the command part to the Historical Success/Failure Database, and output a Success Value based on whether the command part matches success parameters and attributes stored in the Historical Success/Failure Database, the Impact Analysis Module comprising programming instructions to create an audio or text user notification presented by the Intelligent Interactive Agent before a user command is executed when an success mismatch has occurred, and to create a second audio or text user notification presented by the Intelligent Interactive Agent after a user command is executed, and comprising programming instructions to request by audio or text presented by the Intelligent Interactive Agent a confirmation of success or failure, and comprising programming instructions to store the Success Value and the confirmation of success of failure to the Historical Success/Failure Database, the Success Value used in part to computationally derive the Behavioral State of the Intelligent Interactive Agent, the Impact Analysis Module comprising programming instructions to dynamically change the Behavioral State of the Intelligent Interactive Agent based on changes in user data wherein a sequence of Behavioral States are programmed to increase or decrease in intensity, the Impact Analysis Module linked to the Display Output Module and comprising programming instructions to dynamically change the graphic animation of the Intelligent Interactive Agent to synchronize a specific Behavioral State to a specific facial expression, voice modality, text modality, and gesture, rendered in the graphic animation; and

,

a Facial Expressions Animation Module comprising programming instructions to dynamically render facial graphic animation responsive to the Intelligent Interactive Agent Behavioral State and Mental State, the Facial Expressions Animation Module comprising programming instructions to match a Behavioral State to a Facial Expression wherein a Nervous Behavioral State is paired to a Nervous Facial Expression, an Afraid Behavioral State is paired to an Afraid Facial Expression, a Smiling Behavioral State is paired to a Smiling Facial Expression, a Laughing Behavioral State is paired to a Laughing Facial Expression, a Shocked Behavioral State is paired to a Shocked Facial Expression, a Confused Behavioral State is paired to a Confused Facial Expression, an Embarrassed Behavioral State is paired to an Embarrassed Facial Expression, an Overjoyed Behavioral State is paired to an Overjoyed Facial Expression, a Relieved Behavioral State is paired to a Relieved Facial Expression, an Angry Behavioral State is paired to an Angry Facial Expression, a Sorry Behavioral State is paired to a Sorry Facial Expression, a Confident Behavioral State is paired to a Confident Facial Expression, a Crying Behavioral State is paired to a Crying Facial Expression, an Indifference Behavioral State is paired to an Indifference Facial Expression, and a Frowning Behavioral State is paired to a Frowning Facial Expression, the Facial Expressions Animation Module comprising programming instructions to assign an Expressiveness Factor ranging from 0 to 5 to each Facial Expression where EF-0 is low expressivity and EF-5 is highest expressivity,

the facial graphic animation comprising programming instructions to comprise Atomic Expressions that involve movement of the facial muscle groups of the forehead, eyebrows, eyelids, eyes, nose, cheeks, lips and the jaw bone, each movement programmed by calculating a set of Morph Targets on an animation mesh for each Facial Expression, each set of Morph Targets comprising a set of key points on the animation mesh that move from an initial position to a final position;

a Hand Gestures Animation Module comprising programming instructions to dynamically render hand gesture graphic animation responsive to the Intelligent Interactive Agent Behavioral State and Mental State, each Behavioral State paired to a specific hand gesture, the hand gesture graphic animation programmed to comprise movement of shoulders, elbows, neck, spine, wrist, and fingers, each movement programmed by calculating a set of Morph Targets on an animation mesh for each Hand Gesture, each set of Morph Targets comprising a set of key points on the animation mesh that move from an initial position to a final position; and

,

a Body Gestures Animation Module comprising programming instructions to dynamically render body gesture graphic animation responsive to the Intelligent Interactive Agent Behavioral State and Mental State, each Behavioral State paired to a specific Body Gesture, the body gesture graphic animation programmed to comprise movement of shoulders, elbows, neck, spine, each movement programmed by calculating a set of Morph Targets on an animation mesh for each Body Gesture, each set of Morph Targets comprising a set of key points on the animation mesh that move from an initial position to a final position.