Intelligent assistant with intent-based information resolution

• US 10,496,905 B2
• Filed: 07/21/2017
• Issued: 12/03/2019
• Est. Priority Date: 02/14/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Solomon, Oz, Quirk, Christopher Brian, Fung, Han Yee Mimi, Herold, Keith Coleman
• Primary Examiner(s): Saint Cyr, Leonard
• Application Number:
  US15/657,031
• Publication Number:
  US 20180233141A1
• Time in Patent Office: 865 Days
• Field of Search: 704 7- 10
• CPC Class Codes: A61B 5/0205 Simultaneously evaluating both cardiovascular conditions and different types of body conditions, e.g. heart and respiratory condition
• CPC Class Codes: A61B 5/05 Detecting, measuring or recording for diagnosis by means of electric currents or magnetic fields; Measuring using microwaves or radio waves impedance plethysmography A61B5/0295; measuring movement of the entire body or parts thereof A61B5/11; detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof A61B5/24
• CPC Class Codes: A61B 5/0507 using microwaves or terahertz waves
• CPC Class Codes: A61B 5/1113 Local tracking of patients, e.g. in a hospital or private home
• CPC Class Codes: A61B 5/1118 Determining activity level
• CPC Class Codes: A61B 5/117 Identification of persons methods or arrangements for recognising patterns, e.g. fingerprints, G06F18/00, G06V40/00; identification of persons by analysing their voice or speech G10L17/00
• CPC Class Codes: A61B 5/7475 User input or interface means, e.g. keyboard, pointing device, joystick
• CPC Class Codes: G01S 11/14 using ultrasonic, sonic, or infrasonic waves
• CPC Class Codes: G01S 13/38 wherein more than one modulation frequency is used
• CPC Class Codes: G01S 13/726 Multiple target tracking
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/888 through wall detection
• CPC Class Codes: G01S 5/16 using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 5/18 using ultrasonic, sonic, or infrasonic waves
• CPC Class Codes: G01S 5/28 by co-ordinating position lines of different shape, e.g. hyperbolic, circular, elliptical or radial
• CPC Class Codes: G06F 1/3206 Monitoring of events, devices or parameters that trigger a change in power modality
• CPC Class Codes: G06F 1/3231 Monitoring the presence, absence or movement of users
• CPC Class Codes: G06F 1/324 by lowering clock frequency
• CPC Class Codes: G06F 1/329 by task scheduling
• CPC Class Codes: G06F 16/70 of video data
• CPC Class Codes: G06F 18/251 : of input or preprocessed data
• CPC Class Codes: G06F 18/28 : Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06F 18/29 : Graphical models, e.g. Bayesian networks
• CPC Class Codes: G06F 18/41 : Interactive pattern learning with a human teacher
• CPC Class Codes: G06F 21/00 : Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
• CPC Class Codes: G06F 21/32 : using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06F 21/35 : communicating wirelessly
• CPC Class Codes: G06F 2203/0381 : Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 2221/2111 : Location-sensitive, e.g. geographical location, GPS
• CPC Class Codes: G06F 2221/2117 : User registration
• CPC Class Codes: G06F 3/011 : Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 : Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 : Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0482 : Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 : Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0488 : using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/167 : Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/211 : Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
• CPC Class Codes: G06F 40/35 : Discourse or dialogue representation
• CPC Class Codes: G06N 20/00 : Machine learning
• CPC Class Codes: G06N 3/044 : Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 5/025 : Extracting rules from data
• CPC Class Codes: G06N 5/047 : Pattern matching networks; Rete networks
• CPC Class Codes: G06T 2207/10016 : Video; Image sequence
• CPC Class Codes: G06T 2207/10024 : Color image
• CPC Class Codes: G06T 2207/20101 : Interactive definition of point of interest, landmark or seed
• CPC Class Codes: G06T 2207/30196 : Human being; Person
• CPC Class Codes: G06T 2207/30201 : Face
• CPC Class Codes: G06T 2207/30204 : Marker
• CPC Class Codes: G06T 2207/30232 : Surveillance
• CPC Class Codes: G06T 7/248 : involving reference images or patches
• CPC Class Codes: G06T 7/292 : Multi-camera tracking
• CPC Class Codes: G06T 7/60 : Analysis of geometric attributes
• CPC Class Codes: G06T 7/70 : Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/74 : involving reference images or patches
• CPC Class Codes: G06V 10/94 : Hardware or software architectures specially adapted for image or video understanding
• CPC Class Codes: G06V 20/40 : in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/52 : Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/653 : by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 2201/05 : Recognition of patterns representing particular kinds of hidden objects, e.g. weapons, explosives, drugs
• CPC Class Codes: G06V 30/274 : Syntactic or semantic context, e.g. balancing
• CPC Class Codes: G06V 40/10 : Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/103 : Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/166 : using acquisition arrangements
• CPC Class Codes: G06V 40/167 : using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/172 : Classification, e.g. identification
• CPC Class Codes: G06V 40/173 : face re-identification, e.g. recognising unknown faces across different face tracks
• CPC Class Codes: G06V 40/23 : Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: G06V 40/25 : Recognition of walking or running movements, e.g. gait recognition
• CPC Class Codes: G07C 9/28 : the pass enabling tracking or indicating presence
• CPC Class Codes: G07C 9/32 : in combination with an identity check
• CPC Class Codes: G08B 13/1427 : with transmitter-receiver for distance detection
• CPC Class Codes: G08B 29/186 : Fuzzy logic; neural networks
• CPC Class Codes: G10L 15/00 : Speech recognition G10L17/00 takes precedence
• CPC Class Codes: G10L 15/02 : Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/063 : Training
• CPC Class Codes: G10L 15/08 : Speech classification or search
• CPC Class Codes: G10L 15/18 : using natural language modelling
• CPC Class Codes: G10L 15/1815 : Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/1822 : Parsing for meaning understanding
• CPC Class Codes: G10L 15/19 : Grammatical context, e.g. disambiguation of the recognition hypotheses based on word sequence rules
• CPC Class Codes: G10L 15/22 : Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/24 : Speech recognition using non-acoustical features
• CPC Class Codes: G10L 15/26 : Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 15/28 : Constructional details of speech recognition systems
• CPC Class Codes: G10L 15/32 : Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems
• CPC Class Codes: G10L 17/00 : Speaker identification or verification techniques
• CPC Class Codes: G10L 17/04 : Training, enrolment or model building
• CPC Class Codes: G10L 17/06 : Decision making techniques; Pattern matching strategies
• CPC Class Codes: G10L 17/08 : Use of distortion metrics or a particular distance between probe pattern and reference templates
• CPC Class Codes: G10L 17/22 : Interactive procedures; Man-machine interfaces
• CPC Class Codes: G10L 2015/0635 : updating or merging of old and new templates; Mean values; Weighting
• CPC Class Codes: G10L 2015/088 : Word spotting
• CPC Class Codes: G10L 2015/223 : Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/225 : Feedback of the input speech
• CPC Class Codes: G10L 2015/228 : of application context
• CPC Class Codes: G10L 25/51 : for comparison or discrimination
• CPC Class Codes: H04L 51/02 : using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
• CPC Class Codes: H04L 63/102 : Entity profiles
• CPC Class Codes: H04L 67/12 : specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 67/535 : Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34
• CPC Class Codes: H04N 21/231 : Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion
• CPC Class Codes: H04N 21/42203 : sound input device, e.g. microphone
• CPC Class Codes: H04N 21/44218 : Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/44222 : Analytics of user selections, e.g. selection of programs or purchase activity monitoring of user selections in data processing systems G06F11/34; arrangements for monitoring the user's behaviour or opinions in broadcast systems H04H60/33
• CPC Class Codes: H04N 21/44224 : Monitoring of user activity on external systems, e.g. Internet browsing
• CPC Class Codes: H04N 23/11 : for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/611 : where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/188 : Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position signal generation from motion picture films H04N5/253
• CPC Class Codes: H04R 1/406 : microphones
• CPC Class Codes: H04R 3/005 : for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407
• CPC Class Codes: H04W 4/029 : Location-based management or tracking services
• CPC Class Codes: H04W 4/33 : for indoor environments, e.g. buildings
• CPC Class Codes: Y02D 10/00 : Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

A method for use with a computing device is provided. The method may include executing one or more programs of an intelligent digital assistant system at a processor and presenting a user interface to a user. At the processor, the method may include receiving natural language user input from the user, parsing the user input at an intent handler to determine an intent template with slots, populating the slots in the intent template with information from user input, and performing resolution on the intent template to partially resolve unresolved information. If a slot with missing slot information exists in the partially resolved intent template, a loop may be executed at the processor to fill the slots. The method may include, at the processor, determining that all required information is available and resolved and generating a rule based upon the intent template with all required information being available and resolved.

Claims

1. A method executed by a computing device, comprising:

at a processor;

executing one or more programs of an intelligent digital assistant system to present a user interface to a user;

receiving natural language user input from the user via the user interface;

parsing the user input at an intent handler of the system to determine an intent template with slots for information;

performing the following actions in a loop executed for the intent template until all slots required by the intent template are both filled and resolved;

populating the slots in the intent template with collected information from the user input;

performing resolution on the intent template to at least partially resolve unresolved information, thereby resulting in an at least partially resolved intent template;

filling implicitly missing slot information where possible with implicit knowledge derived from previously filled slots and other context information available to the system;

if, in the at least partially resolved intent template, not all slots required by the intent template are both filled and resolved, then performing the following actions for each slot required by the intent template;

determining a state of each slot that is not both filled and resolved;

presenting a query to the user to fill or resolve a selected slot that is not both filled and resolved based on query selection criteria;

receiving a user response to the query;

based on the user response to the query, altering the state of the selected slot; and

re-executing the loop with the user input and the user response to the query;

exiting the loop upon a determination that all slots required by the intent template are both filled and resolved;

generating a rule based upon the intent template with all required information being available and resolved; and

performing a defined action that applies the generated rule.

2. The method of claim 1, wherein

the state of each slot that is not both filled and resolved selected from the group consisting of implicitly missing slot information, explicitly missing slot information, and available and unresolved slot information, and

the query selection criteria include at least one criteria selected from the group consisting of random selection of missing or unresolved slot information, an internal order of explicitly missing slot information that is required to be filled, a greatest potential to fill and resolve a highest number of slots, an estimation of a user expected query, and a selection criteria calculated by the one or more programs.

3. The method of claim 1, wherein the intent template includes a trigger condition definition and an action definition, one or more of the slots being included in each of the trigger condition definition and the action definition, each slot configured to contain information to define the trigger condition or action, respectively, and wherein the rule that is generated is a rule to perform an action defined by the action definition upon an occurrence of a trigger defined by the trigger condition definition.

4. The method of claim 3, wherein

the slots have an information type selected from the group consisting of not required, required and implicitly missing, required and explicitly missing, required and available but unresolved, and required and available and not requiring resolution.

5. The method of claim 1, wherein the intent template is a first intent template, and wherein the first intent template is selected from among a plurality of candidate intent templates when it is determined, the method further comprising:

determining that a second intent template of the plurality of candidate intent templates is a closer match to the user input and the user response to the query, based on slots filled and resolved within the loop;

changing the first intent template to the second intent template with slots for information;

populating the second intent template based on the user input and the user response to the query; and

re-performing resolution on the populated second intent template.

6. The method of claim 1, further comprising, based on information collected by the system, determining that slot information occupies incorrect slots, and moving slot information to correct slots.

7. The method of claim 1, further comprising, based on information collected by the system, extracting new information and replacing existing slot information with the new information.

8. The method of claim 1, wherein implicit knowledge is derived by the system via the processor from data selected from the group consisting of user preferences, past usage of the user, identification of an entity, identification of an environment, placement of the entity in the environment, activities of the entity, placement of other entities in the environment, activities of the other entities, and crowd behavior.

9. The method of claim 1, wherein performing resolution via the system includes executing lookup resolvers that translate identifiers in the user input into internal representation data.

10. The method of claim 1, wherein performing resolution via the system includes executing anaphoric resolvers that resolve information based on antecedent or postcedent expression in a context of user input.

11. The method of claim 1, wherein performing resolution via the system includes executing deixis resolvers that resolve information for words that cannot be interpreted without additional contextual information.

12. A computing system, comprising:

at least a processor configured to execute one or more programs of an intelligent digital assistant system, to;

receive natural language user input from a user via a user interface;

at an intent handler of the system, parse the user input to determine an intent template with slots for information;

perform the following actions in a loop executed for the intent template until all slots required by the intent template are both filled and resolved;

populate the slots in the intent template with collected information from the user input;

perform resolution on the intent template to at least partially resolve unresolved information, thereby resulting in an at least partially resolved intent template;

fill implicitly missing slot information where possible with implicit knowledge derived from previously filled slots and other context information available to the system;

if, in the at least partially resolved intent template, not all slots required by the intent template are both filled and resolved, then perform the following actions for each slot required by the intent template;

determine a state of each slot that is not both filled and resolved;

present a query to the user to fill or resolve a selected slot that is not both filled and resolved based on query selection criteria;

receive a user response to the query;

based on the user response to the query, alter the state of the selected slot; and

re-execute the loop with the user input and the user response to the query;

exit the loop upon a determination that all slots required by the intent template are both filled and resolved;

generate a rule based upon the intent template with all required information being available and resolved; and

perform a defined action that applies the generated rule.

13. The computing system of claim 12, wherein

the state of each slot that is not both filled and resolved selected from the group consisting of implicitly missing slot information, explicitly missing slot information, and available and unresolved slot information, and

the query selection criteria are at least one selected from the group consisting of random selection of missing or unresolved slot information, an internal order of explicitly missing slot information that is required to be filled, a greatest potential to fill and resolve a highest number of slots, an estimation of a user expected query, and a selection criteria calculated by the one or more programs.

14. The computing system of claim 12, wherein the intent template includes a trigger condition definition and an action definition, one or more of the slots being included in each of the trigger condition definition and the action definition, each slot configured to contain information to define the trigger condition or action, respectively, and wherein the rule that is generated is a rule to perform an action defined by the action definition upon an occurrence of a trigger defined by the trigger condition definition.

15. The computing system of claim 14, wherein

the slots have an information type selected from the group consisting of not required, required and implicitly missing, required and explicitly missing, required and available but unresolved, and required and available and not requiring resolution.

16. The computing system of claim 12, wherein the intent template is a first intent template, and wherein the first intent template is selected from among a plurality of candidate intent templates when it is determined, the system further being configured to:

determine that a second intent template of the plurality of candidate intent templates is a closer match to the user input and the user response to the query, based on slots filled and resolved within the loop;

change the first intent template to the second intent template with slots for information;

populate the second intent template based on the user input and the user response to the query; and

re-perform resolution on the populated second intent template.

17. The computing system of claim 12, wherein

the processor is further configured to, based on information collected by the system, determine that slot information occupies incorrect slots and move slot information to correct slots;

and/or

the processor is further configured to, based on information collected by the system, extract new information and replace existing slot information with the new information.

18. The computing system of claim 12, wherein implicit knowledge is derived by the system via the processor from data selected from the group consisting of user preferences, past usage of the user, identification of an entity, identification of an environment, placement of the entity in the environment, activities of the entity, placement of other entities in the environment, activities of the other entities, and crowd behavior.

19. The computing system of claim 12, wherein

the processor is further configured to perform resolution by:

execution of lookup resolvers that translate identifiers in the user input into internal representation data;

execution of anaphoric resolvers that resolve information based on antecedent or postcedent expression in a context of user input; and

/or

execution of deixis resolvers that resolve information for words that cannot be interpreted without additional contextual information.

20. A method executed by a computing device, comprising:

at a processor;

executing one or more programs of an intelligent digital assistant system to present a user interface to a user;

receiving natural language user input from the user via the user interface;

parsing the user input at an intent handler of the system to determine an intent template with slots for information;

performing the following actions in a loop executed for the intent template until all slots required by the intent template are both filled and resolved;

populating the slots in the intent template with collected information from the user input;

performing resolution on the intent template to at least partially resolve unresolved information, thereby resulting in an at least partially resolved intent template;

filling implicitly missing slot information where possible with implicit knowledge derived from previously filled slots and other context information available to the system;

if, in the at least partially resolved intent template, not all slots required by the intent template are both filled and resolved, then performing the following actions for each slot required by the intent template;

determining a state of each slot that is not both filled and resolved, the state of each slot that is not both filled and resolved selected from the group consisting of implicitly missing slot information, explicitly missing slot information, and available and unresolved slot information;

presenting a query to the user to fill or resolve a selected slot that is not both filled and resolved based on query selection criteria;

receiving a user response to the query;

based on the user response to the query, altering the state of the selected slot; and

re-executing the loop with the user input and user response to the query;

exiting the loop upon a determination that all slots required by the intent template are both filled and resolved;

generating a rule based upon the intent template with all required information being available and resolved; and

performing a defined action that applies the generated rule;

wherein

the query selection criteria are at least one selected from the group consisting of random selection of missing or unresolved slot information, an internal order of explicitly missing slot information that is required to be filled, a greatest potential to fill and resolve a highest number of slots, an estimation of a user expected query, and a selection criteria calculated by the one or more programs; and

the intent template includes a trigger condition definition and an action definition, one or more of the slots being included in each of the trigger condition definition and the action definition, each slot configured to contain information to define the trigger condition or action, respectively, and wherein the rule that is generated is a rule to perform an action defined by the action definition upon an occurrence of a trigger defined by the trigger condition definition.