Natural language interaction for smart assistant

• US 11,126,825 B2
• Filed: 10/11/2019
• Issued: 09/21/2021
• Est. Priority Date: 02/14/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Herold, Keith Coleman, Solomon, Oz
• Primary Examiner(s): Abebe, Daniel
• Application Number:
  US16/599,426
• Publication Number:
  US 20200042839A1
• Time in Patent Office: 711 Days
• CPC Class Codes: A61B 5/0205 Simultaneously evaluating both cardiovascular conditions and different types of body conditions, e.g. heart and respiratory condition
• CPC Class Codes: A61B 5/05 Detecting, measuring or recording for diagnosis by means of electric currents or magnetic fields; Measuring using microwaves or radio waves impedance plethysmography A61B5/0295; measuring movement of the entire body or parts thereof A61B5/11; detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof A61B5/24
• CPC Class Codes: A61B 5/0507 using microwaves or terahertz waves
• CPC Class Codes: A61B 5/1113 Local tracking of patients, e.g. in a hospital or private home
• CPC Class Codes: A61B 5/1118 Determining activity level
• CPC Class Codes: A61B 5/117 Identification of persons methods or arrangements for recognising patterns, e.g. fingerprints, G06F18/00, G06V40/00; identification of persons by analysing their voice or speech G10L17/00
• CPC Class Codes: A61B 5/7475 User input or interface means, e.g. keyboard, pointing device, joystick
• CPC Class Codes: G01S 11/14 using ultrasonic, sonic, or infrasonic waves
• CPC Class Codes: G01S 13/38 wherein more than one modulation frequency is used
• CPC Class Codes: G01S 13/726 Multiple target tracking
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/888 through wall detection
• CPC Class Codes: G01S 5/16 using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 5/18 using ultrasonic, sonic, or infrasonic waves
• CPC Class Codes: G01S 5/28 by co-ordinating position lines of different shape, e.g. hyperbolic, circular, elliptical or radial
• CPC Class Codes: G06F 1/3206 Monitoring of events, devices or parameters that trigger a change in power modality
• CPC Class Codes: G06F 1/3231 Monitoring the presence, absence or movement of users
• CPC Class Codes: G06F 1/324 by lowering clock frequency
• CPC Class Codes: G06F 1/329 by task scheduling
• CPC Class Codes: G06F 16/70 of video data
• CPC Class Codes: G06F 18/00 : Pattern recognition
• CPC Class Codes: G06F 18/251 : of input or preprocessed data
• CPC Class Codes: G06F 18/28 : Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06F 18/29 : Graphical models, e.g. Bayesian networks
• CPC Class Codes: G06F 18/41 : Interactive pattern learning with a human teacher
• CPC Class Codes: G06F 21/00 : Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
• CPC Class Codes: G06F 21/32 : using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06F 21/35 : communicating wirelessly
• CPC Class Codes: G06F 2203/0381 : Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 2221/2111 : Location-sensitive, e.g. geographical location, GPS
• CPC Class Codes: G06F 2221/2117 : User registration
• CPC Class Codes: G06F 3/011 : Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 : Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 : Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0482 : Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 : Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0488 : using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/167 : Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/211 : Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
• CPC Class Codes: G06F 40/35 : Discourse or dialogue representation
• CPC Class Codes: G06N 20/00 : Machine learning
• CPC Class Codes: G06N 3/044 : Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 5/025 : Extracting rules from data
• CPC Class Codes: G06N 5/047 : Pattern matching networks; Rete networks
• CPC Class Codes: G06T 2207/10016 : Video; Image sequence
• CPC Class Codes: G06T 2207/10024 : Color image
• CPC Class Codes: G06T 2207/20101 : Interactive definition of point of interest, landmark or seed
• CPC Class Codes: G06T 2207/30196 : Human being; Person
• CPC Class Codes: G06T 2207/30201 : Face
• CPC Class Codes: G06T 2207/30204 : Marker
• CPC Class Codes: G06T 2207/30232 : Surveillance
• CPC Class Codes: G06T 7/248 : involving reference images or patches
• CPC Class Codes: G06T 7/292 : Multi-camera tracking
• CPC Class Codes: G06T 7/60 : Analysis of geometric attributes
• CPC Class Codes: G06T 7/70 : Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/74 : involving reference images or patches
• CPC Class Codes: G06V 10/94 : Hardware or software architectures specially adapted for image or video understanding
• CPC Class Codes: G06V 20/40 : in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/52 : Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/653 : by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 2201/05 : Recognition of patterns representing particular kinds of hidden objects, e.g. weapons, explosives, drugs
• CPC Class Codes: G06V 30/274 : Syntactic or semantic context, e.g. balancing
• CPC Class Codes: G06V 40/10 : Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/103 : Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/166 : using acquisition arrangements
• CPC Class Codes: G06V 40/167 : using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/172 : Classification, e.g. identification
• CPC Class Codes: G06V 40/173 : face re-identification, e.g. recognising unknown faces across different face tracks
• CPC Class Codes: G06V 40/23 : Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: G06V 40/25 : Recognition of walking or running movements, e.g. gait recognition
• CPC Class Codes: G07C 9/28 : the pass enabling tracking or indicating presence
• CPC Class Codes: G07C 9/32 : in combination with an identity check
• CPC Class Codes: G08B 13/1427 : with transmitter-receiver for distance detection
• CPC Class Codes: G08B 29/186 : Fuzzy logic; neural networks
• CPC Class Codes: G10L 15/00 : Speech recognition G10L17/00 takes precedence
• CPC Class Codes: G10L 15/02 : Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/063 : Training
• CPC Class Codes: G10L 15/08 : Speech classification or search
• CPC Class Codes: G10L 15/18 : using natural language modelling
• CPC Class Codes: G10L 15/1815 : Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/1822 : Parsing for meaning understanding
• CPC Class Codes: G10L 15/19 : Grammatical context, e.g. disambiguation of the recognition hypotheses based on word sequence rules
• CPC Class Codes: G10L 15/22 : Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/24 : Speech recognition using non-acoustical features
• CPC Class Codes: G10L 15/26 : Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 15/28 : Constructional details of speech recognition systems
• CPC Class Codes: G10L 15/32 : Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems
• CPC Class Codes: G10L 17/00 : Speaker identification or verification techniques
• CPC Class Codes: G10L 17/04 : Training, enrolment or model building
• CPC Class Codes: G10L 17/06 : Decision making techniques; Pattern matching strategies
• CPC Class Codes: G10L 17/08 : Use of distortion metrics or a particular distance between probe pattern and reference templates
• CPC Class Codes: G10L 17/22 : Interactive procedures; Man-machine interfaces
• CPC Class Codes: G10L 2015/0635 : updating or merging of old and new templates; Mean values; Weighting
• CPC Class Codes: G10L 2015/088 : Word spotting
• CPC Class Codes: G10L 2015/223 : Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/225 : Feedback of the input speech
• CPC Class Codes: G10L 2015/228 : of application context
• CPC Class Codes: G10L 25/51 : for comparison or discrimination
• CPC Class Codes: H04L 51/02 : using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
• CPC Class Codes: H04L 63/102 : Entity profiles
• CPC Class Codes: H04L 67/12 : specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 67/535 : Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34
• CPC Class Codes: H04N 21/231 : Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion
• CPC Class Codes: H04N 21/42203 : sound input device, e.g. microphone
• CPC Class Codes: H04N 21/44218 : Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/44222 : Analytics of user selections, e.g. selection of programs or purchase activity monitoring of user selections in data processing systems G06F11/34; arrangements for monitoring the user's behaviour or opinions in broadcast systems H04H60/33
• CPC Class Codes: H04N 21/44224 : Monitoring of user activity on external systems, e.g. Internet browsing
• CPC Class Codes: H04N 23/11 : for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/611 : where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/188 : Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position signal generation from motion picture films H04N5/253
• CPC Class Codes: H04R 1/406 : microphones
• CPC Class Codes: H04R 3/005 : for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407
• CPC Class Codes: H04W 4/029 : Location-based management or tracking services
• CPC Class Codes: H04W 4/33 : for indoor environments, e.g. buildings
• CPC Class Codes: Y02D 10/00 : Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

A method for natural language interaction includes recording speech provided by a human user. The recorded speech is translated into a machine-readable natural language input relating to an interaction topic. An interaction timer is maintained that tracks a length of time since a last machine-readable natural language input referring to the interaction topic was translated. Based on a current value of the interaction timer being greater than an interaction engagement threshold, a message relating to the interaction topic is delivered with a first natural language phrasing that includes an interaction topic reminder. Based on the current value of the interaction timer being less than the interaction engagement threshold, the message relating to the interaction topic is delivered with a second natural language phrasing that lacks the interaction topic reminder.

Claims

1. A method for natural language interaction, comprising:

receiving sensor data via a network;

translating the sensor data into a machine-readable natural language input relating to an interaction topic;

maintaining an interaction timer tracking a length of time since a last machine-readable natural language input relating to the interaction topic;

based on a current value of the interaction timer being greater than an interaction engagement threshold, outputting a message relating to the interaction topic with a first natural language phrasing that includes an interaction topic reminder, where the interaction engagement threshold is dynamically adjusted based on one or more user engagement factors;

or

based on the current value of the interaction timer being less than the interaction engagement threshold, outputting the message relating to the interaction topic with a second natural language phrasing that lacks the interaction topic reminder.

2. The method of claim 1, where the first natural language phrasing includes more words than the second natural language phrasing.

3. The method of claim 1, where the interaction topic reminder includes a summary of a most recent interaction with a human user relating to the interaction topic.

4. The method of claim 1, where the one or more user engagement factors include the interaction topic.

5. The method of claim 1, where the one or more user engagement factors include a current time of day.

6. The method of claim 1, where the one or more user engagement factors include an identity of a human user who provided the sensor data translated into the machine-readable natural language input.

7. The method of claim 1, where the one or more user engagement factors include a language spoken by a human user when providing the sensor data translated into the machine-readable natural language input.

8. The method of claim 1, where the interaction engagement threshold is reduced based on determining that a human user has left a local environment of a smart assistant device.

9. The method of claim 1, where the interaction engagement threshold is reduced based on determining that a human user has begun a new activity since the sensor data was translated into the machine-readable natural language input.

10. The method of claim 1, where the interaction engagement threshold is reduced based on determining that a human user who provided the sensor data translated into the machine-readable natural language input is interacting with one or more other humans.

11. The method of claim 1, where the second natural language phrasing includes one or more non-specific pronouns that implicitly refer to the interaction topic.

12. The method of claim 1, where the interaction topic reminder includes one or more words explicitly referring to the interaction topic.

13. The method of claim 1, where the interaction topic reminder refers to the interaction topic using at least one proper noun.

14. A method for natural language interaction, comprising:

receiving sensor data;

translating the sensor data into a machine-readable natural language input;

resolving an ambiguity in the machine-readable natural language input to determine an interaction topic;

maintaining an interaction timer tracking a length of time since a last machine-readable natural language input relating to the interaction topic;

based on a current value of the interaction timer being greater than an interaction engagement threshold, outputting a message relating to the interaction topic with a first natural language phrasing that includes an interaction topic reminder, where the interaction engagement threshold is dynamically adjusted based on one or more user engagement factors;

or

based on the current value of the interaction timer being less than the interaction engagement threshold, outputting the message relating to the interaction topic with a second natural language phrasing that lacks the interaction topic reminder.

15. The method of claim 14, where resolving the ambiguity in the machine-readable natural language input includes analyzing prior utterances of a human user to identify information missing in the machine-readable natural language input.

16. The method of claim 14, where the first natural language phrasing includes more words than the second natural language phrasing.

17. The method of claim 14, where the interaction topic reminder includes a summary of a most recent interaction with a human user relating to the interaction topic.

18. A computing device, comprising:

a logic machine; and

a storage machine holding instructions executable by the logic machine to;

receive sensor data;

translate the sensor data into a machine-readable natural language input;

maintain an interaction timer tracking a length of time since a last machine-readable natural language input relating to the interaction topic;

based on a current value of the interaction timer being greater than an interaction engagement threshold, output a message relating to the interaction topic with a first natural language phrasing that includes an interaction topic reminder, where the interaction engagement threshold is dynamically adjusted based on one or more user engagement factors;

or

based on the current value of the interaction timer being less than the interaction engagement threshold, output the message relating to the interaction topic with a second natural language phrasing that lacks the interaction topic reminder.