Position calibration for intelligent assistant computing device

• US 10,824,921 B2
• Filed: 12/05/2017
• Issued: 11/03/2020
• Est. Priority Date: 02/14/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Bathiche, Steven Nabil, Ribeiro, Flavio Protasio, Pradeep, Vivek
• Primary Examiner(s): Toatley, Jr., Gregory J
• Assistant Examiner(s): Dinh, Lynda
• Application Number:
  US15/832,672
• Publication Number:
  US 20180233145A1
• Time in Patent Office: 1,064 Days
• CPC Class Codes: A61B 5/0205 Simultaneously evaluating both cardiovascular conditions and different types of body conditions, e.g. heart and respiratory condition
• CPC Class Codes: A61B 5/05 Detecting, measuring or recording for diagnosis by means of electric currents or magnetic fields; Measuring using microwaves or radio waves impedance plethysmography A61B5/0295; measuring movement of the entire body or parts thereof A61B5/11; detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof A61B5/24
• CPC Class Codes: A61B 5/0507 using microwaves or terahertz waves
• CPC Class Codes: A61B 5/1113 Local tracking of patients, e.g. in a hospital or private home
• CPC Class Codes: A61B 5/1118 Determining activity level
• CPC Class Codes: A61B 5/117 Identification of persons methods or arrangements for recognising patterns, e.g. fingerprints, G06F18/00, G06V40/00; identification of persons by analysing their voice or speech G10L17/00
• CPC Class Codes: A61B 5/7475 User input or interface means, e.g. keyboard, pointing device, joystick
• CPC Class Codes: G01S 11/14 using ultrasonic, sonic, or infrasonic waves
• CPC Class Codes: G01S 13/38 wherein more than one modulation frequency is used
• CPC Class Codes: G01S 13/726 Multiple target tracking
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/888 through wall detection
• CPC Class Codes: G01S 5/16 using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 5/18 using ultrasonic, sonic, or infrasonic waves
• CPC Class Codes: G01S 5/28 by co-ordinating position lines of different shape, e.g. hyperbolic, circular, elliptical or radial
• CPC Class Codes: G06F 1/3206 Monitoring of events, devices or parameters that trigger a change in power modality
• CPC Class Codes: G06F 1/3231 Monitoring the presence, absence or movement of users
• CPC Class Codes: G06F 1/324 by lowering clock frequency
• CPC Class Codes: G06F 1/329 by task scheduling
• CPC Class Codes: G06F 16/70 of video data
• CPC Class Codes: G06F 18/00 : Pattern recognition
• CPC Class Codes: G06F 18/251 : of input or preprocessed data
• CPC Class Codes: G06F 18/28 : Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06F 18/29 : Graphical models, e.g. Bayesian networks
• CPC Class Codes: G06F 18/41 : Interactive pattern learning with a human teacher
• CPC Class Codes: G06F 21/00 : Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
• CPC Class Codes: G06F 21/32 : using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06F 21/35 : communicating wirelessly
• CPC Class Codes: G06F 2203/0381 : Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 2221/2111 : Location-sensitive, e.g. geographical location, GPS
• CPC Class Codes: G06F 2221/2117 : User registration
• CPC Class Codes: G06F 3/011 : Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 : Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 : Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0482 : Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 : Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0488 : using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/167 : Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/211 : Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
• CPC Class Codes: G06F 40/35 : Discourse or dialogue representation
• CPC Class Codes: G06N 20/00 : Machine learning
• CPC Class Codes: G06N 3/044 : Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 5/025 : Extracting rules from data
• CPC Class Codes: G06N 5/047 : Pattern matching networks; Rete networks
• CPC Class Codes: G06T 2207/10016 : Video; Image sequence
• CPC Class Codes: G06T 2207/10024 : Color image
• CPC Class Codes: G06T 2207/20101 : Interactive definition of point of interest, landmark or seed
• CPC Class Codes: G06T 2207/30196 : Human being; Person
• CPC Class Codes: G06T 2207/30201 : Face
• CPC Class Codes: G06T 2207/30204 : Marker
• CPC Class Codes: G06T 2207/30232 : Surveillance
• CPC Class Codes: G06T 7/248 : involving reference images or patches
• CPC Class Codes: G06T 7/292 : Multi-camera tracking
• CPC Class Codes: G06T 7/60 : Analysis of geometric attributes
• CPC Class Codes: G06T 7/70 : Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/74 : involving reference images or patches
• CPC Class Codes: G06V 10/94 : Hardware or software architectures specially adapted for image or video understanding
• CPC Class Codes: G06V 20/40 : in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/52 : Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/653 : by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 2201/05 : Recognition of patterns representing particular kinds of hidden objects, e.g. weapons, explosives, drugs
• CPC Class Codes: G06V 30/274 : Syntactic or semantic context, e.g. balancing
• CPC Class Codes: G06V 40/10 : Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/103 : Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/166 : using acquisition arrangements
• CPC Class Codes: G06V 40/167 : using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/172 : Classification, e.g. identification
• CPC Class Codes: G06V 40/173 : face re-identification, e.g. recognising unknown faces across different face tracks
• CPC Class Codes: G06V 40/23 : Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: G06V 40/25 : Recognition of walking or running movements, e.g. gait recognition
• CPC Class Codes: G07C 9/28 : the pass enabling tracking or indicating presence
• CPC Class Codes: G07C 9/32 : in combination with an identity check
• CPC Class Codes: G08B 13/1427 : with transmitter-receiver for distance detection
• CPC Class Codes: G08B 29/186 : Fuzzy logic; neural networks
• CPC Class Codes: G10L 15/00 : Speech recognition G10L17/00 takes precedence
• CPC Class Codes: G10L 15/02 : Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/063 : Training
• CPC Class Codes: G10L 15/08 : Speech classification or search
• CPC Class Codes: G10L 15/18 : using natural language modelling
• CPC Class Codes: G10L 15/1815 : Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/1822 : Parsing for meaning understanding
• CPC Class Codes: G10L 15/19 : Grammatical context, e.g. disambiguation of the recognition hypotheses based on word sequence rules
• CPC Class Codes: G10L 15/22 : Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/24 : Speech recognition using non-acoustical features
• CPC Class Codes: G10L 15/26 : Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 15/28 : Constructional details of speech recognition systems
• CPC Class Codes: G10L 15/32 : Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems
• CPC Class Codes: G10L 17/00 : Speaker identification or verification techniques
• CPC Class Codes: G10L 17/04 : Training, enrolment or model building
• CPC Class Codes: G10L 17/06 : Decision making techniques; Pattern matching strategies
• CPC Class Codes: G10L 17/08 : Use of distortion metrics or a particular distance between probe pattern and reference templates
• CPC Class Codes: G10L 17/22 : Interactive procedures; Man-machine interfaces
• CPC Class Codes: G10L 2015/0635 : updating or merging of old and new templates; Mean values; Weighting
• CPC Class Codes: G10L 2015/088 : Word spotting
• CPC Class Codes: G10L 2015/223 : Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/225 : Feedback of the input speech
• CPC Class Codes: G10L 2015/228 : of application context
• CPC Class Codes: G10L 25/51 : for comparison or discrimination
• CPC Class Codes: H04L 51/02 : using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
• CPC Class Codes: H04L 63/102 : Entity profiles
• CPC Class Codes: H04L 67/12 : specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 67/535 : Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34
• CPC Class Codes: H04N 21/231 : Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion
• CPC Class Codes: H04N 21/42203 : sound input device, e.g. microphone
• CPC Class Codes: H04N 21/44218 : Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/44222 : Analytics of user selections, e.g. selection of programs or purchase activity monitoring of user selections in data processing systems G06F11/34; arrangements for monitoring the user's behaviour or opinions in broadcast systems H04H60/33
• CPC Class Codes: H04N 21/44224 : Monitoring of user activity on external systems, e.g. Internet browsing
• CPC Class Codes: H04N 23/11 : for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/611 : where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/188 : Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position signal generation from motion picture films H04N5/253
• CPC Class Codes: H04R 1/406 : microphones
• CPC Class Codes: H04R 3/005 : for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407
• CPC Class Codes: H04W 4/029 : Location-based management or tracking services
• CPC Class Codes: H04W 4/33 : for indoor environments, e.g. buildings
• CPC Class Codes: Y02D 10/00 : Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

A first intelligent assistant computing device configured to receive and respond to natural language inputs provided by human users syncs to a reference clock of a wireless computer network. The first intelligent assistant computing device receives a communication sent by a second intelligent assistant computing device indicating a signal emission time at which the second intelligent assistant computing device emitted a position calibration signal. The first intelligent assistant computing device records a signal detection time at which the position calibration signal was detected. Based on a difference between 1) the signal emission time and the signal detection time, and 2) a known propagation speed of the position calibration signal, a distance between the first and second intelligent assistant computing devices is calculated.

Claims

1. A method, comprising:

at a first intelligent assistant computing device configured to receive and respond to natural language inputs provided by human users, syncing to a reference clock of a wireless computer network, the reference clock being supported by a second intelligent assistant computing device communicatively coupled to the first intelligent assistant computing device via the wireless computer network;

receiving a communication sent by the second intelligent assistant computing device over the wireless computer network, the communication indicating a signal emission time at which the second intelligent assistant computing device emitted a position calibration signal, the signal emission time being defined relative to the reference clock;

recording a signal detection time at which the position calibration signal was detected by the first intelligent assistant computing device, the signal detection time being defined relative to the reference clock; and

based at least on a difference between the signal emission time and the signal detection time and a known propagation speed of the position calibration signal, calculating a distance between the first intelligent assistant computing device and the second intelligent assistant computing device.

2. The method of claim 1, where the position calibration signal is an emission of sound.

3. The method of claim 2, where the emission of sound includes a plurality of different frequencies, one or more of the plurality of frequencies being an ultrasonic frequency.

4. The method of claim 2, further comprising recording an amplitude of the detected position calibration signal, and where calculating the distance further includes comparing the recorded amplitude to a known emission amplitude of the position calibration signal.

5. The method of claim 2, where the first intelligent assistant computing device detects the emission of sound via a beamforming microphone array, the method further comprises recording a direction from which the emission of sound was detected.

6. The method of claim 2, where recording the signal detection time includes recording a first signal detection time of a first detection of the emission of sound and a second signal detection time of a second detection of the emission of sound, the second detection of the emission of sound corresponding to a reflection of the emission of sound off a reflection surface in an environment of the first intelligent assistant computing device, the method further comprises calculating a cumulative distance between a) the second intelligent assistant computing device and the reflection surface, and b) the reflection surface and the first intelligent assistant computing device.

7. The method of claim 6, further comprising calculating a plurality of distances between the first intelligent assistant computing device and a plurality of reflection surfaces in the environment based on a plurality of detections of the emission of sound, and estimating a layout of the environment based on the plurality of distances.

8. The method of claim 1, further comprising, upon detecting a change in position of the first intelligent assistant computing device, calculating an updated distance between the first and second intelligent assistant computing devices.

9. The method of claim 8, where the change in position of the first intelligent assistant computing device is detected by one or more motion sensors of the first intelligent assistant computing device, and the updated distance is calculated based on data output by the one or more motion sensors.

10. The method of claim 8, where the first intelligent assistant computing device includes a camera, detecting the change in position of the first intelligent assistant computing device includes detecting a change in images captured by the camera, and the updated distance is calculated based on the detected change.

11. The method of claim 8, further comprising, upon detecting the change in position of the first intelligent assistant computing device, instructing the second intelligent assistant computing device to emit a new position calibration signal.

12. The method of claim 1, further comprising:

detecting a natural language input provided by a human user;

based on the distance between the first and second intelligent assistant computing devices, determining which of the first and second intelligent assistant computing devices is closer to a position of the human user; and

responding to the natural language input via the closer intelligent assistant computing device.

13. A method, comprising:

at a first intelligent assistant computing device, emitting a position calibration signal via a signal emitter, the position calibration signal being an emission of light;

at a second intelligent assistant computing device, via one or more cameras, recording a set of parameters describing a detection of the position calibration signal;

based on the set of parameters describing the detection, estimating relative positions of the first and second intelligent assistant computing devices;

receiving a natural language input from a human user in an environment of the first and second intelligent assistant computing devices;

based on the relative positions of the first and second intelligent assistant computing devices, determining which of the first and second intelligent assistant computing devices is closer to a position of the human user; and

responding to the natural language input via the closer intelligent assistant computing device.

14. The method of claim 13, where the signal emitter is an infrared (IR) light emitter, and the emission of light is an emission of IR light.

15. The method of claim 13, where the first intelligent assistant computing device includes a set of signal emitters having a known spatial relationship, emitting the position calibration signal includes emitting a separate instance of the position calibration signal from each of the set of signal emitters and the second intelligent assistant computing device records multiple detections of the position calibration signal corresponding to separate instances of the position calibration signal, the method further comprises, based on comparing a spatial relationship of the multiple detections to the known spatial relationship of the set of signal emitters, calculating an orientation of the first intelligent assistant computing device relative to the second intelligent assistant computing device.

16. The method of claim 15, where emitting the position calibration signal includes emitting at least one instance of the position calibration signal differently from other instances of the position calibration signal.

17. The method of claim 13, further comprising recording a set of parameters describing a second detection of the position calibration signal corresponding to a reflection of the position calibration signal off a reflection surface in the environment, the method further comprises, based on the set of parameters describing the second detection, estimating a relative position of the reflection surface.

18. A method, comprising:

at a first intelligent assistant computing device, detecting presence of a human user at a first detection position within a field-of-detection (FOD) of the first intelligent assistant computing device;

localizing the human user to a real-world position relative to an environment of the first intelligent assistant computing device corresponding to the first detection position;

via a wireless computer network, receiving, from a second intelligent assistant computing device, an indication that the second intelligent assistant computing device detected the human user at a second detection position within a FOD of the second intelligent assistant computing device, the second detection position corresponding to the real-world position; and

using the real-world position of the human user as a landmark, estimating real-world positions for each of the first and second intelligent assistant computing devices.

19. The method of claim 18, further comprising:

detecting a natural language input provided by the human user;

comparing the real-world positions of the first and second intelligent assistant computing devices to the real-world position of the human user to identify which of the first and second intelligent assistant computing devices are closer to the human user; and

responding to the natural language input via the closer intelligent assistant computing device.

20. The method of claim 18, further comprising, upon detecting a change in position of the first intelligent assistant computing device, calculating an updated distance between the first and second intelligent assistant computing devices.