USER REGISTRATION FOR INTELLIGENT ASSISTANT COMPUTER

• US 20180232201A1
• Filed: 08/21/2017
• Published: 08/16/2018
• Est. Priority Date: 02/14/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: HOLTMANN, Michelle Lynn
• Granted Patent: US 10,579,912 B2
• Time in Patent Office: Days
• CPC Class Codes: A61B 5/0205 Simultaneously evaluating both cardiovascular conditions and different types of body conditions, e.g. heart and respiratory condition
• CPC Class Codes: A61B 5/05 Detecting, measuring or recording for diagnosis by means of electric currents or magnetic fields; Measuring using microwaves or radio waves impedance plethysmography A61B5/0295; measuring movement of the entire body or parts thereof A61B5/11; detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof A61B5/24
• CPC Class Codes: A61B 5/0507 using microwaves or terahertz waves
• CPC Class Codes: A61B 5/1113 Local tracking of patients, e.g. in a hospital or private home
• CPC Class Codes: A61B 5/1118 Determining activity level
• CPC Class Codes: A61B 5/117 Identification of persons methods or arrangements for recognising patterns, e.g. fingerprints, G06F18/00, G06V40/00; identification of persons by analysing their voice or speech G10L17/00
• CPC Class Codes: A61B 5/7475 User input or interface means, e.g. keyboard, pointing device, joystick
• CPC Class Codes: G01S 11/14 using ultrasonic, sonic, or infrasonic waves
• CPC Class Codes: G01S 13/38 wherein more than one modulation frequency is used
• CPC Class Codes: G01S 13/726 Multiple target tracking
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/888 through wall detection
• CPC Class Codes: G01S 5/16 using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 5/18 using ultrasonic, sonic, or infrasonic waves
• CPC Class Codes: G01S 5/28 by co-ordinating position lines of different shape, e.g. hyperbolic, circular, elliptical or radial
• CPC Class Codes: G06F 1/3206 Monitoring of events, devices or parameters that trigger a change in power modality
• CPC Class Codes: G06F 1/3231 Monitoring the presence, absence or movement of users
• CPC Class Codes: G06F 1/324 by lowering clock frequency
• CPC Class Codes: G06F 1/329 by task scheduling
• CPC Class Codes: G06F 16/70 of video data
• CPC Class Codes: G06F 18/00 : Pattern recognition
• CPC Class Codes: G06F 18/251 : of input or preprocessed data
• CPC Class Codes: G06F 18/28 : Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06F 18/29 : Graphical models, e.g. Bayesian networks
• CPC Class Codes: G06F 18/41 : Interactive pattern learning with a human teacher
• CPC Class Codes: G06F 21/00 : Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
• CPC Class Codes: G06F 21/32 : using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06F 21/35 : communicating wirelessly
• CPC Class Codes: G06F 2203/0381 : Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 2221/2111 : Location-sensitive, e.g. geographical location, GPS
• CPC Class Codes: G06F 2221/2117 : User registration
• CPC Class Codes: G06F 3/011 : Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 : Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 : Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0482 : Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 : Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/0488 : using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/167 : Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 40/211 : Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
• CPC Class Codes: G06F 40/35 : Discourse or dialogue representation
• CPC Class Codes: G06N 20/00 : Machine learning
• CPC Class Codes: G06N 3/044 : Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 5/025 : Extracting rules from data
• CPC Class Codes: G06N 5/047 : Pattern matching networks; Rete networks
• CPC Class Codes: G06T 2207/10016 : Video; Image sequence
• CPC Class Codes: G06T 2207/10024 : Color image
• CPC Class Codes: G06T 2207/20101 : Interactive definition of point of interest, landmark or seed
• CPC Class Codes: G06T 2207/30196 : Human being; Person
• CPC Class Codes: G06T 2207/30201 : Face
• CPC Class Codes: G06T 2207/30204 : Marker
• CPC Class Codes: G06T 2207/30232 : Surveillance
• CPC Class Codes: G06T 7/248 : involving reference images or patches
• CPC Class Codes: G06T 7/292 : Multi-camera tracking
• CPC Class Codes: G06T 7/60 : Analysis of geometric attributes
• CPC Class Codes: G06T 7/70 : Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/74 : involving reference images or patches
• CPC Class Codes: G06V 10/94 : Hardware or software architectures specially adapted for image or video understanding
• CPC Class Codes: G06V 20/40 : in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/52 : Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/653 : by matching three-dimensional models, e.g. conformal mapping of Riemann surfaces
• CPC Class Codes: G06V 2201/05 : Recognition of patterns representing particular kinds of hidden objects, e.g. weapons, explosives, drugs
• CPC Class Codes: G06V 30/274 : Syntactic or semantic context, e.g. balancing
• CPC Class Codes: G06V 40/10 : Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
• CPC Class Codes: G06V 40/103 : Static body considered as a whole, e.g. static pedestrian or occupant recognition
• CPC Class Codes: G06V 40/166 : using acquisition arrangements
• CPC Class Codes: G06V 40/167 : using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/172 : Classification, e.g. identification
• CPC Class Codes: G06V 40/173 : face re-identification, e.g. recognising unknown faces across different face tracks
• CPC Class Codes: G06V 40/23 : Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: G06V 40/25 : Recognition of walking or running movements, e.g. gait recognition
• CPC Class Codes: G07C 9/28 : the pass enabling tracking or indicating presence
• CPC Class Codes: G07C 9/32 : in combination with an identity check
• CPC Class Codes: G08B 13/1427 : with transmitter-receiver for distance detection
• CPC Class Codes: G08B 29/186 : Fuzzy logic; neural networks
• CPC Class Codes: G10L 15/00 : Speech recognition G10L17/00 takes precedence
• CPC Class Codes: G10L 15/02 : Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/063 : Training
• CPC Class Codes: G10L 15/08 : Speech classification or search
• CPC Class Codes: G10L 15/18 : using natural language modelling
• CPC Class Codes: G10L 15/1815 : Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
• CPC Class Codes: G10L 15/1822 : Parsing for meaning understanding
• CPC Class Codes: G10L 15/19 : Grammatical context, e.g. disambiguation of the recognition hypotheses based on word sequence rules
• CPC Class Codes: G10L 15/22 : Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/24 : Speech recognition using non-acoustical features
• CPC Class Codes: G10L 15/26 : Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 15/28 : Constructional details of speech recognition systems
• CPC Class Codes: G10L 15/32 : Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems
• CPC Class Codes: G10L 17/00 : Speaker identification or verification techniques
• CPC Class Codes: G10L 17/04 : Training, enrolment or model building
• CPC Class Codes: G10L 17/06 : Decision making techniques; Pattern matching strategies
• CPC Class Codes: G10L 17/08 : Use of distortion metrics or a particular distance between probe pattern and reference templates
• CPC Class Codes: G10L 17/22 : Interactive procedures; Man-machine interfaces
• CPC Class Codes: G10L 2015/0635 : updating or merging of old and new templates; Mean values; Weighting
• CPC Class Codes: G10L 2015/088 : Word spotting
• CPC Class Codes: G10L 2015/223 : Execution procedure of a spoken command
• CPC Class Codes: G10L 2015/225 : Feedback of the input speech
• CPC Class Codes: G10L 2015/228 : of application context
• CPC Class Codes: G10L 25/51 : for comparison or discrimination
• CPC Class Codes: H04L 51/02 : using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages
• CPC Class Codes: H04L 63/102 : Entity profiles
• CPC Class Codes: H04L 67/12 : specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 67/535 : Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34
• CPC Class Codes: H04N 21/231 : Content storage operation, e.g. caching movies for short term storage, replicating data over plural servers, prioritizing data for deletion
• CPC Class Codes: H04N 21/42203 : sound input device, e.g. microphone
• CPC Class Codes: H04N 21/44218 : Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/44222 : Analytics of user selections, e.g. selection of programs or purchase activity monitoring of user selections in data processing systems G06F11/34; arrangements for monitoring the user's behaviour or opinions in broadcast systems H04H60/33
• CPC Class Codes: H04N 21/44224 : Monitoring of user activity on external systems, e.g. Internet browsing
• CPC Class Codes: H04N 23/11 : for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/611 : where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/188 : Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position signal generation from motion picture films H04N5/253
• CPC Class Codes: H04R 1/406 : microphones
• CPC Class Codes: H04R 3/005 : for combining the signals of two or more microphones specially adapted for hearing aids H04R25/407
• CPC Class Codes: H04W 4/029 : Location-based management or tracking services
• CPC Class Codes: H04W 4/33 : for indoor environments, e.g. buildings
• CPC Class Codes: Y02D 10/00 : Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

Registration of a person with an intelligent assistant computer includes obtaining one or more image frames captured via one or more cameras that depict an initially unregistered person. Facial recognition data for the initially unregistered person is extracted from the one or more image frames. A spoken command to register the initially unregistered person is received via one or more microphones. Upon determining that the spoken command originated from the registered person having the pre-established registration privilege, the initially unregistered person is registered as a newly registered person by associating one or more additional privileges with the facial recognition data in a person profile for the newly registered person.

Claims

1. A method performed by a computing system for registering a person with an intelligent assistant computer, the method comprising:

obtaining one or more image frames captured via one or more cameras that depict an initially unregistered person;

extracting facial recognition data for the initially unregistered person from the one or more image frames;

receiving a spoken command to register the initially unregistered person via one or more microphones;

determining that the spoken command originated from a registered person having a pre-established registration privilege; and

upon determining that the spoken command originated from the registered person having the pre-established registration privilege, registering the initially unregistered person as a newly registered person by associating one or more additional privileges with the facial recognition data in a person profile for the newly registered person.

2. The method of claim 1, wherein the one or more image frames are captured after receiving the spoken command to register the initially unregistered person.

3. The method of claim 2, further comprising:

responsive to receiving the spoken command to register the initially unregistered person, directing the initially unregistered person to position its face within a field of view of the one or more cameras to capture the one or more image frames for facial recognition.

4. The method of claim 3, wherein directing the initially unregistered person includes one or more of outputting an audible direction via an audio speaker and/or outputting a visual direction via a graphical display device.

5. The method of claim 1, further comprising:

responsive to receiving the spoken command to register the initially unregistered person, directing the initially unregistered person to speak one or more words or phrases;

obtaining one or more audio segments captured via one or more microphones that include the one or more words or phrases spoken by the initially unregistered person;

extracting speaker recognition data from the one or more audio segments for the initially unregistered person; and

associating the speaker recognition data with the person profile for the newly registered person.

6. The method of claim 1, wherein the spoken command to register the initially unregistered person is received after the one or more image frames are captured via the one or more cameras.

7. The method of claim 6, further comprising:

storing the one or more image frames in a data storage system prior to receiving the spoken command to register the initially registered person;

retrieving the one or more image frames from the data storage system;

presenting the one or more image frames via a graphical display device for review by the registered person; and

wherein the spoken command is received during or after presentation of the one or more image frames.

8. The method of claim 7, wherein the one or more image frames are presented after the initially unregistered person leaves a field of view of the one or more cameras.

9. The method of claim 7, wherein the one or more image frames are presented responsive to another command initiated by the registered person.

10. The method of claim 1, wherein the one or more image frames form part of one or more video segments captured via the one or more cameras;

and

wherein the method further comprises;

identifying a speaking activity of the initially unregistered person within the one or more video segments;

obtaining one or more audio segments, captured via one or more microphones, that are time-matched to the one or more video segments;

extracting speaker recognition data for the initially unregistered person from the one or more audio segments based on one or more spoken words or phrases that correspond to the speaking activity of the initially unregistered person; and

associating the speaker recognition data with the person profile.

11. The method of claim 10, further comprising:

receiving a subsequent spoken command to perform one or more operations via one or more microphones;

determining that the subsequent spoken command originated from the newly registered person having the one or more additional privileges based on the speaker recognition data; and

performing an operation of the one or more operations that is permitted by the one or more additional privileges responsive to the spoken command.

12. The method of claim 1, wherein the spoken command forms part of a spoken phrase that originated from the registered person that further includes a person identifier for the newly registered person;

and

wherein the method further comprises associating the person identifier with the person profile for the newly registered person.

13. The method of claim 1, wherein the spoken command forms part of a spoken phrase that originated from the registered person that further includes a privileges identifier for the newly registered person that identifies the one or more additional privileges associated with the person profile;

wherein each privilege of the one or more additional privileges permits one or more operations, not previously permitted prior to registration, to be performed by the intelligent assistant computer responsive to a command originating from the newly registered person.

14. The method of claim 13, wherein the privilege identifier indicates whether the newly registered person is permitted to register other initially unregistered persons.

15. A computing system, comprising:

one or more cameras to capture image data;

one or more microphones to capture audio data;

one or more computing devices implementing an intelligent assistant service configured to;

obtain one or more image frames captured via the one or more cameras that depict an initially unregistered person;

extract facial recognition data for the initially unregistered person from the one or more image frames;

receive a spoken command to register the initially unregistered person via the one or more microphones;

determine that the spoken command originated from a registered person having a pre-established registration privilege; and

upon determining that the spoken command originated from the registered person having the pre-established registration privilege, register the initially unregistered person as a newly registered person by associating one or more additional privileges with the facial recognition data in a person profile for the newly registered person stored in a data storage system of the one or more computing devices.

16. The computing system of claim 15, wherein the one or more image frames are captured after receiving the spoken command to register the initially unregistered person;

and

wherein the intelligent assistant service is further configured to;

responsive to receiving the spoken command to register the initially unregistered person, direct the initially unregistered person to position its face within a field of view of the one or more cameras to capture the one or more image frames for facial recognition.

17. The computing system of claim 15, wherein the intelligent assistant service is further configured to:

responsive to receiving the spoken command to register the initially unregistered person, direct the initially unregistered person to speak one or more words or phrases;

obtain one or more audio segments captured via the one or more microphones that include the one or more words or phrases spoken by the initially unregistered person;

extract speaker recognition data from the one or more audio segments for the initially unregistered person; and

associate the speaker recognition data with the person profile for the newly registered person.

18. The computing system of claim 15, wherein the spoken command to register the initially unregistered person is received after the one or more image frames are captured via the one or more cameras;

and

wherein the intelligent assistant service is further configured to;

store the one or more image frames in the data storage system prior to receiving the spoken command to register the initially registered person;

retrieve the one or more image frames from the data storage system after the initially unregistered person leaves a field of view of the one or more cameras;

present the one or more image frames via a graphical display device for review by the registered person; and

wherein the spoken command is received during or after presentation of the one or more image frames.

19. The computing system of claim 15, wherein the intelligent assistant service is further configured to:

receive a subsequent spoken command to perform one or more operations via the one or more microphones;

determine that the subsequent spoken command originated from the newly registered person having the one or more additional privileges; and

perform an operation that is permitted by the one or more additional privileges responsive to the spoken command.

20. A method performed by a computing system for registering a person with an intelligent assistant computer, the method comprising:

obtaining one or more image frames captured via one or more cameras that depict an initially unregistered person;

extracting facial recognition data for the initially unregistered person from the one or more image frames;

obtaining one or more audio segments captured via one or more microphones that include one or more words and/or phrases spoken by the initially unregistered person;

extracting speaker recognition data for the initially unregistered person from the one or more audio segments;

receiving a spoken command to register the initially unregistered person via one or more microphones;

determining that the spoken command originated from a registered person having a pre-established registration privilege;

upon determining that the spoken command originated from the registered person having the pre-established registration privilege, registering the initially unregistered person as a newly registered person by associating one or more additional privileges with the facial recognition data and the voice recognition data in a person profile for the newly registered person;

following registration of the newly registered person, receiving a subsequent spoken command to perform one or more operations via the one or more microphones;

determining that the subsequent spoken command originated from the newly registered person having the one or more additional privileges based on the speaker recognition data; and

performing an operation that is permitted by the one or more additional privileges responsive to the spoken command.