System and method for detecting user attention

• US 10,076,705 B2
• Filed: 01/07/2014
• Issued: 09/18/2018
• Est. Priority Date: 09/15/2005
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Deshpande, Hrishikesh R., Chen, Ruxin
• Primary Examiner(s): Marandi, James R
• Application Number:
  US14/149,241
• Publication Number:
  US 20140121002A1
• Time in Patent Office: 1,715 Days
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/424 involving acoustic input signals, e.g. by using the results of pitch or rhythm extraction or voice recognition
• CPC Class Codes: A63F 13/49 Saving the game status; Pausing or ending the game
• CPC Class Codes: A63F 2300/1081 Input via voice recognition
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/636 involving process of starting or resuming a game
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: H04N 21/42203 sound input device, e.g. microphone
• CPC Class Codes: H04N 21/42204 User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor constructive details of casings for the remote control device H01H9/0235; user interfaces for controlling a tuning device of a television receiver through a remote control H03J9/00; remote control of peripheral devices connected to a television receiver through the remote control device of the television receiver H04B1/205
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/426 Internal components of the client ; Characteristics thereof H04N5/44 takes precedence
• CPC Class Codes: H04N 21/4333 Processing operations in response to a pause request
• CPC Class Codes: H04N 21/43632 involving a wired protocol, e.g. IEEE 1394 high-speed IEEE 1394 serial bus H04L12/40052
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/4415 : using biometric characteristics of the user, e.g. by voice recognition or fingerprint scanning cryptography using biological data H04L9/3231; authentication in networks using biometric H04L63/0861
• CPC Class Codes: H04N 21/44218 : Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45

Abstract

A system and method for conditioning execution of a control function on a determination of whether or not a person'"'"'s attention is directed toward a predetermined device. The method involves acquiring data concerning the activity of a person who is in the proximity of the device, the data being in the form of one or more temporal samples. One or more of the temporal samples is then analyzed to determine if the person'"'"'s activity during the time of the analyzed samples indicates that the person'"'"'s attention is not directed toward the device. The results of the determination are used to ascertain whether or not the control function should be performed.

Claims

1. A voice-controllable system that responds to a spoken command only when a camera has imaged a frontal face at the same time that the command was spoken, comprising:

a memory for storing image data generated by the camera; and

a processor configured to

access the image data stored in the memory and process the image data to determine whether or not the frontal face has been imaged;

provide a signal when a determination has been made that the frontal face has been imaged,

wherein the processor determines that the frontal face has been imaged by determining whether or not a first image includes a first frontal face, performing a first frontal face detection operation on a subsequent image when the first image does include the first frontal face, and performing a second frontal face detection operation, more computationally intensive than the first frontal face detection operation, on the subsequent image when the first image does not include the first frontal face, and

wherein when a multiple of frontal faces are detected in the first image one or more of the detected frontal faces is used as a template to perform a template-matching search for one or more frontal faces in a subsequent image, and when the template-matching search does not detect one or more frontal faces in the subsequent image the second, more computationally intensive, frontal face detection operation is performed on the subsequent image such that (i) the more computationally intensive, frontal face detection operation is performed on the entire subsequent image only after the template-matching search is performed on the entire subsequent image and the template-matching search does not detect one or more frontal faces in the subsequent image, and (ii) the more computationally intensive, frontal face detection operation is not performed on the subsequent image when the template-matching search does detect one or more frontal faces in the subsequent image.

2. The voice-controllable system according to claim 1, wherein the spoken command is a command to pause video being displayed on a display screen.

3. The voice-controllable system according to claim 1, wherein the system is a video game.

4. The voice-controllable system according to claim 3, wherein the video game comprises a console.

5. The voice-controllable system according to claim 4, wherein the camera is integrated within the console.

6. The voice-controllable system according to claim 3, wherein the video game comprises a microphone for detecting the spoken command.

7. The voice-controllable system according to claim 6, wherein the video game comprises a console and the microphone is integrated within the console.

8. The voice-controllable system according to claim 1, wherein when the processor performs the first frontal face detection operation on the subsequent image and the first frontal face detection operation does not detect a frontal face in the subsequent image, the processor then performs the second frontal face detection operation on the subsequent image.

9. The voice-controllable system according to claim 1, wherein the processor determines that the command was spoken by analyzing audio data in subsets of samples that correspond to periods of about 0.3 seconds or greater.