System and method for detecting user attention

• US 8,645,985 B2
• Filed: 03/06/2006
• Issued: 02/04/2014
• Est. Priority Date: 09/15/2005
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Chen, Ruxin, Deshpande, Hrishikesh R.
• Primary Examiner(s): Chung, Jason J
• Assistant Examiner(s): Marandi, James R
• Application Number:
  US11/368,766
• Publication Number:
  US 20070061851A1
• Time in Patent Office: 2,892 Days
• Field of Search: None
• US Class Current: 725/10
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/424 involving acoustic input signals, e.g. by using the results of pitch or rhythm extraction or voice recognition
• CPC Class Codes: A63F 13/49 Saving the game status; Pausing or ending the game
• CPC Class Codes: A63F 2300/1081 Input via voice recognition
• CPC Class Codes: A63F 2300/1087 comprising photodetecting means, e.g. a camera
• CPC Class Codes: A63F 2300/636 involving process of starting or resuming a game
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: H04N 21/42203 sound input device, e.g. microphone
• CPC Class Codes: H04N 21/42204 User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor constructive details of casings for the remote control device H01H9/0235; user interfaces for controlling a tuning device of a television receiver through a remote control H03J9/00; remote control of peripheral devices connected to a television receiver through the remote control device of the television receiver H04B1/205
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/426 Internal components of the client ; Characteristics thereof H04N5/44 takes precedence
• CPC Class Codes: H04N 21/4333 Processing operations in response to a pause request
• CPC Class Codes: H04N 21/43632 involving a wired protocol, e.g. IEEE 1394 high-speed IEEE 1394 serial bus H04L12/40052
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/4415 : using biometric characteristics of the user, e.g. by voice recognition or fingerprint scanning cryptography using biological data H04L9/3231; authentication in networks using biometric H04L63/0861
• CPC Class Codes: H04N 21/44218 : Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45

Abstract

A system and method for conditioning execution of a control function on a determination of whether or not a person'"'"'s attention is directed toward a predetermined device. The method involves acquiring data concerning the activity of a person who is in the proximity of the device, the data being in the form of one or more temporal samples. One or more of the temporal samples is then analyzed to determine if the person'"'"'s activity during the time of the analyzed samples indicates that the person'"'"'s attention is not directed toward the device. The results of the determination are used to ascertain whether or not the control function should be performed.

Claims

1. A system for pausing video being displayed on a video display screen, comprising:

at least one sensor for acquiring data concerning the activity of a person who is in the proximity of the display screen, the data being in the form of one or more temporal samples; and

a processor for analyzing one or more of the temporal samples to determine if the person is in the vicinity of the display screen but has looked away from the screen for a predetermined period of time and has engaged in a conversation for a predetermined period of time, and automatically pausing the video being displayed on the display screen when it is determined that the person is in the vicinity of the display screen but has looked away from the screen for a predetermined period of time and has engaged in a conversation for a predetermined period of time,

a determination of whether or not a person in the vicinity of the display screen has looked away from the screen for a predetermined period of time includes searching successive images of the vicinity to detect the presence or absence of a frontal face corresponding to the person, wherein when a multiple of frontal faces are detected in one of the images each of the detected frontal faces is used as a template for searching for one or more frontal faces in at least the immediately succeeding image, and

a determination of whether or not a person in the vicinity of the display screen has engaged in a conversation for a predetermined period of time includes searching a plurality of successive subsets of audio data having a predetermined duration to detect the presence or absence of human voice activity in each subset and determining whether or not human voice activity is present in a predetermined consecutive number of the subsets,

the images that are searched when determining whether or not a person in the vicinity of the display screen has looked away from the screen are acquired by a camera that is a component of a video game system with which the person interacts through a handheld controller, and the camera is oriented such that the camera images an area within which the controller can be located while the game is in play.

2. The system according to claim 1, wherein the at least one sensor includes a microphone.

3. The system according to claim 2, wherein the data is digital audio samples.

4. The system according to claim 3, wherein the processor analyzes the spectral content of the data.

5. The system according to claim 4, the processor determines whether or not the data includes a human voice fundamental frequency in the range of 60 Hz to 350 Hz.

6. The system according to claim 3, wherein the processor applies one or more previously trained acoustic models to the data to detect whether or not the data indicates the presence of a human voice.

7. The system according to claim 1, wherein the temporal samples are pixel arrays, each temporal sample being an array of pixels that represents an image.

8. The system according to claim 7, wherein the processor analyzes at least one of the pixel arrays to determine whether or not the image corresponding to the pixel array includes a frontal face.

9. The system according to claim 8, wherein the processor analyzes at least one of the pixel arrays by selectively applying an intensive frontal face detection algorithm and a less intensive frontal face detection algorithm.

10. A method for pausing video being displayed on a video display screen, comprising the steps of:

acquiring data concerning the activity of a person who is in the proximity of the display screen, the data being in the form of one or more temporal samples;

analyzing one or more of the temporal samples to determine if the person is in the vicinity of the display screen but has looked away from the screen for a predetermined period of time and has engaged in a conversation for a predetermined period of time; and

automatically pausing the video being displayed on the screen when it is determined that the person is in the vicinity of the display screen but has looked away from the screen for a predetermined period of time and has engaged in a conversation for a predetermined period of time,

a determination of whether or not a person in the vicinity of the display screen has looked away from the screen for a predetermined period of time includes searching successive images of the vicinity to detect the presence or absence of a frontal face corresponding to the person, wherein when a multiple of frontal faces are detected in one of the images each of the detected frontal faces is used as a template for searching for one or more frontal faces in at least the immediately succeeding image, and

a determination of whether or not a person in the vicinity of the display screen has engaged in a conversation for a predetermined period of time includes searching a plurality of successive subsets of audio data having a predetermined duration to detect the presence or absence of human voice activity in each subset and determining whether or not human voice activity is present in a predetermined consecutive number of the subsets,

the images that are searched when determining whether or not a person in the vicinity of the display screen has looked away from the screen are acquired by a camera that is a component of a video game system with which the person interacts through a handheld controller, and the camera is oriented such that the camera images an area within which the controller can be located while the game is in play.

11. The method according to claim 10, wherein the step of acquiring data comprises the step of acquiring data through use of a microphone.

12. The method according to claim 11, wherein the step of acquiring data comprises the step of acquiring plurality of digital audio samples.

13. The method according to claim 12, wherein the step of analyzing comprises the step of analyzing the spectral content of the data.

14. The method according to claim 13, wherein the step of analyzing includes the step of determining whether or not the data includes a human voice fundamental frequency in the range of 60 Hz to 350 Hz.

15. The system according to claim 10, wherein the step of analyzing includes the step of applying one or more previously trained acoustic models to the data to detect whether or not the data indicates the presence of a human voice.

16. The method according to claim 10, wherein the step of acquiring data comprises the step of acquiring a plurality of pixel arrays representing images.

17. The method according to claim 16, wherein the step of analyzing comprises the step of analyzing at least one of the pixel arrays to determine whether or not an image corresponding to the pixel array includes a frontal face.

18. The method according to claim 17, wherein the step of analyzing includes selectively applying an intensive frontal face detection algorithm and a less intensive frontal face detection algorithm.

19. A system for pausing a video game, comprising:

at least one sensor for acquiring data concerning the activity of a person playing the video game, the game being displayed on a video display screen, and the data being in the form of one or more temporal samples; and

a processor for analyzing one or more of the temporal samples to determine if the person is in the vicinity of the display screen but has looked away from the screen for a predetermined period of time and has engaged in a conversation for a predetermined period of time, and automatically pausing the video being displayed on the screen when it is determined that the person is in the vicinity of the display screen but has looked away from the screen for a predetermined period of time and has engaged in a conversation for a predetermined period of time,

a determination of whether or not a person in the vicinity of the display screen has looked away from the screen for a predetermined period of time includes searching successive images of the vicinity to detect the presence or absence of a frontal face corresponding to the person, wherein when a multiple of frontal faces are detected in one of the images each of the detected frontal faces is used as a template for searching for one or more frontal faces in at least the immediately succeeding image, and

a determination of whether or not a person in the vicinity of the display screen has engaged in a conversation for a predetermined period of time includes searching a plurality of successive subsets of audio data having a predetermined duration to detect the presence or absence of human voice activity in each subset and determining whether or not human voice activity is present in a predetermined consecutive number of the subsets,

the images that are searched when determining whether or not a person in the vicinity of the display screen has looked away from the screen are acquired by a camera that is a component of a video game system with which the person interacts through a handheld controller, and the camera is oriented such that the camera images an area within which the controller can be located while the game is in play.

20. The system according to claim 19, wherein the at least one sensor includes a microphone.

21. A system for conditioning execution of a control function on a determination of whether or not a person'"'"'s attention is directed toward a predetermined device, comprising:

at least one sensor for acquiring data concerning the activity of a person who is in the proximity of the device, the data being in the form of one or more temporal samples; and

a processor for analyzing one or more of the temporal samples to make a determination as to whether or not the person is in the vicinity of the device but has looked away from the device for a predetermined period of time and has engaged in a conversation for a predetermined period of time, and selectively executing the control function when it is determined that the person is in the vicinity of the device but has looked away from the device for a predetermined period of time and has engaged in a conversation for a predetermined period of time,

a determination of whether or not a person in the vicinity of the device has looked away from the device for a predetermined period of time includes searching successive images of the vicinity to detect the presence or absence of a frontal face corresponding to the person, wherein when a multiple of frontal faces are detected in one of the images each of the detected frontal faces is used as a template for searching for one or more frontal faces in at least the immediately succeeding image, and

a determination of whether or not a person in the vicinity of the device has engaged in a conversation for a predetermined period of time includes searching a plurality of successive subsets of audio data having a predetermined duration to detect the presence or absence of human voice activity in each subset and determining whether or not human voice activity is present in a predetermined consecutive number of the subsets,

the images that are searched when determining whether or not a person in the vicinity of the device has looked away from the device are acquired by a camera that is a component of a video game system with which the person interacts through a handheld controller, and the camera is oriented such that the camera images an area within which the controller can be located while the game is in play.

22. The system according to claim 21, wherein execution of the control function is enabled only when the person'"'"'s attention is directed toward the device.

23. The system according to claim 22, wherein the control function is a function executed in response to a voice command.

24. The system according to claim 22, wherein the device is a display screen.

25. The system according to claim 21, wherein execution of the control function is enabled only when the person'"'"'s attention is not directed toward the device.

26. The system according to claim 25, wherein the control function is a function executed in response to a voice command.

27. The system according to claim 25, wherein the device is a display screen.