Audio, video, simulation, and user interface paradigms

• US 9,405,363 B2
• Filed: 08/13/2014
• Issued: 08/02/2016
• Est. Priority Date: 09/15/2005
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Chen, Ruxin, Hernandez-Abrego, Gustavo, Menendez-Pidal, Xavier, Osman, Steven, Deshpande, Rishi, Michaud-Wideman, Care, Marks, Richard, Larsen, Eric J., Mao, Xiaodong
• Primary Examiner(s): Godbold, Douglas
• Assistant Examiner(s): Villena, Mark
• Application Number:
  US14/459,248
• Publication Number:
  US 20140347272A1
• Time in Patent Office: 720 Days
• Field of Search: 345/156, 386/219, 725/10
• US Class Current: 1/1
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/217 using environment-related information, i.e. information generated otherwise than by the player, e.g. ambient temperature or humidity
• CPC Class Codes: A63F 13/428 involving motion or position input signals, e.g. signals representing the rotation of an input controller or a player's arm motions sensed by accelerometers or gyroscopes
• CPC Class Codes: G06F 2203/011 Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G10L 17/00 Speaker identification or verification techniques
• CPC Class Codes: G10L 17/04 Training, enrolment or model building

Abstract

Consumer electronic devices have been developed with enormous information processing capabilities, high quality audio and video outputs, large amounts of memory, and may also include wired and/or wireless networking capabilities. Additionally, relatively unsophisticated and inexpensive sensors, such as microphones, video camera, GPS or other position sensors, when coupled with devices having these enhanced capabilities, can be used to detect subtle features about users and their environments. A variety of audio, video, simulation and user interface paradigms have been developed to utilize the enhanced capabilities of these devices. These paradigms can be used separately or together in any combination. One paradigm automatically creating user identities using speaker identification. Another paradigm includes a control button with 3-axis pressure sensitivity for use with game controllers and other input devices.

Claims

1. A method comprising:

acquiring activity data of a user located within range of one or more sensors, the one or more sensors associated with a display screen displaying video content, the displayed video content having a volume level, the activity data being in the form of one or more temporal audio or video samples;

analyzing, using at least one processor operatively coupled with a memory, one or more of the temporal video samples to determine if the user has looked away from the display screen for a first predetermined period of time by searching successive images of the temporal video samples to detect a presence or absence of a frontal face corresponding to the user;

analyzing one or more of the temporal audio samples to determine if the user has had an emotional response relative to the activity on the display screen for a second predetermined period of time by searching a plurality of successive subsets of the temporal audio samples having a predetermined duration to detect a presence or absence of the user'"'"'s voice activity in each subset, and determining whether or not the user'"'"'s voice activity is present in a predetermined consecutive number of the subsets at a volume greater than a predetermined level;

determining that the user has looked away from the display screen for the first predetermined period of time based upon the analyzing of the one or more of the temporal video samples;

determining that the user has not had an emotional response relative to the activity on the display screen for the second predetermined period of time based upon the analyzing of the one or more of the temporal audio samples; and

automatically pausing the video content based upon the determining that the user has looked away from the display screen for the first predetermined period of time and based upon the determining that the user has not had an emotional response relative to the activity on the display screen for the second predetermined period of time.

2. The method of claim 1, wherein the one or more sensors include a microphone.

3. The method of claim 1, wherein the one or more sensors include a video camera.

4. The method of claim 1, wherein analyzing one or more of the temporal audio samples to determine if the user has had an emotional response relative to the activity on the display screen for the second predetermined period of time uses a noise and speaker recognition module configured to receive audio information from a microphone.

5. The method of claim 1, further comprising:

automatically adjusting the volume level of the video content based upon a determination that the user has looked away from the display screen for the first predetermined period of time.

6. The method of claim 1, further comprising:

automatically adjusting the volume level of the video content based upon a determination that the user has not had an emotional response relative to the activity on the display screen for the second predetermined period of time.

7. A system comprising:

one or more sensors configured for acquiring activity data of a user located within range of the one or more sensors associated with a display screen displaying video content, the displayed video content having a volume level, the activity data being in the form of one or more temporal audio or video samples;

a processor configured for analyzing one or more of the temporal video samples to determine if the user has looked away from the display screen for a first predetermined period of time by searching successive images of the temporal video samples to detect a presence or absence of a frontal face corresponding to the user;

the processor configured for analyzing one or more of the temporal audio samples to determine if the user has had an emotional response relative to the activity on the display screen for a second predetermined period of time by searching a plurality of successive subsets of the temporal audio samples having a predetermined duration to detect a presence or absence of the user'"'"'s voice activity in each subset, and determining whether or not the user'"'"'s voice activity is present in a predetermined consecutive number of the subsets at a volume greater than a predetermined level;

the processor configured for determining that the user has looked away from the display screen for the first predetermined period of time based upon the analyzing of the one or more of the temporal video samples;

the processor configured for determining that the user has not had an emotional response relative to the activity on the display screen for the second predetermined period of time based upon the analyzing of the one or more of the temporal audio samples; and

the processor configured for automatically pausing the video content based upon the determining that the user has looked away from the display screen for the first predetermined period of time and based upon the determining that the user has not had an emotional response relative to the activity on the display screen for the second predetermined period of time.

8. The system of claim 7, wherein the one or more sensors include a microphone.

9. The system of claim 7, wherein the one or more sensors include a video camera.

10. The system of claim 7, wherein the processor is configured for analyzing one or more of the temporal audio samples to determine if the user has had an emotional response relative to the activity on the display screen for the second predetermined period of time using a noise and speaker recognition module configured to receive audio information from a microphone.

11. The system of claim 7, wherein the processor is further configured for:

automatically adjusting the volume level of the video content based upon the determination by the processor that the user has looked away from the display screen for the first predetermined period of time.

12. The system of claim 7, wherein the processor is further configured for:

automatically adjusting the volume level of the video content based upon the determination by the processor that the user has not had an emotional response relative to the activity on the display screen for the second predetermined period of time.

13. A machine-readable non-transitory medium embodying information indicative of instructions for causing one or more machines to perform operations comprising:

acquiring activity data of a user located within range of one or more sensors, the one or more sensors associated with a display screen displaying video content, the displayed video content having a volume level, the activity data being in the form of one or more temporal audio or video samples;

analyzing one or more of the temporal video samples to determine if the user has looked away from the display screen for a first predetermined period of time by searching successive images of the temporal video samples to detect a presence or absence of a frontal face corresponding to the user;

analyzing one or more of the temporal audio samples to determine if the user has had an emotional response relative to the activity on the display screen for a second predetermined period of time by searching a plurality of successive subsets of the temporal audio samples having a predetermined duration to detect a presence or absence of the user'"'"'s voice activity in each subset, and determining whether or not the user'"'"'s voice activity is present in a predetermined consecutive number of the subsets at a volume greater than a predetermined level;

determining that the user has looked away from the display screen for the first predetermined period of time based upon the analyzing of the one or more of the temporal video samples;

determining that the user has not had an emotional response relative to the activity on the display screen for the second predetermined period of time based upon the analyzing of the one or more of the temporal audio samples; and

automatically pausing the video content based upon the determining that the user has looked away from the display screen for the first predetermined period of time and based upon the determining that the user has not had an emotional response relative to the activity on the display screen for the second predetermined period of time.

14. The medium of claim 13 wherein the one or more sensors include a microphone.

15. The medium of claim 13 wherein the one or more sensors include a video camera.

16. The medium of claim 13 wherein analyzing one or more of the temporal audio samples to determine if the user has had an emotional response relative to the activity on the display screen for the second predetermined period of time uses a noise and speaker recognition module configured to receive audio information from a microphone.

17. The medium of claim 13 further comprising instructions for:

automatically adjusting the volume level of the video content based upon the determination that the user has not had an emotional response relative to the activity on the display screen for the second predetermined period of time.