METHOD AND SYSTEM FOR PROVIDING AN IMPROVED AUDIO EXPERIENCE FOR VIEWERS OF VIDEO

• US 20120317594A1
• Filed: 04/21/2011
• Published: 12/13/2012
• Est. Priority Date: 04/21/2011
• Status: Abandoned Application

• Current Assignee: Sony Mobile Communications AB (Sony Group Corp.)
• Original Assignee: Sony Mobile Communications AB (Sony Group Corp.)
• Inventors: Aronsson, Par-Anders, Ek, Martin, Landqvist, Magnus, Thorn, Ola, Jendbro, Magnus, Stenberg, Par
• Application Number:
  US13/503,061
• Publication Number:
  US 20120317594A1
• Time in Patent Office: Days
• US Class Current: 725/18
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/439 Processing of audio elementary streams
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 5/607 for more than one sound signal, e.g. stereo, multilanguages H04N5/602 takes precedence

Abstract

A method and system for enhancing audio for a viewer watching digital video with sound, such as a movie or video game. The method and system determine where in the scene (40) the viewer'"'"'s attention is focused, correlate the viewer'"'"'s focal region with one of a plurality of regions (42a-d) of the video scene (40), preferably associated with a depth map, and enhance the sound corresponding to the focal region of the scene (40) as compared to the sound corresponding to the non-focal regions of the scene (40).

Claims

1. A method for providing an improved audio experience for a viewer of video comprising:

receiving input data associated with a viewer'"'"'s focus;

identifying a focal region of the video corresponding the viewer'"'"'s focus;

selecting at least one focal audio component corresponding to the focal region; and

enhancing the selected focal audio component with respect to at least one non-focal audio component corresponding to a non-focal region of the video.

2. The method of any one of the preceding claims wherein enhancing the selected focal audio component with respect to the at least one non-focal audio component comprises improving the viewer'"'"'s perception of the selected focal audio component.

3. The method of any one of the preceding claims wherein enhancing the selected focal audio component with respect to the at least one non-focal audio component comprises reducing the viewer'"'"'s perception of the at least one non-focal audio component.

4. The method of any one of the preceding claims wherein enhancing the selected focal audio component with respect to the at least one non-focal audio component comprises adjusting levels of the audio components with respect to one another.

5. The method of any one of the preceding claims wherein identifying a focal region of the video corresponding the viewer'"'"'s focus comprises determining an area of a display that has a viewer'"'"'s focus and determining a focal region of the video corresponding to the focus area.

6. The method of any one of the preceding claims wherein multiple focal regions are identified and multiple focal audio components are enhanced.

7. The method of any one of the preceding claims further comprising defining a plurality of regions within a video scene.

8. The method of claim 7 wherein the regions are defined based on one or more of:

the display, the content of the video scene, the identified focal region, or a standard grid.

9. The method of any one of claims 7-8 wherein each of the plurality of regions corresponds to a region of a depth map.

10. The method of any one of the preceding claims wherein the focal region and the non-focal region correspond to different regions of a depth map.

11. The method of any one of the preceding claims further comprising associating audio components and areas of video with regions of a depth map.

12. The method of any one of the preceding claims further comprising mixing the audio components associated with the focal region and the non-focal regions to generate two channel audio.

13. The method of any one of the preceding claims wherein input data associated with a viewer'"'"'s focus is obtained using eye tracking technology.

14. The method of any one of the preceding claims further comprising automatically returning the audio components to their pre-enhanced states.

15. The method of claim 14 wherein returning the audio components to their pre-enhanced states is triggered by at least one of the following:

a change of scene;

change of the viewer'"'"'s focus;

a decrease in levels of the audio component associated with the focal region;

or elapsed time.

16. A method for providing an improved audio experience for a viewer of video comprising:

associating a video scene with a depth map having a plurality of regions;

associating a plurality of audio components with the plurality of regions of the depth map;

tracking at least one of the viewer'"'"'s eyes to determine the viewer'"'"'s focal region of the depth map; and

increasing the level of at least one audio component associated with the focal region compared to the level of an audio component associated with a non-focal region of the depth map.

17. The method of claim 16 wherein multiple focal audio components are enhanced.

18. The method of any one of claims 16-17 wherein the regions are defined based on one or more of:

the display, the content of the video scene, the identified focal region, or standard grid.

19. The method of any one of claims 16-18 further comprising mixing the audio components associated with the focal region and the non-focal regions to generate two channel audio.

20. A system for providing an improved audio experience for a viewer of video comprising:

a display (12) screen for displaying video having a plurality of regions (42a-d);

a viewer monitor digital camera (20) having a field of view directed towards the viewer;

a focus determination module (38) adapted to receive a sequence of images from the viewer monitor digital camera (20) and determine which region (42a-d) video being displayed on the display (12) screen has the viewer'"'"'s focus; and

an audio enhancement module (39) adapted to select at least one focal audio component corresponding to the focal region of the video and enhance the selected focal audio component with respect to at least one non-focal audio component corresponding to a non-focal region of the video.