Video content assisted audio object extraction

• US 10,200,804 B2
• Filed: 02/24/2016
• Issued: 02/05/2019
• Est. Priority Date: 02/25/2015
• Status: Active Grant

• Current Assignee: Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Original Assignee: Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Inventors: Chen, Lianwu, Sun, Xuejing, Lu, Lie
• Primary Examiner(s): Ramakrishnaiah, Melur
• Application Number:
  US15/553,536
• Publication Number:
  US 20180054689A1
• Time in Patent Office: 1,077 Days
• Field of Search: 700 94, 386285, 382154, 382190
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G10L 19/167 Audio streaming, i.e. formatting and decoding of an encoded audio signal representation into a data stream for transmission or storage purposes
• CPC Class Codes: G10L 19/20 using sound class specific coding, hybrid encoders or object based coding
• CPC Class Codes: H04S 2400/01 Multi-channel, i.e. more than two input channels, sound reproduction with two speakers wherein the multi-channel information is substantially preserved
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2420/11 Application of ambisonics in stereophonic audio systems
• CPC Class Codes: H04S 3/008 in which the audio signals are in digital form, i.e. employing more than two discrete digital channels data reduction aspects thereof based on psychoacoustics G10L19/02
• CPC Class Codes: H04S 3/02 of the matrix type, i.e. in which input signals are combined algebraically, e.g. after having been phase shifted with respect to each other

Abstract

Embodiments of the present invention relate to video content assisted audio object extraction. A method of audio object extraction from channel-based audio content is disclosed. The method comprises extracting at least one video object from video content associated with the channel-based audio content, and determining information about the at least one video object. The method further comprises extracting from the channel-based audio content an audio object to be rendered as an upmixed audio signal based on the determined information. Corresponding system and computer program product are also disclosed.

Claims

1. A method of audio object extraction from channel-based audio content comprising:

extracting at least one video object from video content associated with the channel-based audio content;

determining information about the at least one video object, the information comprising a position of the at least one video object; and

extracting from the channel-based audio content an audio object based on the determined information, the extracting comprising determining a mixing matrix for object audio extraction based on the position of the at least one video object, the extracting being based on the determined mixing matrix, the extracted audio object being subsequently rendered as an upmixed audio signal.

2. The method according to claim 1, wherein the determining information about the at least one video object further comprises at least one of:

determining an audio template of the at least one video object;

determining a size of the at least one video object;

or

determining the number of the at least one video object.

3. The method according to claim 2, wherein the determining an audio template of the at least one video object comprises:

identifying a type of the at least one video object; and

determining, based on the type of the at least one video object, the audio template of the at least one video object from predefined audio templates.

4. The method according to claim 2, wherein the extracting from the channel-based audio content an audio object based on the determined information further comprises:

determining an audio object reference signal based on the audio template of the at least one video object; and

extracting the audio object from the channel-based audio content based on the audio object reference signal.

5. The method according to claim 2, wherein the information includes the number of the at least one video object, and wherein the extracting from the channel-based audio content an audio object based on the determined information further comprises:

determining the number of audio objects to be extracted based on the number of the at least one video object; and

extracting the audio object from the channel-based audio content based on the number of the audio objects to be extracted.

6. The method according to claim 2, wherein the information includes the number of the at least one video object, and wherein the extracting from the channel-based audio content an audio object based on the determined information further comprises:

determining a gain for audio object extraction based on the number of the at least one video object; and

modifying the extracted audio object based on the gain.

7. The method according to claim 2, further comprising:

determining metadata of the audio object from the channel-based audio content; and

modifying the metadata of the audio object based on the determined information.

8. The method according to claim 7, wherein the metadata includes a position of the audio object, and wherein the modifying the metadata of the audio object based on the determined information comprising:

determining a video object aligned with the audio object based on correlation between the video object and the audio object; and

modifying the position of the audio object based on the position of the aligned video object.

9. The method according to claim 8, wherein the correlation between the video object and the audio object is based on at least one of:

a distance between the position of the video object and position of the audio object in the metadata of the audio object, or

a similarity between the audio template of the video object and the audio object.

10. The method according to claim 7, wherein the metadata includes a size of the audio object, and wherein the modifying the metadata of the audio object based on the determined information comprising:

determining a video object aligned with the audio object based on correlation between the video object and the audio object; and

modifying the size of the audio object based on the size of the aligned video object.

11. The method according to claim 1, wherein the determining a position of the at least one video object comprises:

determining a two-dimensional position and depth of the at least one video object in a video frame of the video content; and

mapping the two-dimensional position and the depth into a three-dimensional space of listening environment to obtain the position of the at least one video object.

12. A computer program product of audio object extraction from channel-based audio content, the computer program product being tangibly stored on a non-transient computer-readable medium and comprising machine executable instructions which, when executed, cause the machine to perform steps of the method according to claim 1.

13. A system of audio object extraction from channel-based audio content comprising:

a video object extraction unit configured to extract at least one video object from video content associated with the channel-based audio content;

an information determination unit configured to determine information about the at least one video object and comprising a position determination unit configured to determine a position of the at least one video object; and

an audio object extraction unit configured to extract from the channel-based audio content an audio object based on the determined information, the extracting comprising determining a mixing matrix for object audio extraction based on the position of the at least one video object, the extracting being based on the determined mixing matrix, the extracted audio object being subsequently rendered as an upmixed audio signal.

14. The system according to claim 13, wherein the information determination unit further comprises at least one of:

an audio template determination unit configured to determine an audio template of the at least one video object;

a size determination unit configured to determine a size of the at least one video object;

or

a video object number determination unit configured to determine the number of the at least one video object.

15. The system according to claim 14, wherein the audio template determination unit is further configured to:

identify a type of the at least one video object; and

determine, based on the type of the at least one video object, the audio template of the at least one video object from predefined audio templates.

16. The system according to claim 14, wherein the audio object extraction unit is further configured to:

determine an audio object reference signal based on the audio template of the at least one video object; and

extract the audio object from the channel-based audio content based on the audio object reference signal.

17. The system according to claim 14, wherein the information includes the number of the at least one video object, and wherein the audio object extraction unit is further configured to:

determine the number of audio objects to be extracted based on the number of the at least one video object; and

extract the audio object from the channel-based audio content based on the number of the audio objects to be extracted.

18. The system according to claim 14, wherein the information includes the number of the at least one video object, and wherein the audio object extraction unit is further configured to:

determine a gain for audio object extraction based on the number of the at least one video object; and

modify the extracted audio object based on the gain.

19. The system according to claim 14, further comprising:

a metadata determination unit configured to determine metadata of the audio object from the channel-based audio content; and

a metadata modification unit configured to modify the metadata of the audio object based on the determined information.

20. The system according to claim 19, wherein the metadata includes a position of the audio object, and wherein the metadata modification unit is further configured to:

determine a video object aligned with the audio object based on correlation between the video object and the audio object; and

modify the position of the audio object based on the position of the aligned video object.

21. The system according to claim 20, wherein the correlation between the video object and the audio object is based on at least one of:

a distance between the position of the video object and position of the audio object in the metadata of the audio object, or

a similarity between the audio template of the video object and the audio object.

22. The system according to claim 19, wherein the metadata includes a size of the audio object, and wherein the metadata modification unit is further configured to:

determine a video object aligned with the audio object based on correlation between the video object and the audio object; and

modify the size of the audio object based on the size of the aligned video object.

23. The system according to claim 13, wherein the position determination unit is further configured to:

determine a two-dimensional position and depth of the at least one video object in a video frame of the video content; and

map the two-dimensional position and the depth into a three-dimensional space of listening environment to obtain the position of the at least one video object.