METHOD AND APPARATUS FOR GENERATING AND PLAYING AUDIO SIGNALS, AND SYSTEM FOR PROCESSING AUDIO SIGNALS

• US 20110164769A1
• Filed: 02/25/2011
• Published: 07/07/2011
• Est. Priority Date: 08/27/2008
• Status: Active Grant

• Current Assignee: Huawei Device Company Limited (Huawei Investment & Holding Co., Ltd.)
• Original Assignee: Huawei Device Company Limited (Huawei Investment & Holding Co., Ltd.)
• Inventors: ZHAN, Wuzhou, WANG, Dongqi
• Granted Patent: US 8,705,778 B2
• Time in Patent Office: Days
• US Class Current: 381/307
• CPC Class Codes: G10L 19/008 Multichannel audio signal coding or decoding using interchannel correlation to reduce redundancy, e.g. joint-stereo, intensity-coding or matrixing
• CPC Class Codes: G10L 2021/02166 Microphone arrays; Beamforming
• CPC Class Codes: G10L 21/028 using properties of sound source
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 13/30 Image reproducers optical systems for producing stereoscopic or other three-dimensional effects G02B30/00
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04R 1/403 loud-speakers
• CPC Class Codes: H04R 1/406 microphones
• CPC Class Codes: H04R 2203/12 Beamforming aspects for stereophonic sound reproduction with loudspeaker arrays
• CPC Class Codes: H04S 2400/11 Positioning of individual sound objects, e.g. moving airplane, within a sound field H04S2420/13 takes precedence
• CPC Class Codes: H04S 2400/15 Aspects of sound capture and related signal processing for recording or reproduction
• CPC Class Codes: H04S 2420/03 Application of parametric coding in stereophonic audio systems
• CPC Class Codes: H04S 7/30 Control circuits for electronic adaptation of the sound field

Abstract

A method for generating and playing audio signals and a system for processing audio signals are disclosed. The method for generating audio signals includes: generating distance information about an audio signal corresponding to a view point position, according to obtained auxiliary video and direction information about the audio signal, where the auxiliary video is a disparity map or a depth map; encoding the direction information and distance information about the audio signal, and sending the encoded information. The apparatus for generating audio signals includes an audio signal distance information obtaining module and an audio signal encoding module. With the present invention, the position information, including direction information and distance information, about the audio signal may be obtained accurately in combination with a three-dimensional video signal and a three-dimensional audio signal, without increasing the size of a microphone array, and the audio signal is sent and played.

Claims

1. A method for generating audio signals, comprising:

generating distance information about an audio signal corresponding to a view point position, according to obtained auxiliary video and direction information about the audio signal, wherein the auxiliary video is a disparity map or a depth map; and

encoding the audio signal, the direction information about the audio signal, and the distance information about the audio signal, and sending the encoded information.

2. The method of claim 1, wherein the step of obtaining the auxiliary video and the direction information about the audio signal comprises:

capturing, by a microphone array, at least two audio signals as an input audio stream;

processing the input audio stream by using a microphone array processing method, and obtaining an enhanced audio signal and the direction information about the audio signal;

capturing, by a camera set, at least two video signals as an input video stream; and

obtaining a main video and the auxiliary video according to the input video stream.

3. The method of claim 2, wherein the step of capturing at least two audio signals as the input audio stream by the microphone array comprises:

capturing, by the microphone array, at least two audio signals as a first input audio stream, wherein each audio signal is a mixed audio signal composed of sounds from multiple sound sources; and

separating each audio signal from the first input audio stream by using an audio signal separation method, obtaining audio signals corresponding to sounds of each sound source, and combining the sounds of each sound source into an input audio stream.

4. The method of claim 1, wherein the step of generating the distance information about the audio signal corresponding to the view point position comprises:

obtaining depth information about the audio signal, according to the auxiliary video and the direction information about the audio signal;

obtaining coordinate information about the audio signal at a presence site, according to the depth information and the direction information about the audio signal; and

generating the distance information about the audio signal corresponding to the view point position, according to the coordinate information about the audio signal and the view point position information.

5. The method of claim 4, wherein the step of obtaining the depth information about the audio signal comprises:

obtaining coordinates of the audio signal in the auxiliary video according to the direction information about the audio signal, and judging whether the auxiliary video is a depth map or a disparity map;

if the auxiliary video is a depth map, obtaining the depth information about the audio signal directly from the depth map, according to the coordinates; and

if the auxiliary video is a disparity map, obtaining the disparity of the audio signal from the disparity map, according to the coordinates, and obtaining the depth information about the audio signal through a calculation according to the disparity.

6. An apparatus for generating audio signals, comprising:

an audio signal distance information obtaining module, configured to generate distance information about an audio signal corresponding to a view point position, according to obtained auxiliary video and direction information about the audio signal, wherein the auxiliary video is a disparity map or a depth map; and

an audio signal encoding module, configured to encode the audio signal, the direction information about the audio signal, and the distance information about the audio signal, and send the encoded information.

7. The apparatus of claim 6, further comprising:

a microphone array, configured to capture at least two audio signals as an input audio stream;

an audio input signal processing module, configured to;

process the input audio stream by using a microphone array processing method, obtain an enhanced audio signal and direction information about the audio signal, and send the audio signal and the direction information about the audio signal to the audio signal encoding module;

a video collecting module, configured to capture at least two video signals as an input video stream; and

a video input signal processing module, configured to obtain a main video and the auxiliary video according to the input video stream.

8. The apparatus of claim 7, wherein the microphone array comprises:

a microphone array unit, configured to capture at least two audio signals as a first input audio stream, wherein each audio signal is a mixed audio signal composed of sounds from multiple sound sources; and

an audio signal separating unit, configured to;

separate each audio signal from the first input audio stream by using an audio signal separation method, obtain audio signals corresponding to sounds of each sound source, and combine the audio signals into an input audio stream.

9. The apparatus of claim 6, wherein the audio signal distance information obtaining module comprises:

a depth information obtaining unit, configured to obtain depth information about the audio signal, according to the auxiliary video and the direction information about the audio signal;

a coordinate information obtaining unit, configured to obtain coordinate information about the audio signal at a presence site, according to the depth information and the direction information about the audio signal; and

a distance information obtaining unit, configured to generate the distance information about the audio signal corresponding to the view point position, according to the coordinate information about the audio signal and the view point position information.

10. A method for playing audio signals, comprising:

decoding received encoded data, and obtaining an audio signal and direction information about the audio signal;

obtaining distance information about the audio signal;

processing the audio signal by using an audio signal reproduction method, according to the direction information and distance information about the audio signal, and obtaining speaker signals corresponding to each speaker; and

playing the speaker signals by using a speaker array or a surround stereo system.

11. The method of claim 10, wherein the step of obtaining the distance information about the audio signal comprises:

decoding the received encoded data, and obtaining the distance information about the audio signal.

12. The method of claim 10, wherein the step of obtaining the distance information about the audio signal comprises:

decoding the received encoded data, and obtaining an auxiliary video;

generating the distance information about the audio signal, according to the auxiliary video and the direction information about the audio signal.

13. The method of claim 11, wherein the step of obtaining the distance information about the audio signal comprises:

decoding the received encoded data, and obtaining an auxiliary video;

generating the distance information about the audio signal, according to the auxiliary video and the direction information about the audio signal.

14. The method of claim 12, wherein the step of generating the distance information about the audio signal comprises:

obtaining depth information about the audio signal, according to the auxiliary video and the direction information about the audio signal;

obtaining coordinate information about the audio signal at a presence site, according to the depth information and the direction information about the audio signal; and

generating the distance information about the audio signal corresponding to a view point position, according to the coordinate information about the audio signal and the view point position information.

15. The method of claim 13, wherein the step of generating the distance information about the audio signal comprises:

obtaining depth information about the audio signal, according to the auxiliary video and the direction information about the audio signal;

obtaining coordinate information about the audio signal at a presence site, according to the depth information and the direction information about the audio signal; and

generating the distance information about the audio signal corresponding to a view point position, according to the coordinate information about the audio signal and the view point position information.

16. An apparatus for playing audio signals, comprising:

an audio signal decoding module, configured to decode received encoded data, and obtain an audio signal and direction information about the audio signal;

an audio signal distance information obtaining module, configured to obtain distance information about the audio signal;

a speaker signal obtaining module, configured to;

receive the audio signal and the direction information about the audio signal from the audio signal decoding module, receive the distance information about the audio signal from the audio signal distance information obtaining module, process the audio signal by using an audio signal reproduction method, and obtain speaker signals corresponding to each speaker; and

a speaker signal playing module, configured to play the speaker signals by using a speaker array or a surround stereo system.

17. The apparatus of claim 16, wherein the audio signal distance information obtaining module is an audio signal distance information decoding module configured to decode the received encoded data and obtain the distance information about the audio signal.

18. The apparatus of claim 16, wherein further comprising:

a first video signal decoding module, adapted to decode the received encoded data, and obtain an auxiliary video;

the audio signal distance information obtaining module is an audio signal distance information obtaining module configured to generate the distance information about the audio signal, according to the auxiliary video and the direction information about the audio signal.

19. The apparatus of claim 18, wherein the audio signal distance information obtaining module comprises:

a depth information obtaining unit, configured to obtain depth information about the audio signal, according to the auxiliary video and the direction information about the audio signal;

a coordinate information obtaining unit, configured to obtain coordinate information about the audio signal at a presence site, according to the depth information and the direction information about the audio signal; and

a distance information obtaining unit, configured to generate the distance information about the audio signal corresponding to a view point position, according to the coordinate information about the audio signal and the view point position information.

20. A system for processing audio signals, comprising an apparatus for generating audio signals and an apparatus for playing audio signals, wherein:

the apparatus for generating audio signals comprises;

an audio signal distance information obtaining module, configured to generate distance information about an audio signal corresponding to a view point position, according to obtained auxiliary video and direction information about the audio signal, wherein the auxiliary video is a disparity map or a depth map; and

an audio signal encoding module, configured to encode the audio signal, the direction information about the audio signal, and the distance information about the audio signal, and send the encoded information; and

the apparatus for playing audio signals comprises;

an audio signal decoding module, configured to decode the received encoded data, and obtain the audio signal and the direction information about the audio signal;

an audio signal distance information obtaining module, configured to obtain the distance information about the audio signal;

a speaker signal obtaining module, configured to process the audio signal by using an audio signal reproduction method, according to the direction information and distance information about the audio signal, and obtain speaker signals corresponding to each speaker; and

a speaker signal playing module, configured to play the speaker signal by using a speaker array or a surround stereo system.