Using the location of a near-end user in a video stream to adjust audio settings of a far-end system

• US 9,883,140 B2
• Filed: 09/26/2014
• Issued: 01/30/2018
• Est. Priority Date: 05/19/2014
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Johnson, Martin E., Saux, Tom-Davy William Jendrik, Isaac, Ronald N., Choisel, Sylvain, Family, Afrooz
• Primary Examiner(s): SMITH, CREIGHTON H
• Application Number:
  US15/311,824
• Publication Number:
  US 20170099458A1
• Time in Patent Office: 1,222 Days
• Field of Search: None
• CPC Class Codes: H04N 7/141 between two video terminals, e.g. videophone telephonic communication systems combined with television receiver for reception of entertainment or information matter H04M11/085
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04R 2430/01 Aspects of volume control, not necessarily automatic, in sound systems
• CPC Class Codes: H04R 27/00 Public address systems circuits for preventing acoustic reaction H04R3/02; circuits for distributing signals to loudspeakers H04R3/12; monitoring or testing arrangements for public address systems H04R29/007; amplifiers H03F
• CPC Class Codes: H04R 3/12 for distributing signals to two or more loudspeakers specially adapted for hearing aids H04R25/407
• CPC Class Codes: H04S 2400/13 Aspects of volume control, not necessarily automatic, in stereophonic sound systems
• CPC Class Codes: H04S 2400/15 Aspects of sound capture and related signal processing for recording or reproduction
• CPC Class Codes: H04S 7/30 Control circuits for electronic adaptation of the sound field
• CPC Class Codes: H04S 7/305 Electronic adaptation of stereophonic audio signals to reverberation of the listening space H04S7/301 takes precedence

Abstract

A video conferencing system is described that includes a near-end and a far-end system. The near-end system records both audio and video of one or more users proximate to the near-end system. This recorded audio and video is transmitted to the far-end system through the data connection. The video stream and/or one or more settings of the recording camera are analyzed to determine the amount of a video frame occupied by the recorded user(s). The video conferencing system may directly analyze the video frames themselves and/or a zoom setting of the recording camera to determine a ratio or percentage of the video frame occupied by the recorded user(s). By analyzing video frames associated with an audio stream, the video conferencing system may drive a speaker array of the far-end system to more accurately reproduce sound content based on the position of the recorded user in a video frame.

Claims

1. A method for adjusting sound emitted by a far-end system, comprising:

receiving, by the far-end system, a video stream and an audio stream captured by a near-end system;

determining an amount of area in a first frame of the video stream occupied by a near-end audio source;

determining a first set of audio parameters to apply to the audio stream based on the determined amount of area in the first frame of the video stream occupied by the near-end audio source; and

playing the audio stream using the first set of audio parameters to emit sound through one or more transducers in a speaker system of the far-end system.

2. The method of claim 1, wherein the near-end audio source is a user of the near-end system captured by a near-end video camera.

3. The method of claim 1, wherein determining the amount of area in the first frame occupied by the near-end audio source is determined based on one of (1) a zoom setting of a near-end video camera that captured the first frame and (2) an analysis of the first frame using facial recognition.

4. The method of claim 1, wherein determining the first set of audio parameters comprises:

setting a loudness parameter to a first loudness level for the sound produced by the speaker system in the far-end system while playing the audio stream upon determining that the near-end audio source occupies more than a predefined amount of the first frame; and

setting the loudness parameter to a second loudness level for the sound produced by the speaker system in the far-end system while playing the audio stream upon determining that the near-end audio source occupies less than the predefined amount of the first frame, wherein the first loudness level is larger than the second loudness level.

5. The method of claim 1, wherein determining the first set of audio parameters comprises:

setting a directivity parameter for sound produced by the speaker system in the far-end system while playing the audio stream to a first directivity level upon determining that the near-end audio source occupies more than a predefined amount of the first frame; and

setting the directivity parameter for sound produced by the speaker system in the far-end system while playing the audio stream to a second directivity level upon determining that the near-end audio source occupies less than the predefined amount of the first frame, wherein the first directivity level is greater than the second directivity level.

6. The method of claim 1, wherein determining the first set of audio parameters comprises:

setting an equalization parameter for sound produced by the speaker system in the far-end system while playing the audio stream to a first equalization setting upon determining that the near-end audio source occupies more than a predefined amount of the first frame; and

setting the equalization parameter for sound produced by the speaker system in the far-end system while playing the audio stream to a second equalization setting upon determining that the near-end audio source occupies less than the predefined amount of the first frame, wherein the first equalization setting filters more high frequency elements of the audio stream than the second equalization setting.

7. The method of claim 1, wherein determining the first set of audio parameters comprises:

setting a reverberation parameter for sound produced by the speaker system in the far-end system while playing the audio stream to a first reverberation level upon determining that the near-end audio source occupies more than a predefined amount of the first frame; and

setting the reverberation parameter for sound produced by the speaker system in the far-end system while playing the audio stream to a second reverberation level upon determining that the near-end audio source occupies less than the predefined amount of the first frame, wherein the first reverberation level is less than the second reverberation level.

8. The method of claim 7, wherein playing the audio stream using the first set of audio parameters, including the reverberation parameter, comprises:

detecting separately by one or more microphones in the near-end system (1) reverberant sounds and (2) direct sounds produced by the near-end audio source, wherein the audio stream includes both the reverberant sounds and direct sounds from the near-end audio source; and

driving the speaker system in the far-end system to emit the reverberant sounds off axis from the direct sounds, wherein the reverberation parameter represents a loudness of reverberant sounds.

9. The method of claim 8, wherein the one or more microphones vary widths or directions of one or more associated beam patterns over time to capture the reverberant sounds and the direct sounds.

10. The method of claim 1, further comprising:

determining an amount of area in a second frame of the video stream occupied by the near-end audio source;

determining a second set of audio parameters to apply to the audio stream based on the determined amount of area in the second frame of the video stream occupied by the near-end audio source;

determining one or more intermediary audio parameters between the first set of audio parameters and the second set of audio parameters; and

playing the audio stream to emit sound through the speaker system of the far-end system sequentially using the first set of audio parameters, the one or more intermediate audio parameters, and the second set of audio parameters.

11. An article of manufacture, comprising:

a non-transitory machine-readable storage medium that stores instructions which, when executed by a processor in a computer,

determine an amount of area in a first frame of a video stream occupied by a near-end audio source, wherein the video stream is captured by a near-end system along with a corresponding audio stream;

determine a first set of audio parameters to apply to the audio stream based on the determined amount of area in the first frame of the video stream occupied by the near-end audio source; and

play the audio stream using the first set of audio parameters to emit sound through one or more transducers in a speaker system of a far-end system.

12. The article of manufacture of claim 11, wherein determining the amount of area in the first frame occupied by the near-end audio source is determined based on one of (1) a zoom setting of a near-end video camera that captured the first frame and (2) an analysis of the first frame using facial recognition.

13. The article of manufacture of claim 11, wherein the non-transitory machine-readable storage medium stores further instructions which, when executed by the processor:

set a loudness parameter to a first loudness level for the sound produced by the speaker system in the far-end system while playing the audio stream upon determining that the near-end audio source occupies more than a predefined amount of the first frame; and

set the loudness parameter to a second loudness level for the sound produced by the speaker system in the far-end system while playing the audio stream upon determining that the near-end audio source occupies less than the predefined amount of the first frame, wherein the first loudness level is larger than the second loudness level.

14. The article of manufacture of claim 11, wherein the non-transitory machine-readable storage medium stores further instructions which, when executed by the processor:

set a directivity parameter for sound produced by the speaker system in the far-end system while playing the audio stream to a first directivity level upon determining that the near-end audio source occupies more than a predefined amount of the first frame; and

set the directivity parameter for sound produced by the speaker system in the far-end system while playing the audio stream to a second directivity level upon determining that the near-end audio source occupies less than the predefined amount of the first frame, wherein the first directivity level is greater than the second directivity level.

15. The article of manufacture of claim 11, wherein the non-transitory machine-readable storage medium stores further instructions which, when executed by the processor:

set a reverberation parameter for sound produced by the speaker system in the far-end system while playing the audio stream to a first reverberation level upon determining that the near-end audio source occupies more than a predefined amount of the first frame; and

set the reverberation parameter for sound produced by the speaker system in the far-end system while playing the audio stream to a second reverberation level upon determining that the near-end audio source occupies less than the predefined amount of the first frame, wherein the first reverberation level is less than the second reverberation level.

16. The article of manufacture of claim 11, wherein the non-transitory machine-readable storage medium stores further instructions which, when executed by the processor:

set an equalization parameter for sound produced by the speaker system in the far-end system while playing the audio stream to a first equalization setting upon determining that the near-end audio source occupies more than a predefined amount of the first frame; and

set the equalization parameter for sound produced by the speaker system in the far-end system while playing the audio stream to a second equalization setting upon determining that the near-end audio source occupies less than the predefined amount of the first frame, wherein the first equalization setting filters more high frequency elements of the audio stream than the second equalization setting.

17. The article of manufacture of claim 11, wherein the non-transitory machine-readable storage medium stores further instructions which, when executed by the processor:

determine an amount of area in a second frame of the video stream occupied by the near-end audio source;

determine a second set of audio parameters to apply to the audio stream based on the determined amount of area in the second frame of the video stream occupied by the near-end audio source;

determine one or more intermediary audio parameters between the first set of audio parameters and the second set of audio parameters; and

play the audio stream to emit sound through the speaker system of the far-end system sequentially using the first set of audio parameters, the one or more intermediate audio parameters, and the second set of audio parameters.

18. A far-end system for adjusting audio, comprising:

a network interface for receiving an audio stream and a video stream captured by a near-end system; and

a hardware processor to;

determine a first set of audio parameters to apply to the audio stream based on a determined amount of area in a first frame of the video stream occupied by a near-end audio source; and

play the audio stream using the first set of audio parameters to emit sound through one or more transducers in a speaker system of the far-end system.

19. The far-end system of claim 18, wherein the hardware processor further:

determines the amount of area in the first frame of the video stream occupied by the near-end audio source.

20. The far-end system of claim 18, wherein determining the amount of area in the first frame occupied by the near-end audio source is determined based on one of (1) a zoom setting of a near-end video camera that captured the first frame and (2) an analysis of the first frame using facial recognition.

21. The far-end system of claim 18, wherein determining the first set of audio parameters comprises:

setting a loudness parameter to a first loudness level for the sound produced by the speaker system in the far-end system while playing the audio stream upon determining that the near-end audio source occupies more than a predefined amount of the first frame; and

setting the loudness parameter to a second loudness level for the sound produced by the speaker system in the far-end system while playing the audio stream upon determining that the near-end audio source occupies less than the predefined amount of the first frame, wherein the first loudness level is larger than the second loudness level.

22. The far-end system of claim 18, wherein determining the first set of audio parameters comprises:

setting a directivity parameter for sound produced by the speaker system in the far-end system while playing the audio stream to a first directivity level upon determining that the near-end audio source occupies more than a predefined amount of the first frame; and

setting the directivity parameter for sound produced by the speaker system in the far-end system while playing the audio stream to a second directivity level upon determining that the near-end audio source occupies less than the predefined amount of the first frame, wherein the first directivity level is greater than the second directivity level.

23. The far-end system of claim 18, wherein determining the first set of audio parameters comprises:

setting a reverberation parameter for sound produced by the speaker system in the far-end system while playing the audio stream to a first reverberation level upon determining that the near-end audio source occupies more than a predefined amount of the first frame; and

setting the reverberation parameter for sound produced by the speaker system in the far-end system while playing the audio stream to a second reverberation level upon determining that the near-end audio source occupies less than the predefined amount of the first frame, wherein the first reverberation level is less than the second reverberation level.

24. The far-end system of claim 18, wherein determining the first set of audio parameters comprises:

setting an equalization parameter for sound produced by the speaker system in the far-end system while playing the audio stream to a first equalization setting upon determining that the near-end audio source occupies more than a predefined amount of the first frame; and

setting the equalization parameter for sound produced by the speaker system in the far-end system while playing the audio stream to a second equalization setting upon determining that the near-end audio source occupies less than the predefined amount of the first frame, wherein the first equalization setting filters more high frequency elements of the audio stream than the second equalization setting.