Adjusting spatial congruency in a video conferencing system

• US 10,015,443 B2
• Filed: 11/18/2015
• Issued: 07/03/2018
• Est. Priority Date: 11/19/2014
• Status: Active Grant

• Current Assignee: Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Original Assignee: Dolby Laboratories Licensing Corporation (Dolby Laboratories Incorporated)
• Inventors: Sun, Xuejing, Eckert, Michael
• Primary Examiner(s): Anwah, Olisa
• Application Number:
  US15/527,272
• Publication Number:
  US 20170374317A1
• Time in Patent Office: 958 Days
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04S 2420/01 Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]
• CPC Class Codes: H04S 2420/11 Application of ambisonics in stereophonic audio systems
• CPC Class Codes: H04S 7/30 Control circuits for electronic adaptation of the sound field

Abstract

Example embodiments disclosed herein relate to spatial congruency adjustment. A method for adjusting spatial congruency in a video conference is disclosed. The method in unwarping a visual scene captured by a video endpoint device into at least one rectilinear scene, the video endpoint device being configured to capture the visual scene in an omnidirectional manner, detecting spatial congruency between the at least one rectilinear scene and an auditory scene captured by an audio endpoint device that is positioned in relation to the video endpoint device. The spatial congruency being a degree of alignment between the auditory scene and the at least one rectilinear scene and in response to the detected spatial congruency being below the threshold, adjusting the spatial congruency. Corresponding system and computer program products are also disclosed.

Claims

1. A method for adjusting spatial congruency in a video conference, the method comprising:

unwarping a visual scene captured by a video endpoint device into at least one rectilinear scene, the video endpoint device being configured to capture the visual scene in an omnidirectional manner, the unwarping being based on a starting position of the at least one rectilinear scene;

detecting spatial congruency between the at least one rectilinear scene and an auditory scene captured by an audio endpoint device that is positioned in relation to the video endpoint device, the spatial congruency being a degree of alignment between the auditory scene and the at least one rectilinear scene; and

in response to the detected spatial congruency being below a predefined threshold, adjusting the spatial congruency by changing the starting position of the at least one rectilinear scene and repeating the unwarping step to create a second at least one rectilinear scene having a spatial congruency greater than the predefined threshold.

2. The method according to claim 1, wherein unwarping the visual scene comprises:

assigning a nominal original direction of the captured visual scene; and

unwarping the captured visual scene into one rectilinear scene split from the nominal original direction.

3. The method according to claim 1, wherein unwarping the visual scene comprises:

assigning a nominal original direction of the captured visual scene; and

unwarping the captured visual scene into two rectilinear scenes split from the nominal original direction and from an opposite direction with regard to the nominal original direction.

4. The method according to claim 1, wherein detecting the spatial congruency between the rectilinear scene and the captured auditory scene comprises:

performing an auditory scene analysis on the basis of the captured auditory scene in order to identify an auditory distribution of an audio object, the auditory distribution being a distribution of the audio object relative to the audio endpoint device;

performing a visual scene analysis on the basis of the rectilinear scene in order to identify a visual distribution of the audio object, the visual distribution being a distribution of the audio object relative to the video endpoint device; and

detecting the spatial congruency in accordance with the auditory scene analysis and the visual scene analysis.

5. The method according to claim 4, wherein performing the auditory scene analysis comprises at least one of:

analyzing a directional of arrival of the audio object;

analyzing a depth of the audio object;

analyzing a key audio object; and

analyzing a conversational interaction between the audio objects.

6. The method according to claim 4, wherein performing the visual scene analysis comprises at least one of:

performing a face detection or recognition for the audio object;

analyzing a region of interest for the captured visual scene or the rectilinear scene; and

performing a lip detection for the audio object.

7. The method according to claim 1, wherein adjusting the spatial congruency further comprises at least one of:

rotating the captured auditory scene;

translating the captured auditory scene with regard to the audio endpoint device;

mirroring the captured auditory scene with regard to an axis defined by the video endpoint device;

scaling the captured auditory scene; and

rotating the captured visual scene.

8. The method according to claim 1, wherein the spatial congruency is detected in-situ or at a server.

9. The method according to claim 1, wherein the spatial congruency is adjusted at a server or at a receiving end of the video conference.

10. A system for adjusting spatial congruency in a video conference, the system comprising:

a video endpoint device configured to capture a visual scene in an omnidirectional manner;

an audio endpoint device configured to capture an auditory scene that is positioned in relation to the video endpoint device;

an unwarping unit configured to unwarp the captured visual scene into at least one rectilinear scene, the unwarping being based on a starting position of the at least one rectilinear scene;

a spatial congruency detecting unit configured to detect the spatial congruency between the at least one rectilinear scene and the captured auditory scene, the spatial congruency being a degree of alignment between the auditory scene and the visual scene;

a spatial congruency adjusting unit configured to adjust the spatial congruency in response to the detected spatial congruency being below a predefined threshold by changing the starting position of the at least one rectilinear scene and repeating the unwarping step to create a second at least one rectilinear scene having a spatial congruency greater than the predefined threshold.

11. The system according to claim 10, wherein the unwarping unit comprises:

a nominal original direction assigning unit configured to assign a nominal original direction of the captured visual scene; and

a splitting unit configured to unwarp the captured visual scene into one rectilinear scene split from the nominal original direction.

12. The system according to claim 10, wherein the unwarping unit comprises:

a nominal original direction assigning unit configured to assign a nominal original direction of the captured visual scene; and

a splitting unit configured to unwarp the captured visual scene into two rectilinear scenes split from the nominal original direction and from an opposite direction with regard to the nominal original direction.

13. The system according to claim 10, wherein the spatial congruency detecting unit comprises:

an auditory scene analyzing unit configured to perform an auditory scene analysis on the basis of the captured auditory scene in order to identify an auditory distribution of an audio object, the auditory distribution being a distribution of the audio object relative to the audio endpoint device; and

a visual scene analyzing unit configured to perform a visual scene analysis on the basis of the rectilinear scene in order to identify a visual distribution of the audio object, the visual distribution being a distribution of the audio object relative to the video endpoint device,

wherein the spatial congruency detecting unit is configured to detect the spatial congruency in accordance with the auditory scene analysis and the visual scene analysis.

14. The system according to claim 13, wherein the auditory scene analyzing unit comprises at least one of:

a directional of arrival analyzing unit configured to analyze a directional of arrival of the audio object;

a depth analyzing unit configured to analyze a depth of the audio object;

a key object analyzing unit configured to analyze a key audio object; and

a conversation analyzing unit configured to analyze a conversational interaction between the audio objects.

15. The system according to claim 13, wherein the visual scene analyzing unit comprises at least one of:

a face analyzing unit configured to perform a face detection or recognition for the audio object;

a region analyzing unit configured to analyze a region of interest for the captured visual scene or the rectilinear scene; and

a lip analyzing unit configured to perform a lip detection for the audio object.

16. The system according to claim 10, wherein the spatial congruency adjusting unit comprises at least one of:

an auditory scene rotating unit configured to rotate the captured auditory scene;

an auditory scene translating unit configured to translate the captured auditory scene with regard to the audio endpoint device;

an auditory scene mirroring unit configured to mirror the captured auditory scene with regard to an axis defined by the video endpoint device;

an auditory scene scaling unit configured to scale the captured auditory scene; and

a visual scene rotating unit configured to rotate the captured visual scene.

17. The system according to claim 10, wherein the spatial congruency is detected in-situ or at a server.

18. The system according to claim 10, wherein the spatial congruency is adjusted at a server or at a receiving end of the video conference.

19. A computer program product for adjusting spatial congruency in a video conference, the computer program product being tangibly stored on a non-transient computer-readable medium and comprising machine executable instructions which, when executed, cause a machine to perform steps of the method according to claim 10.