Methods and systems for close proximity spatial audio rendering

• US 8,411,126 B2
• Filed: 06/24/2010
• Issued: 04/02/2013
• Est. Priority Date: 06/24/2010
• Status: Active Grant

• Current Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Original Assignee: Hewlett-Packard Development Company, L.P. (HP Inc.)
• Inventors: Lee, Bowon, Kalker, Ton, Tan, Kar Han
• Primary Examiner(s): Nguyen, Duc
• Assistant Examiner(s): Ndje, Theodore
• Application Number:
  US12/822,532
• Publication Number:
  US 20110316966A1
• Time in Patent Office: 1,013 Days
• Field of Search: 348/14.16, 348/14.01, 348/14.08, 348/14.03, 348/14.04, 348/14.07, 381/116, 381/71.1
• US Class Current: 348/14.01
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

Disclosed herein are multimedia-conferencing systems and methods enabling local participants to hear remote participants from the direction the remote participants are rendered on a display. In one aspect, a method includes a computing device receives a remote participant'"'"'s image and sound information collected at a remote site. The remote participant'"'"'s image is rendered on a display at a local site. When the local participant is in close proximity to the display, sounds generated by the remote participant are played over stereo loudspeakers so that the local participant perceives the sounds as emanating from the remote participant'"'"'s location rendered on the display.

Claims

1. In a video/audio conference between a local participant and a remote participant, a method of spatial audio rendering the remote participant at a local site using a computing device comprising:

receiving the remote participant'"'"'s image and sound information collected at a remote site;

receiving location information collected at the remote site that indicates the remote participant'"'"'s location in front of a remote display at the remote site;

rendering the remote participant'"'"'s image on a local display at the local site;

determining the local participant'"'"'s location in front of the local display by determining the distance between first and second speakers of the stereo loudspeakers is greater than the distance between the local participant'"'"'s left ear and the left speaker and is greater than the distance between the local participant'"'"'s right ear and the right speaker; and

when the local participant is in close proximity to the local display, playing sounds generated by the remote participant over stereo loudspeakers by adjusting the sounds based at least in part on the remote participant'"'"'s location in front of the remote display at the remote site and the local participant'"'"'s location in front of the local display so that the local participant perceives the sounds as emanating from the remote participant'"'"'s location rendered on the local display.

2. The method of claim 1, wherein playing sounds generated by the remote participant over the stereo loudspeakers further comprises adjusting intensity of each loudspeaker such that the sound intensity output from the speaker located closest to the remote participant'"'"'s location rendered on the local display is greater than the sound intensity output from the speaker located farthest from the remote participant'"'"'s location rendered on the local display.

3. The method of claim 2, wherein adjusting the intensity of each loudspeaker further comprises determining an interaural intensity difference between the intensity of sounds reaching the local participant'"'"'s left ear from the location of the remote participant'"'"'s image rendered on the local display and the intensity of sounds reaching the local participant'"'"'s right ear from the location of the remote participant'"'"'s image rendered on the local display.

4. The method of claim 1, wherein playing sounds generated by the remote participant over the stereo loudspeakers further comprises playing sounds through a first loudspeaker before or after playing the same sounds through a second loudspeaker.

5. The method of claim 4, wherein playing sounds generated by the remote participant over the stereo loudspeakers further comprises applying a time delay to the sound output from the loudspeaker located farthest from the remote participant'"'"'s location rendered on the local display.

6. The method of claim 5, wherein applying the time delay further comprises

determining a first interaural time difference between the time for sound to travel from the first speaker to the local participant'"'"'s left ear and the time for sound to travel from the second speaker to the local participant'"'"'s right ear;

determining a second interaural time difference between the time for sound to travel from local virtual coordinates of the remote participant to the local participant'"'"'s left ear and the time for sound to travel from the local virtual coordinates of the remote participant to the local participant'"'"'s right ear; and

computing the time delay as the difference between the first interaural time difference and the second interaural time difference.

7. The method of claim 1, wherein the loudspeakers are located on opposite sides of the local display.

8. The method of claim 1, further comprising when the local participant is not in close proximity to the local display, playing sounds generated by the remote participant over stereo loudspeakers without an intensity difference or a time delay.

9. A non-transitory computer-readable medium having instructions encoded thereon for enabling a processor to perform the operations of:

receive a remote participant'"'"'s image and sound information collected at a remote site;

receive location information collected at the remote site that indicates the remote participant'"'"'s location in front of a remote display at the remote site;

render the remote participant'"'"'s image on a local display at the local site;

determine the local participant'"'"'s location in front of the local display by determining that the distance of the local participant'"'"'s left and right ear from a left and right speaker of the stereo loudspeakers respectively is less than the distance between the first and second speakers of the stereo loudspeakers; and

when a local participant is in close proximity to the local display, play sounds generated by the remote participant over stereo loudspeakers by adjusting the sounds based at least in part on the remote participant'"'"'s location in front of the remote display at the remote site and the local participant'"'"'s location in front of the local display so that the local participant perceives the sounds as emanating from the remote participant'"'"'s location rendered on the local display.

10. A multimedia-conferencing system comprising:

a local display;

stereo loudspeakers;

a camera system; and

a computing device including at least one processor and memory having instructions encoded thereon for enabling the least one processor to perform the operations of;

receive a remote participant'"'"'s image and sound information collected at a remote site,

receive location information collected at the remote site that indicates the remote participant'"'"'s location in front of a remote display at the remote site,

render the remote participant'"'"'s image on the local display at the local site,

determine a local participant'"'"'s location in front of the local display based on depth information capture by the camera system;

when the local participant is located in close proximity to the local display, play sounds generated by the remote participant over the stereo loudspeakers by adjusting the sounds based at least in part on the remote participant'"'"'s location in front of the remote display at the remote site and the local participant'"'"'s location in front of the local display so that the local participant perceives the sounds as emanating from the remote participant'"'"'s location rendered on the local display; and

when the local participant is not in close proximity to the local display, playing sounds generated by the remote participant over stereo loudspeakers without an intensity difference or a time delay.

11. The system of claim 10, wherein the local display further comprises one of a television, a monitor, or projector and a screen for displaying images generated by the projector.

12. The system of claim 10, wherein stereo loudspeakers further comprise a first speaker and a second speaker located on opposite sides of the local display.

13. The system of claim 10, wherein the camera system further comprises:

a camera that captures images of the local participant; and

a depth camera that captures depth information of the local participant.

14. The system of claim 10, wherein the camera system further comprises a stereo camera that captures three dimensional images of the local participant.

15. The system of claim 10 further comprising one or more microphones for capturing sounds generated by the local participant.

16. The system of claim 10, wherein sounds generated by the remote participant are played over the stereo loudspeakers further comprises adjust intensity of each loudspeaker such that the sound intensity output from the speaker located closest to the remote participant'"'"'s location rendered on the local display is greater than the sound intensity output from the speaker located farthest from the remote participant'"'"'s location rendered on the local display.

17. The system of claim 10, wherein sounds generated by the remote participant are played over the stereo loudspeakers further comprises play sounds through a first loudspeaker before or after playing the same sounds through a second loudspeaker.

18. The system of claim 17, wherein play sounds generated by the remote participant over the stereo loudspeakers further comprises apply a time delay to the sound output from the loudspeaker located farthest from the remote participant'"'"'s location rendered on the local display.