Multi-user video conferencing with perspective correct eye-to-eye contact

• US 7,515,174 B1
• Filed: 12/06/2004
• Issued: 04/07/2009
• Est. Priority Date: 12/06/2004
• Status: Active Grant

• Current Assignee: DreamWorks Animation LLC (Comcast Corporation)
• Original Assignee: DreamWorks Animation LLC (Comcast Corporation)
• Inventors: Francisco, Emmanuel C., Leonard, Edwin R.
• Primary Examiner(s): Ramakrishnaiah; Melur
• Application Number:
  US11/006,166
• Time in Patent Office: 1,583 Days
• Field of Search: 348 1401- 1409, 348 1411- 1416, 348/335, 348/535, 359/296, 313/478, 313/474, 353/10
• US Class Current: 348/14.16
• CPC Class Codes: G02B 30/27 involving lenticular arrays
• CPC Class Codes: H04N 7/144 camera and display on the same optical axis, e.g. optically multiplexing the camera and display for eye to eye contact
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

A video conferencing system provides perspective correct eye-to-eye interaction for multiple participants at sites remote from each other. The system provides at each site a configuration of participant viewing positions, a multidirectional image capture system for capturing images of each position at one site from the perspective of each viewing position at the other site, and a multidirectional display system for displaying images of each viewing position at the other site from the perspective of each viewing position at the one site.

Claims

1. A videoconferencing system providing contact between a first and second site, the system comprising:

at the first site;

a plurality of first actual viewing positions;

a plurality of first multidirectional image capture systems,

each first multidirectional image capture system comprising a plurality of first image capture devices,

the number of first multidirectional image capture systems corresponding to the number of first actual viewing positions,

each first multidirectional image capture system having for each first actual viewing position, a first image capture device oriented to capture an image stream of that first actual viewing position along a line of sight from a virtual viewing position corresponding to a second actual viewing position at the second site to that first actual viewing position; and

at the second site;

a plurality of second actual viewing positions, each of which has a corresponding virtual viewing position at the first site, and

a first plurality of multidirectional displays, each of which is oriented towards a distinct one of the second actual viewing positions, each multidirectional display being communicatively coupled to a subset of the plurality of first image capture devices to receive their respective image streams and to concurrently display the received image streams, the subset comprising the first image capture devices that are oriented towards the same first actual viewing position.

2. The system of claim 1, further comprising:

at the second site;

a plurality of second multidirectional image capture systems, each of the second multidirectional image capture systems comprising a plurality of second image capture devices, the number of second multidirectional image capture systems corresponding to the number of second actual viewing positions, and

each second multidirectional image capture system having for each second actual viewing position, a second image capture device oriented to capture an image stream of that second actual viewing position along a line of sight from a virtual viewing position corresponding to a first actual viewing position at the first site to that second actual viewing position; and

at the first site;

a second plurality of multidirectional displays, each of which is oriented towards a distinct one of the first actual viewing positions, each of these multidirectional displays being communicatively coupled to a subset of the plurality of second image capture devices to receive their respective image streams and to concurrently display the received image streams, the subset comprising the second image capture devices that are oriented towards the same second actual viewing position.

3. The system of claim 1 wherein the arrangement of actual viewing positions at the first site is symmetrical to the arrangement of actual viewing positions at the second site.

4. The system of claim 1 wherein the arrangement of actual viewing positions and virtual viewing positions at the first site is a circular arrangement.

5. The system of claim 1 wherein the arrangement of actual viewing positions and virtual viewing positions at the first site is a rectangular arrangement.

6. The system of claim 1 wherein the multidirectional display devices are high resolution fixed pixel displays that output the received image streams at substantially the full width of the display.

7. The system of claim 1 wherein an angle between the lines of sight of adjacent image capture devices is significantly greater than an angle used for capturing images with stereoscopic disparity.

8. The system of claim 1 wherein an angle between the lines of sights of adjacent image capture devices is approximately 60 degrees.

9. The system of claim 1 wherein:

the second site comprises;

two, second actual viewing positions;

two, multidirectional displays; and

the first site comprises;

two, first actual viewing positions;

two, first multidirectional image capture systems, each of which comprises two image capture devices, wherein

one image capture device in each multidirectional image capture system is oriented along a line of sight from a virtual viewing position corresponding to one of the two second actual viewing positions to one of the two first actual viewing positions, and

the other image capture device in each multidirectional image capture system is oriented along a line of sight from a virtual viewing position corresponding to the other second actual viewing position to the other first actual viewing position.

10. The system of claim 1 wherein:

the second site comprises;

three, second actual viewing positions;

three, multidirectional displays; and

the first site comprises;

three, first actual viewing positions;

three, first multidirectional image capture systems, each of which comprises three image capture devices, wherein

one image capture device in each multidirectional image capture system is oriented along a line of sight from a virtual viewing position corresponding to one of the three second actual viewing positions to one of the three first actual viewing positions, and

the other image capture device in each multidirectional image capture system is oriented along a line of sight from a virtual viewing position corresponding to the other second actual viewing position to the other first actual viewing position.

11. The system of claim 1, further comprising:

at the second site, a processing module coupled to each multidirectional display, each processing module coupled to receive multiplexed image streams comprising one image stream from one first image capture device in each first multi-directional image capture system, the image capture devices being those that are oriented towards the same first actual viewing position,

demultiplex the multiplexed image streams, and

combine the demultiplexed image streams into a single image stream to provide to the respective multi-directional display device for display thereon.

12. The system of claim 11, wherein the processing module further comprises a module which decreases differences in the demultiplexed image streams resulting from the first image capture devices being at varying distances to the first actual viewing positions.

13. The system of claim 11, wherein the processing module further comprises a module which selectively processes individual full images from a demultiplexed image streams, alternately providing the full images to the multi-directional display device in a single image stream.

14. The system of claim 11, wherein the processing module further comprises a module which interleaves selected columns of the demultiplexed image streams to form the single image stream for display on a corresponding multi-directional display device, the single image providing for the concurrent display of the image streams.

15. The system of claim 14, wherein the selected columns of an image stream comprise a subset of all of the columns of the image stream, such that the single image stream has a same resolution as the one of the image streams.

16. The system of claim 14, wherein the selected columns of an image stream comprise all of the columns of the image stream, such that the single image stream has a higher resolution than the image streams.

17. The system of claim 11, wherein the processing module further comprises a converter which converts the resolution of the single image stream into a high definition resolution.

18. The system of claim 11, further comprising:

at the first site, a plurality of transmission processing modules, each transmission processing module coupled to one first image capture device in each first multi-directional image capture system to receive an image stream therefrom, the image capture devices being those that are oriented towards the same first actual viewing position, and multiplex the received image streams, and further coupled to a corresponding one of the display processing modules at the second site to provide the multiplexed image streams thereto.

19. The system of claim 1, wherein each multidirectional display system comprises:

a display panel having a plurality of columns of pixels;

a lens array, having a plurality of lenticular lenses, each lens comprising plural facets, each facet substantially aligned with a single column of pixels to direct light from the column along an line of sight, and wherein the angle between the line of sight and a normal axis of the panel substantially exceeds an angle of stereoscopic disparity.

20. A videoconferencing system providing contact between a first site having actual viewing positions S_j and a second site having actual viewing positions S_i, where i and j are equal and are the number of viewing positions at each site, the system comprising:

at the first site;

multidirectional image capture systems M_j, the number of multidirectional image capture system M_j corresponding to the number of actual viewing positions S_j;

wherein each multidirectional image capture system M_j comprises plural image capture devices T_ij, each of which is disposed at or near a virtual viewing position V_i that corresponds to an actual viewing position S_i at the second site, and directed along a line of site L_ij towards an actual viewing position S_j at the first site, so that each image capture device T_ij captures an image stream of actual viewing position S_j with the correct perspective and eye-to-eye contact that a participant seated at actual viewing position S_i would have of actual viewing position S_j if seated at virtual viewing position V_i; and

at the second site;

multidirectional displays Q_j, each of which is oriented directly towards a distinct one of the actual viewing positions S_i, each multidirectional display Q_j being communicatively coupled to a distinct subset of the plurality of image capture devices T_ij at the first site to receive the image streams of actual viewing position S_j from different virtual positions V_i and concurrently display the received image streams to participants each of the actual viewing positions S_i.

21. The system of claim 20, further comprising:

at the second site;

multidirectional image capture systems M_i, the number of multidirectional image capture systems M_i corresponding to the number of actual viewing positions S_i;

each second multidirectional image capture system having for each second actual viewing position, a second image capture device oriented to capture an image stream of that second actual viewing position along a line of sight from a virtual viewing position corresponding to a first actual viewing position at the first site to that second actual viewing position; and

wherein each multidirectional image capture system M_i comprises plural image capture devices T_ji, each of which is disposed at or near a virtual viewing position V_j that corresponds to an actual viewing position S_j at the first site, and directed along a line of site L towards an actual viewing position S_i at the second site, so that each image capture device T_ji captures an image stream of actual viewing position S_i with the correct perspective and eye-to-eye contact that a participant seated at actual viewing position S_j would have of actual viewing position S_i if seated at virtual viewing position V_j; and

at the first site;

multidirectional displays Q_i, each of which is oriented directly towards a distinct one of the actual viewing positions S_j, each multidirectional display Q_i being communicatively coupled to a distinct subset of the plurality of image capture devices T_ji at the first site to receive the image streams of actual viewing position S_j from different virtual positions V_j and concurrently display the received image streams to participants each of the actual viewing positions S_j.

22. The system of claim 21, wherein i=j=2.

23. The system of claim 21, wherein i=j=3.

24. A videoconferencing method providing contact between a first and second site, the method comprising:

at the first site;

capturing a plurality of image streams of a plurality of first actual viewing positions from respective ones of a plurality of first multidirectional image capture systems, each first multidirectional image capture system comprising a plurality of first image capture devices, the number of first multidirectional image capture systems corresponding to the number of first actual viewing positions, each first multidirectional image capture system having for each first actual viewing position, a first image capture device oriented to capture an image stream of that first actual viewing position along a line of sight from a virtual viewing position corresponding to a second actual viewing position at the second site to that first actual viewing position;

transmitting the captured image streams to the second site; and

at the second site;

concurrently displaying the captured image streams on a first plurality of multidirectional displays, each of which is oriented towards a distinct one of a plurality of second actual viewing positions, each of which has a corresponding virtual viewing position that the first site, each multidirectional display being communicatively coupled to a subset of the plurality of first image capture devices to receive their respective image streams, the subset comprising the first image capture devices that are oriented towards the same first actual viewing position.

25. The method of claim 24, further comprising:

at the second site;

capturing a plurality of image streams of the second actual viewing positions from respective ones of a plurality of second multidirectional image capture systems, each the second multidirectional image capture systems comprising a plurality of second image capture devices, the number of second multidirectional image capture systems corresponding to the number of second actual viewing positions, and each second multidirectional image capture system having for each second actual viewing position, a second image capture device oriented to capture an image stream of that second actual viewing position along a line of sight from a virtual viewing position corresponding to a first actual viewing position at the first site to that second actual viewing position;

transmitting the captured images to the first site; and

at the first site;

concurrently displaying the image streams captured at the first site on a second plurality of multidirectional displays, each of which is oriented towards a distinct one of the first actual viewing positions, each of which has a corresponding virtual viewing position that the second site, each of the second plurality of multidirectional displays being communicatively coupled to a subset of the plurality of second image capture devices to receive their respective image streams, the subset comprising the second image capture devices that are oriented towards the same second actual viewing position.

26. The method of claim 24 wherein the arrangement of actual viewing positions at the first site is symmetrical to the arrangement of actual viewing positions at the second site.

27. The method of claim 24 wherein the arrangement of actual viewing positions and virtual viewing positions at the first site is circular.

28. The method of claim 24 wherein the arrangement of actual viewing positions and virtual viewing positions at the first site is rectangular.

29. The method of claim 24 wherein the multidirectional display devices are high resolution fixed pixel displays that output the received image streams at substantially the full width of the display.

30. The method of claim 24 wherein an angle between the lines of sights of adjacent image capture devices is significantly greater than an angle used for capturing images with stereoscopic disparity.

31. The method of claim 24 wherein an angle between the lines of sights of adjacent image capture devices is approximately 60 degrees.

32. The method of claim 24 wherein:

the second site comprises;

two, second actual viewing positions;

two, multidirectional displays; and

the first site comprises;

two, first actual viewing positions;

two, first multidirectional image capture systems, each of which comprises two image capture devices, wherein

one image capture device in each multidirectional image capture system is oriented along a line of sight from a virtual viewing position corresponding to one of the two second actual viewing positions to one of the two first actual viewing positions, and

the other image capture device in each multidirectional image capture system is oriented along a line of sight from a virtual viewing position corresponding to the other second actual viewing position to the other first actual viewing position.

33. The method of claim 24 wherein:

the second site comprises;

three, second actual viewing positions;

three, multidirectional displays; and

the first site comprises;

three, first actual viewing positions;

three, first multidirectional image capture systems, each of which comprises three image capture devices, wherein

one image capture device in each multidirectional image capture system is oriented along a line of sight from a virtual viewing position corresponding to one of the three second actual viewing positions to one of the three first actual viewing positions, and

the other image capture device in each multidirectional image capture system is oriented along a line of sight from a virtual viewing position corresponding to the other second actual viewing position to the other first actual viewing position.