Controlled three-dimensional communication endpoint

• US 9,332,222 B2
• Filed: 02/13/2015
• Issued: 05/03/2016
• Est. Priority Date: 10/10/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Smith, Yancey Christopher, Lang, Eric G., Huitema, Christian F., Zhang, Zhengyou
• Primary Examiner(s): SMITH, CREIGHTON H
• Application Number:
  US14/621,781
• Publication Number:
  US 20150163454A1
• Time in Patent Office: 445 Days
• Field of Search: 348/14.09, 370/260
• US Class Current: 1/1
• CPC Class Codes: G06T 15/10 Geometric effects
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 2213/003 Aspects relating to the "2D+depth" image format
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827
• CPC Class Codes: H04N 9/3185 Geometric adjustment, e.g. keystone or convergence optical or mechanical adjustment of convergence H04N9/317; using scanning means H04N3/22; optical or mechanical adjustments of projectors not peculiar to the presence of an electronic spatial light modulator G03B21/14

Abstract

A controlled three-dimensional (3D) communication endpoint system and method for simulating an in-person communication between participants in an online meeting or conference and providing easy scaling of a virtual environment when additional participants join. This gives the participants the illusion that the other participants are in the same room and sitting around the same table with the viewer. The controlled communication endpoint includes a plurality of camera pods that capture video of a participant from 360 degrees around the participant. The controlled communication endpoint also includes a display device configuration containing display devices placed at least 180 degrees around the participant and display the virtual environment containing geometric proxies of the other participants. Placing the participants at a round virtual table and increasing the diameter of the virtual table as additional participants are added easily achieves scalability.

Claims

1. A method for simulating an in-person communication, comprising:

capturing three-dimensional video of a local participant at a local endpoint;

creating a local geometric proxy for the local participant using the captured three-dimensional video;

generating a scene geometry having correct eye gaze and conversational geometry consistent with an in-person communication;

placing the local geometric proxy in the scene geometry to create a virtual environment;

transmitting the local geometric proxy and the scene geometry to a remote participant at a remote endpoint, wherein the local endpoint and the remote endpoint are at different physical locations to simulate an in-person communication between the local participant and the remote participant; and

displaying the virtual environment to the remote participant at the remote endpoint on a display device occupying space at least 180 degrees around the remote participant to provide the remote participant with a virtual viewpoint of the virtual environment.

2. The method of claim 1, wherein capturing the three-dimensional video is performed in a controlled manner at the local endpoint using a plurality of camera pods that capture both RGB data and depth information about the local participant from 360 degrees around the local participant.

3. The method of claim 1, further comprising adding the RGB data and the depth information to create the local geometric proxy.

4. The method claim 1, further comprising:

capturing a three-dimensional video of the remote participant at the remote endpoint;

creating a remote geometric proxy for the remote participant;

placing the remote geometric proxy in the scene geometry and virtual environment; and

,

rendering both the local geometric proxy and the remote geometric proxy to each other in the scene geometry and virtual environment.

5. The method of claim 4, further comprising transmitting the rendered local geometric proxy, the rendered remote geometric proxy, and the scene geometry to the local endpoint and the remote endpoint.

6. The method of claim 1, wherein the display device comprises a first display device disposed in front of the remote participant, a second display device on one side of the first display device, and a third display device on an other side of the first display device.

7. The method of claim 6, further comprising:

positioning the second display device at a right angle to the first display device; and

positioning the third display device at a right angle to the first display device.

8. The method of claim 6, further comprising:

positioning the second display device at a first angle that is less than 90 degrees to the first display device; and

positioning the third display device at a second angle that is less than 90 degrees to the first display device.

9. The method of claim 8, further comprising setting the first angle and the second angle equal to each other.

10. The method of claim 4, further comprising:

defining a virtual table in the virtual environment; and

placing the local geometric proxy and the remote geometric proxy around the virtual table to simulate the in-person communication in the virtual environment.

11. The method of claim 10, further comprising defining the virtual table as having a circular shape with a first diameter.

12. The method of claim 11, further comprising:

scaling up virtual environment by increasing a number of participants from two participants to more than two participants;

increasing a size of the virtual table from a first diameter to a second diameter, wherein the second diameter is greater than the first diameter; and

placing geometric proxies of each of the participants at the virtual table.

13. The method of claim 1, further comprising adding depth to the virtual viewpoint using motion parallax.

14. The method of claim 13, further comprising:

tracking a head of the remote participant; and

changing what is displayed to the remote participant through the virtual viewpoint based on a position and orientation of the remote participant'"'"'s head.

15. The method of claim 1, further comprising:

tracking an eye gaze of the remote participant; and

changing what is displayed to the remote participant through the virtual viewpoint based on a position and orientation of the remote participant'"'"'s eye gaze.

16. A controlled three-dimensional (3D) endpoint system, comprising:

a plurality of camera pods configured to capture 3D video of a participant at a first endpoint;

a geometric proxy for the participant obtained by adding captured RGB data and captured depth information from the 3D video;

a scene geometry having correct eye gaze and conversational geometry consistent with an in-person communication to create a virtual environment; and

a display device configuration having a plurality of display devices situated at a second endpoint such that the display devices are placed at least 180 degrees around a viewer at the second endpoint to enable the viewer to view the participant through a virtual viewpoint, where the viewer'"'"'s perspective of the participant in the virtual environment changes based on a position and orientation of the viewer'"'"'s head.

17. The controlled 3D endpoint system of claim 16, wherein the plurality of camera pods are disposed around the first endpoint to capture the 3D video of the participant at the first endpoint such that 360 degrees around the participant is captured by the plurality of camera pods.

18. The controlled 3D endpoint system of claim 16, further comprising:

a round virtual table positioned in the virtual environment; and

a rendered geometric proxy for the participant placed around the round virtual table along with other participants at other endpoints of an online meeting tracking a head of the remote participant.

19. The controlled 3D endpoint system of claim 16, wherein the display device comprises a first display device disposed in front of the remote participant, a second display device on one side of the first display device, and a third display device on an other side of the first display device.

20. The controlled 3D endpoint system of claim 19, wherein the system is configured to:

position the second display device at a right angle to the first display device;

position the third display device at a right angle to the first display device;

position the second display device at a first angle that is less than 90 degrees to the first display device; and

position the third display device at a second angle that is less than 90 degrees to the first display device.