CONTROLLED THREE-DIMENSIONAL COMMUNICATION ENDPOINT

• US 20140098183A1
• Filed: 10/10/2012
• Published: 04/10/2014
• Est. Priority Date: 10/10/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Smith, Yancey Christopher, Lang, Eric G., Huitema, Christian F., Zhang, Zhengyou
• Granted Patent: US 8,976,224 B2
• Time in Patent Office: Days
• US Class Current: 348/14.16
• CPC Class Codes: G06T 15/10 Geometric effects
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 2213/003 Aspects relating to the "2D+depth" image format
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827
• CPC Class Codes: H04N 9/3185 Geometric adjustment, e.g. keystone or convergence optical or mechanical adjustment of convergence H04N9/317; using scanning means H04N3/22; optical or mechanical adjustments of projectors not peculiar to the presence of an electronic spatial light modulator G03B21/14

Abstract

A controlled three-dimensional (3D) communication endpoint system and method for simulating an in-person communication between participants in an online meeting or conference and providing easy scaling of a virtual environment when additional participants join. This gives the participants the illusion that the other participants are in the same room and sitting around the same table with the viewer. The controlled communication endpoint includes a plurality of camera pods that capture video of a participant from 360 degrees around the participant. The controlled communication endpoint also includes a display device configuration containing display devices placed at least 180 degrees around the participant and display the virtual environment containing geometric proxies of the other participants. Placing the participants at a round virtual table and increasing the diameter of the virtual table as additional participants are added easily achieves scalability.

Claims

1. A method for simulating an in-person communication, comprising:

capturing three-dimensional video of a local participant at a local endpoint;

creating a local geometric proxy for the local participant using the captured three-dimensional video;

generating a scene geometry having correct eye gaze and conversational geometry consistent with an in-person communication;

placing the local geometric proxy in the scene geometry to create a virtual environment; and

transmitting the local geometric proxy and the scene geometry to a remote participant at a remote endpoint, wherein the local endpoint and the remote endpoint are at different physical locations to simulate an in-person communication between the local participant and the remote participant.

2. The method of claim 1, further comprising capturing the three-dimensional video in a controlled manner at the local endpoint using a plurality of camera pods that capture both RGB data and depth information about the local participant from 360 degrees around the local participant.

3. The method of claim 2, further comprising adding the RGB data and the depth information to create the local geometric proxy.

4. The method claim 1, further comprising:

capturing a three-dimensional video of the remote participant at the remote endpoint;

creating a remote geometric proxy for the remote participant;

placing the remote geometric proxy in the scene geometry and virtual environment; and

,

rendering both the local geometric proxy and the remote geometric proxy to each other in the scene geometry and virtual environment.

5. The method of claim 4, further comprising transmitting the rendered local geometric proxy, the rendered remote geometric proxy, and the scene geometry to the local endpoint and the remote endpoint.

6. The method of claim 1, further comprising displaying the virtual environment to the remote participant at the remote endpoint on a display device occupying space at least 180 degrees around the remote participant to provide the remote participant with a virtual viewpoint of the virtual environment.

7. The method of claim 6, wherein the display device comprises a first display device disposed in front of the remote participant, a second display device on one side of the first display device, and a third display device on an other side of the first display device.

8. The method of claim 7, further comprising:

positioning the second display device at a right angle to the first display device; and

positioning the third display device at a right angle to the first display device.

9. The method of claim 7, further comprising:

positioning the second display device at a first angle that is less than 90 degrees to the first display device; and

positioning the third display device at a second angle that is less than 90 degrees to the first display device.

10. The method of claim 9, further comprising setting the first angle and the second angle equal to each other.

11. The method of claim 4, further comprising:

defining a virtual table in the virtual environment; and

placing the local geometric proxy and the remote geometric proxy around the virtual table to simulate the in-person communication in the virtual environment.

12. The method of claim 11, further comprising defining the virtual table as having a circular shape with a first diameter.

13. The method of claim 12, further comprising:

scaling up virtual environment by increasing a number of participants from two participants to more than two participants;

increasing a size of the virtual table from a first diameter to a second diameter, wherein the second diameter is greater than the first diameter; and

placing geometric proxies of each of the participants at the virtual table.

14. The method of claim 6, further comprising adding depth to the virtual viewpoint using motion parallax.

15. The method of claim 14, further comprising:

tracking a head of the remote participant; and

changing what is displayed to the remote participant through the virtual viewpoint based on a position and orientation of the remote participant'"'"'s head.

16. A controlled three-dimensional (3D) endpoint system, comprising:

a plurality of camera pods disposed around a first endpoint to capture a 3D video of a participant at the first endpoint such that 360 degrees around the participant is captured by the plurality of camera pods;

a geometric proxy for the participant obtained by adding captured RGB data and captured depth information from the 3D video;

a scene geometry having correct eye gaze and conversational geometry consistent with an in-person communication to create a virtual environment; and

a display device configuration having a plurality of display devices situated at a second endpoint such that the display devices are placed at least 180 degrees around a viewer at the second endpoint to enable the viewer to view the participant through a virtual viewpoint, where the viewer'"'"'s perspective of the participant in the virtual environment changes based on a position and orientation of the viewer'"'"'s head.

17. The controlled 3D endpoint system of claim 16, further comprising:

a round virtual table positioned in the virtual environment; and

a rendered geometric proxy for the participant placed around the round virtual table along with other participants at other endpoints of an online meeting. tracking a head of the remote participant; and

18. A method for scaling a number of participants in an online meeting, comprising:

organizing a controlled capture environment at endpoints having a plurality of camera pods disposed around each of the endpoints;

capturing three-dimensional video of each participant at the each endpoint using the plurality of camera pods;

creating geometric proxies for each of the participants;

generating scene geometry based on the number of participants, the scene geometry including a virtual table;

rendering each of the geometric proxies to each other in the scene geometry that is consistent with an in-person communication;

placing the rendered geometric proxies in the scene geometry around the virtual table to create a virtual environment;

organizing a controlled viewing environment at the endpoints having a display device that wraps around at least 180 degrees of the participants at the endpoints;

displaying the virtual environment to the participants in the controlled viewing environment using the display device;

changing a virtual viewpoint of participants viewing the display device based on a position and orientation of each participant'"'"'s head;

increasing the number of participants such that additional participants are added; and

increasing a size of the virtual table to accommodate the additional participants.

19. The method of claim 18, further comprising defining the virtual table as a round virtual table having a diameter.

20. The method of claim 19, wherein increasing a size of the virtual table further comprises increasing the diameter of the round virtual table to accommodate the additional participants.