Geometric proxy for a participant in an online meeting

• US 9,836,870 B2
• Filed: 04/13/2016
• Issued: 12/05/2017
• Est. Priority Date: 05/31/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Smith, Yancey Christopher, Lang, Eric G., Zhang, Zhengyou, Huitema, Christian F.
• Primary Examiner(s): Nguyen, Joseph J
• Application Number:
  US15/098,091
• Publication Number:
  US 20160316170A1
• Time in Patent Office: 601 Days
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 15/08 Volume rendering
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2210/56 Particle system, point based geometry or rendering
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827
• CPC Class Codes: H04R 2227/005 Audio distribution systems for home, i.e. multi-room use
• CPC Class Codes: H04S 2400/15 Aspects of sound capture and related signal processing for recording or reproduction

Abstract

A perspective-correct communication window system and method for communicating between participants in an online meeting, where the participants are not in the same physical locations. Embodiments of the system and method provide an in-person communications experience by changing virtual viewpoint for the participants when they are viewing the online meeting. The participant sees a different perspective displayed on a monitor based on the location of the participant'"'"'s eyes. Embodiments of the system and method include a capture and creation component that is used to capture visual data about each participant and create a realistic geometric proxy from the data. A scene geometry component is used to create a virtual scene geometry that mimics the arrangement of an in-person meeting. A virtual viewpoint component displays the changing virtual viewpoint to the viewer and can add perceived depth using motion parallax.

Claims

1. A computer-implemented method comprising:

creating a geometric proxy for a particular participant in an online meeting using color and depth information from an image of the particular participant;

generating scene geometry based on a number of participants in the online meeting including the particular participant and one or more other participants;

rendering the geometric proxy for the particular participant and one or more geometric proxies for the one or more other participants in the online meeting within the scene geometry to cause the scene geometry to visually simulate an in-person conversation; and

causing a virtual viewpoint to be displayed for the particular participant that includes one or more rendered geometric proxies for the one or more other participants along with a background that is part of the scene geometry, the virtual viewpoint being displayed based on a detected position of the particular participant'"'"'s face such that a change in the detected position of the particular participant'"'"'s face causes a change in the virtual viewpoint of at least one of the one or more rendered geometric proxies for the one or more other participants or the background.

2. The method of claim 1, further comprising:

wherein said causing comprises causing the virtual viewpoint to be displayed via a lenticular display that allows multiple viewing angles to enable the particular participant to view the lenticular display from a particular angle to obtain a particular field-of-view and enable the one or more other participants to view the lenticular display from a different angle from the particular field-of-view to obtain a different field-of-view for each of the one or more other participants.

3. The method of claim 1, further comprising:

using a face tracking technique to track the particular participant and the one or more other participants; and

causing different views to be displayed on a monitor to the particular participant and the one or more other participants based on an orientation of their faces.

4. The method of claim 1, further comprising using a face tracking technique to track the orientation of the particular participant'"'"'s face.

5. The method of claim 1, further comprising maintaining the virtual viewpoint level with an eye gaze of the particular participant using a face tracking technique.

6. The method of claim 1, wherein said virtual viewpoint is created using a virtual camera that is a composition of images from at least two camera pods.

7. The method of claim 1, further comprising:

using virtual boxes to ensure that an eye gaze and a conversational geometry between participants in the meeting meets prescribed characteristics; and

determining a number of virtual boxes to use based on the number of participants in the meeting.

8. The method of claim 7, further comprising:

determining that there are two participants in the meeting; and

creating a first virtual box and a second virtual box that are facing each other.

9. The method of claim 7, further comprising:

determining that there are three participants in the meeting; and

creating a first virtual box, a second virtual box, and a third virtual box that are placed around a virtual round table in an equidistant manner.

10. The method of claim 1, further comprising:

tracking the position of the particular participant'"'"'s face using a face tracking technique; and

causing a visual representation of the particular participant to face at least one of the one or more other participants using the position of the particular participant'"'"'s face.

11. A system comprising:

one or more processors; and

one or more computer-readable storage media storing instructions that are executable by the one or more processors to perform operations including;

creating a geometric proxy for a particular participant in an online meeting;

generating scene geometry based on a number of participants in the online meeting including the particular participant and one or more other participants;

rendering the geometric proxy for the particular participant and one or more geometric proxies for the one or more other participants in the online meeting within the scene geometry to cause the scene geometry to visually simulate an in-person conversation; and

causing a virtual viewpoint to be displayed for the particular participant that includes one or more rendered geometric proxies for the one or more other participants and a background that is part of the scene geometry, such that a change in a detected position of the particular participant'"'"'s face causes a change in the virtual viewpoint of at least one of the one or more rendered geometric proxies for the one or more other participants or the background.

12. The system of claim 11, wherein the operations further include:

using a face tracking technique to track the particular participant and the one or more other participants; and

providing different views on a monitor to the particular participant and the one or more other participants based on an orientation of their faces.

13. The system of claim 11, wherein the operations further include using a face tracking technique to track the orientation of the particular participant'"'"'s face.

14. The system of claim 11, wherein the operations further include maintaining the virtual viewpoint level with an eye gaze of the particular participant using a face tracking technique.

15. The system of claim 11, wherein said virtual viewpoint is created using a virtual camera that is a composition of images from at least two camera pods.

16. The system of claim 11, wherein the operations further include:

using virtual boxes to ensure that an eye gaze and a conversational geometry between the participants in the meeting meets prescribed characteristics; and

determining a number of virtual boxes to use based on the number of participants in the meeting.

17. The system of claim 16, wherein the operations further include:

determining that there are two participants in the meeting; and

creating a first virtual box and a second virtual box that are facing each other.

18. The system of claim 16, wherein the operations further include:

determining that there are three participants in the meeting; and

creating a first virtual box, a second virtual box, and a third virtual box that are placed around a virtual round table in an equidistant manner.

19. A method comprising:

generating scene geometry for an online meeting based on a number of participants in the online meeting, the participants including a particular participant and one or more one or more other participants;

rendering a geometric proxy for the particular participant and one or more geometric proxies for the one or more one or more other participants in the online meeting within the scene geometry to cause the scene geometry to visually simulate an in-person conversation; and

causing a virtual viewpoint to be displayed for the particular participant that includes one or more rendered geometric proxies for the one or more one or more other participants along with a background that is part of the scene geometry, the virtual viewpoint being displayed based on a detected position of the particular participant'"'"'s face such that a change in the detected position of the particular participant'"'"'s face causes a change in the virtual viewpoint of at least one of the one or more rendered geometric proxies for the one or more other participants or the background.

20. The method of claim 19, further comprising:

using a face tracking technique to track the position of the particular participant'"'"'s face; and

causing a visual representation of the particular participant to face at least some of the one or more other participants using the position of the particular participant'"'"'s face.