Perspective-correct communication window with motion parallax

• US 9,332,218 B2
• Filed: 10/01/2015
• Issued: 05/03/2016
• Est. Priority Date: 05/31/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Smith, Yancey Christopher, Lang, Eric G., Zhang, Zhengyou, Huitema, Christian F.
• Primary Examiner(s): NGUYEN, PHUNG HOANG JOSEPH
• Application Number:
  US14/873,190
• Publication Number:
  US 20160028991A1
• Time in Patent Office: 215 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 15/04 Texture mapping
• CPC Class Codes: G06T 15/08 Volume rendering
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2210/56 Particle system, point based geometry or rendering
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/243 using three or more 2D image sensors
• CPC Class Codes: H04N 13/246 Calibration of cameras
• CPC Class Codes: H04N 13/257 Colour aspects
• CPC Class Codes: H04N 7/142 Constructional details of the terminal equipment, e.g. arrangements of the camera and the display
• CPC Class Codes: H04N 7/15 Conference systems
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827
• CPC Class Codes: H04R 2227/005 Audio distribution systems for home, i.e. multi-room use
• CPC Class Codes: H04S 2400/15 Aspects of sound capture and related signal processing for recording or reproduction

Abstract

A perspective-correct communication window system and method for communicating between participants in an online meeting, where the participants are not in the same physical locations. Embodiments of the system and method provide an in-person communications experience by changing virtual viewpoint for the participants when they are viewing the online meeting. The participant sees a different perspective displayed on a monitor based on the location of the participant'"'"'s eyes. Embodiments of the system and method include a capture and creation component that is used to capture visual data about each participant and create a realistic geometric proxy from the data. A scene geometry component is used to create a virtual scene geometry that mimics the arrangement of an in-person meeting. A virtual viewpoint component displays the changing virtual viewpoint to the viewer and can add perceived depth using motion parallax.

Claims

1. A method for creating an in-person communication experience between participants in an online meeting, comprising:

capturing Red-Green-Blue (RGB) data and depth information of the local participant from a plurality of camera pods arranged around a monitor that a local participant is viewing;

creating a geometric proxy for the local participant by adding the RGB and the depth information;

generating scene geometry based on a number of participants in the online meeting, including the local participant;

rendering the geometric proxy for the local participant and geometric proxies for each of the other participants in the online meeting to each other in the scene geometry so that the scene geometry is consistent with an in-person conversation;

transmitting the rendered geometric proxies and scene geometry to each of the participants;

displaying a virtual viewpoint to the local participant that includes a rendered geometric proxy for each of the participants along with a background that is part of the scene geometry;

changing the virtual viewpoint based on a position of the local participant'"'"'s face such that the local participant'"'"'s view of other participants and the background is dependent on an orientation of the local participant'"'"'s face; and

adding an illusion of depth to the virtual viewpoint using motion parallax to create the in-person communication experience between the participants in the online meeting.

2. The method of claim 1, further comprising:

having the local participant and another participant at the same physical location and viewing the monitor, where the monitor has a lenticular display that allows multiple viewing angles; and

having the local participant view the monitor from the right side to obtain a right field-of-view and having the other participant view the monitor from the left side to obtain a left field-of-view, the right field-of-view and the left field-of-view being different.

3. The method of claim 1, further comprising:

having the local participant and another participant at the same physical location and viewing the monitor;

using a face tracking technique to track the local participant and the other participant; and

providing different views on the monitor to the local participant and the other participant based on an orientation of their faces.

4. The method of claim 1, further comprising using a face tracking technique to track the orientation of the local participant'"'"'s face.

5. The method of claim 1, further comprising keeping the virtual viewpoint level with an eye gaze of the local participant using a face tracking technique.

6. The method of claim 1, wherein said virtual viewpoint is created using a virtual camera that is a composition of images from at least two of the plurality of camera pods.

7. The method of claim 1, further comprising:

using virtual boxes to ensure that an eye gaze and conversational geometry between the participants meets prescribed characteristics and that to other participants the conversational geometry also meets the prescribed characteristics; and

determining a number of virtual boxes to use based on the number of participants.

8. The method of claim 7, further comprising:

determining that there are two participants in the meeting; and

creating a first virtual box and a second virtual box that are facing each other.

9. The method of claim 7, further comprising:

determining that there are three participants in the meeting; and

creating a first virtual box, a second virtual box and a third virtual box that are place around a virtual round table in an equidistant manner.

10. The method of claim 1, further comprising:

tracking the position of the local participant'"'"'s face using a face tracking technique; and

keeping the local participant facing at least some of the other participants using the position of the local participant'"'"'s face to ensure that the local participant and at least one of the other participants are always looking straight at each other.

11. A system for creating an in-person communication experience between participants in an online meeting, comprising:

a plurality of camera pods arranged around a monitor that a local participant is viewing;

one or more computing devices, said computing devices being in communication with each other via a computer network whenever there is a plurality of computing devices; and

a computer program having sub-programs executable by the one or more computing devices, the one or more computing devices being directed by the sub-programs of the computer program to,

capture Red-Green-Blue (RGB) data and depth information of the local participant from the plurality of camera pods,

create a geometric proxy for the local participant by adding the RGB and the depth information,

generate scene geometry based on a number of participants in the online meeting, including the local participant,

render the geometric proxy for the local participant and geometric proxies for each of the other participants in the online meeting to each other in the scene geometry so that the scene geometry is consistent with an in-person conversation,

transmit the rendered geometric proxies and scene geometry to each of the participants,

display a virtual viewpoint to the local participant that includes a rendered geometric proxy for each of the participants along with a background that is part of the scene geometry,

change the virtual viewpoint based on a position of the local participant'"'"'s face such that the local participant'"'"'s view of other participants and the background is dependent on an orientation of the local participant'"'"'s face, and

add an illusion of depth to the virtual viewpoint using motion parallax to create the in-person communication experience between the participants in the online meeting.

12. The system of claim 11, wherein the local participant and another participant are at the same physical location and viewing the monitor, the computer program further comprising sub-programs for:

using a face tracking technique to track the local participant and the other participant; and

providing different views on the monitor to the local participant and the other participant based on an orientation of their faces.

13. The system of claim 11, wherein the computer program further comprising sub-programs for:

using, a face tracking technique to track the orientation of the local participant'"'"'s face.

14. The system of claim 11, wherein the computer program further comprising sub-programs for:

keeping the virtual viewpoint level with an eye gaze of the local participant using a face tracking technique.

15. The system of claim 11, wherein said virtual viewpoint is created using a virtual camera that is a composition of images from at least two of the plurality of camera pods.

16. The system of claim 11, wherein the computer program further comprising sub-programs for:

using virtual boxes to ensure that an eye gaze and conversational geometry between the participants meets prescribed characteristics and that to other participants the conversational geometry also meets the prescribed characteristics; and

determining a number of virtual boxes to use based on the number of participants.

17. The system of claim 16, wherein the sub-program for determining a number of virtual boxes comprising sub-programs for:

determining that there are two participants in the meeting; and

creating a first virtual box and a second virtual box that are facing each other.

18. The system of claim 16, wherein the sub-program for determining a number of virtual boxes comprising sub-programs for:

determining that there are three participants in the meeting; and

creating a first virtual box, a second virtual box, and a third virtual box that are place around a virtual round table in an equidistant manner.

19. The system of claim 11 wherein the computer program further comprising sub-programs for:

using a face tracking technique to track the position of the local participant'"'"'s face; and

keeping the local participant facing at least some of the other participants using the position of the local participant'"'"'s face to ensure that the local participant and at least one of the other participants are always looking straight at each other.

20. A computer storage device having stored thereon computer-executable instructions for creating an in-person communication experience between participants in an online meeting, comprising:

capturing Red-Green-Blue (RGB) data and depth information of the local participant from a plurality of camera pods arranged around a monitor that a local participant is viewing;

creating a geometric proxy for the local participant by adding the RGB and the depth information;

generating scene geometry based on a number of participants in the online meeting, including the local participant;

rendering the geometric proxy for the local participant and geometric proxies for each of the other participants in the online meeting to each other in the scene geometry so that the scene geometry is consistent with an in-person conversation;

transmitting the rendered geometric proxies and scene geometry to each of the participants;

displaying a virtual viewpoint to the local participant that includes a rendered geometric proxy for each of the participants along with a background that is part of the scene geometry;

changing the virtual viewpoint based on a position of the local participant'"'"'s face such that the local participant'"'"'s view of other participants and the background is dependent on an orientation of the local participant'"'"'s face; and

adding an illusion of depth to the virtual viewpoint using motion parallax to create the in-person communication experience between the participants in the online meeting.