Providing a tele-immersive experience using a mirror metaphor

• US 10,044,982 B2
• Filed: 03/28/2017
• Issued: 08/07/2018
• Est. Priority Date: 02/20/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Wilson, Andrew D., Zhang, Zhengyou, Chou, Philip A., Fishman, Neil S., Gillett, Donald M., Benko, Hrvoje
• Primary Examiner(s): Anwah, Olisa
• Application Number:
  US15/471,709
• Publication Number:
  US 20170201722A1
• Time in Patent Office: 497 Days
• Field of Search: 348 1401, 348 1402, 348 1407, 348 1408, 348 1411
• CPC Class Codes: A47G 1/02 Mirrors used as equipment hand, pocket, or shaving mirrors A45D42/00; optical viewing means for vehicles B60R1/00
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827

Abstract

A tele-immersive environment is described that provides interaction among participants of a tele-immersive session. The environment includes two or more set-ups, each associated with a participant. Each set-up, in turn, includes mirror functionality for presenting a three-dimensional virtual space for viewing by a local participant. The virtual space shows at least some of the participants as if the participants were physically present at a same location and looking into a mirror. The mirror functionality can be implemented as a combination of a semi-transparent mirror and a display device, or just a display device acting alone. According to another feature, the environment may present a virtual object in a manner that allows any of the participants of the tele-immersive session to interact with the virtual object.

Claims

1. A tele-immersive environment for providing interaction among participants of a tele-immersive session, the tele-immersive environment comprising:

a first set-up system that is operating in a first geographical location to provide a first interactive experience of the tele-immersive session to a remote participant, wherein the first set-up system is configured to capture first camera information that represents the remote participant; and

a second set-up system that is operating in a second geographical location to provide a second interactive experience of the tele-immersive session to a local participant, wherein the second set-up system comprises;

image capture functionality to capture second camera information that represents the local participant;

a physical semi-transparent mirror for presenting a virtual image of the local participant, wherein the local participant is physically positioned at a front side of the physical semi-transparent mirror; and

a display device for presenting, based at least in part on the first camera information, a virtual space showing the remote participant as if the remote participant is physically positioned at the front side of the physical semi-transparent mirror.

2. The tele-immersive environment of claim 1, wherein the second set-up system further comprises a local processing system configured to:

determine three-dimensional object information associated with at least one object; and

cause the display device to present three-dimensional scene information based at least in part on the three-dimensional object information.

3. The tele-immersive environment of claim 2, wherein the at least one object is a virtual object presented by the display device, and wherein the local processing system includes functionality that enables both the local participant and the remote participant to interact with the virtual object.

4. The tele-immersive environment of claim 2, wherein the first camera information further represents the at least one object.

5. The tele-immersive environment of claim 2, wherein the local processing system is further configured to:

process depth information associated with associated with the at least one object to create a plurality of three-dimensional meshes of the at least one object; and

generate the three-dimensional object information by applying video information to the plurality of three-dimensional meshes.

6. The tele-immersive environment of claim 1, wherein the image capture functionality is configured to provide depth information for use in constructing a depth image of the local participant.

7. The tele-immersive environment of claim 1, wherein the display device is physically positioned at a rear side of the physical semi-transparent mirror.

8. The tele-immersive environment of claim 7, wherein the display device is substantially adjacent to the rear side of the physical semi-transparent mirror.

9. A method comprising:

receiving local camera information that represents a local participant associated with a local system that is operating in a first geographical location, wherein the local participant is physically positioned at a front side of a physical semi-transparent mirror;

receiving remote input information that represents a remote participant associated with a remote system that is operating in a second geographical location that is different from the first geographical location;

composing scene information based at least in part on the local camera information and the remote input information; and

causing a display device to project imagery from a rear side of the physical semi-transparent mirror to present a virtual space based at least in part on the scene information, the virtual space showing the remote participant as if the remote participant is physically positioned at the front side of the physical semi-transparent mirror.

10. The method of claim 9, further comprising transferring, to the remote system, local input information associated with an object identified by the local system.

11. The method of claim 10, wherein the local input information comprises one or more of the local camera information or local three-dimensional object information that provides a three-dimensional representation of the object, wherein the object is located in a real space that is adjacent to the local system.

12. The method of claim 9, further comprising providing a functionality to enable the local participant to interact with a virtual object presented in the virtual space by the display device.

13. The method of claim 12, wherein the virtual object lacks a corresponding counterpart physical object in a real space associated with the local system or the remote system.

14. The method of claim 9, further comprising:

tracking a location or movement of the local participant in a real space that is adjacent to the local system to determine tracked location or movement information corresponding to the local participant; and

transferring, to the remote system, the tracked location or movement information corresponding to the local participant.

15. A method to facilitate a tele-immersive session, the method comprising:

receiving local camera information from a local image capture functionality corresponding to a local set-up system that is operating in a first geographical location;

transferring local input information to a remote system that is operating in a second geographical location that is remote from the first geographical location, the remote system being associated with a remote participant of the tele-immersive session, the local input information associated with a physical object that is identified by the local set-up system;

receiving, from the remote system, remote input information that is associated with the remote participant;

composing scene information based on the local input information and the remote input information; and

presenting, based on the scene information, a virtual space by projecting imagery from a rear side of a physical semi-transparent mirror, the virtual space showing the remote participant as if the remote participant were physically present at the first geographical location and looking into the physical semi-transparent mirror; and

providing a functionality to enable a local participant, that is physically present at the first geographical location, and the remote participant to jointly manipulate a virtual object that is a counterpart of the physical object that is located at the first geographical location.

16. The method of claim 15, wherein the local camera information is associated with the local participant that is physically present at the first geographical location, the virtual space further showing the local participant as if the local participant were looking into a mirror.

17. The method of claim 16, further comprising providing a second functionality to enable the local participant that is physically present at the first geographical location to interact with a second object that is identified by the remote system, wherein the second object is physically located at the second geographical location.

18. The method of claim 15, wherein the physical object is associated with a workspace table configured to enable the local participant to place physical objects.

19. The method of claim 15, further comprising:

identifying at least one supplemental tag that is affixed to the physical object; and

determining, based on the at least one supplemental tag, supplemental position information associated with the physical object.

20. The method of claim 15, further comprising:

tracking a location or movement of the local participant in a real space that is adjacent to the local system to determine tracked location information corresponding to the local participant; and

transferring the tracked location information to the remote system.