Providing a tele-immersive experience using a mirror metaphor

• US 9,641,805 B2
• Filed: 03/18/2016
• Issued: 05/02/2017
• Est. Priority Date: 02/20/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Wilson, Andrew D., Zhang, Zhengyou, Chou, Philip A., Fishman, Neil S., Gillett, Donald M., Benko, Hrvoje
• Primary Examiner(s): Anwah, Olisa
• Application Number:
  US15/073,928
• Publication Number:
  US 20160205353A1
• Time in Patent Office: 410 Days
• Field of Search: 348 1401, 348 1402, 348 1407, 348 1408, 348 1411
• CPC Class Codes: A47G 1/02 Mirrors used as equipment hand, pocket, or shaving mirrors A45D42/00; optical viewing means for vehicles B60R1/00
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827

Abstract

A tele-immersive environment is described that provides interaction among participants of a tele-immersive session. The environment includes two or more set-ups, each associated with a participant. Each set-up, in turn, includes mirror functionality for presenting a three-dimensional virtual space for viewing by a local participant. The virtual space shows at least some of the participants as if the participants were physically present at a same location and looking into a mirror. The mirror functionality can be implemented as a combination of a semi-transparent mirror and a display device, or just a display device acting alone. According to another feature, the environment may present a virtual object in a manner that allows any of the participants of the tele-immersive session to interact with the virtual object.

Claims

1. A method comprising:

receiving local camera information that represents a local participant associated with a local system;

receiving remote input information that represents at least one remote participant from at least one remote system operating in a geographical location that is different from the local system;

composing scene information based at least in part on the local camera information and the remote input information;

providing a presentation of a virtual space based at least in part on the scene information, the virtual space showing the local participant and the at least one remote participant as if the local participant and the at least one remote participant were physically present at a same location; and

omitting a virtual image of the local participant from the virtual space in response to receiving an instruction from the local participant indicating not to display the virtual image of the local participant in the virtual space.

2. The method of claim 1, further comprising transferring local input information to the at least one remote system, the local input information corresponding to information associated with objects identified by the local system.

3. The method of claim 2, wherein the local input information comprises one or more of the local camera information or local three-dimensional object information that provides a three-dimensional representation of one or more objects in a real space that is local to the local system.

4. The method of claim 1, further comprising providing a functionality to enable the local participant to interact with a virtual object presented in the virtual space.

5. The method of claim 1, further comprising:

tracking a location or movement of the local participant in a real space of the local system; and

transferring the tracked location or movement of the local participant to the at least one remote system.

6. An apparatus comprising:

a local processing system to;

receive remote input information that represents a plurality of remote participants from a plurality of remote systems operating in one or more geographical locations different from each other; and

compose scene information based at least in part on the remote input information; and

a mirror functionality to present a virtual space based at least in part on the scene information, the virtual space showing the plurality of remote participants as if the plurality of remote participants were physically present at a same location without displaying a virtual image of a local participant associated with the local processing system.

7. The apparatus of claim 6, further comprising an image capture functionality to capture local camera information that represents the local participant.

8. The apparatus of claim 7, wherein the local processing system is further configured to generate local object information based at least in part on the local camera information, and wherein the local processing system composes the scene information based further on the local object information.

9. The apparatus of claim 8, wherein the virtual space further shows the local participant as if the local participant and the plurality of remote participants were physically present the same location in response to receiving an instruction from the local participant to display the virtual image of the local participant in the virtual space.

10. The apparatus of claim 6, wherein the local processing system is further configured to transfer local input information to at least one of the plurality of remote systems, the local input information corresponding to information associated with one or more objects identified by the local processing system.

11. The apparatus of claim 6, wherein the mirror functionality causes a display device to receive the scene information and present respective virtual images of the plurality of remote participants.

12. The apparatus of claim 6, wherein the local processing system includes functionality that enables the local participant to interact with one or more virtual-reflected objects and one or more virtual-actual objects, a virtual-reflected object of the one or more virtual-reflected objects corresponding to an object projected from a perspective of reflections on a mirror surface and a virtual-actual object of the one or more virtual-actual objects corresponding to an object projected from a perspective of an entity that is placed before the mirror surface.

13. The apparatus of claim 6, further comprising a management module that receives information specified by the local participant and provides participant-specified information, wherein the local processing system is further configured to present the participant-specified information using the mirror functionality.

14. The apparatus of claim 6, further comprising a management module that provides a control feature and manages one or more interactions with the control feature, wherein the local processing system is further configured to present the control feature using the mirror functionality.

15. A local system comprising:

an image capture functionality to receive local camera information that represents a local participant associated with the local system;

a local processing system to;

receive remote input information that represents at least one remote participant from at least one remote system operating in a geographical location that is different from the local system; and

compose scene information based at least in part on the local camera information and the remote input information; and

a mirror functionality to;

provide a presentation of a virtual space based at least in part on the scene information, the virtual space showing the local participant and the at least one remote participant as if the local participant and the at least one remote participant were physically present at a same location; and

omit a virtual image of the local participant from the virtual space based on an instruction from the local participant indicating not to display the virtual image of the local participant in the virtual space.

16. The local system of claim 15, wherein the local processing system is further configured to transfer local input information to the at least one remote system, the local input information corresponding to information associated with one or more objects identified by the local processing system.

17. The local system of claim 15, wherein the local processing system includes functionality that enables the local participant to interact with one or more virtual-reflected objects and one or more virtual-actual objects, a virtual-reflected object of the one or more virtual-reflected objects corresponding to an object projected from a perspective of reflections on a mirror surface and a virtual-actual object of the one or more virtual-actual objects corresponding to an object projected from a perspective of an entity that is placed before the mirror surface.

18. The local system of claim 15, further comprising a management module that receives information specified by the local participant and provides participant-specified information, wherein the local processing system is further configured to present the participant-specified information using the mirror functionality.

19. The local system of claim 15, further comprising a management module that provides a control feature and manages one or more interactions with the control feature, wherein the local processing system is further configured to present the control feature using the mirror functionality.