Providing a tele-immersive experience using a mirror metaphor

• US 9,325,943 B2
• Filed: 02/20/2013
• Issued: 04/26/2016
• Est. Priority Date: 02/20/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Wilson, Andrew D., Chou, Philip A., Gillett, Donald M., Benko, Hrvoje, Zhang, Zhengyou, Fishman, Neil S.
• Primary Examiner(s): Anwah, Olisa
• Application Number:
  US13/772,252
• Publication Number:
  US 20140232816A1
• Time in Patent Office: 1,161 Days
• Field of Search: 348/14.01, 348/14.03, 348/14.07, 348/14.16, 348/14.08
• US Class Current: 1/1
• CPC Class Codes: A47G 1/02 Mirrors used as equipment hand, pocket, or shaving mirrors A45D42/00; optical viewing means for vehicles B60R1/00
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827

Abstract

A tele-immersive environment is described that provides interaction among participants of a tele-immersive session. The environment includes two or more set-ups, each associated with a participant. Each set-up, in turn, includes mirror functionality for presenting a three-dimensional virtual space for viewing by a local participant. The virtual space shows at least some of the participants as if the participants were physically present at a same location and looking into a mirror. The mirror functionality can be implemented as a combination of a semi-transparent mirror and a display device, or just a display device acting alone. According to another feature, the environment may present a virtual object in a manner that allows any of the participants of the tele-immersive session to interact with the virtual object.

Claims

1. A tele-immersive environment for providing interaction among a plurality of participants of a tele-immersive session, the tele-immersive environment comprising:

a plurality of set-up systems, each set-up system of the plurality of set-up systems operating in a different geographical location and providing an interactive experience of the tele-immersive session to a participant that is local to the respective set-up system, and each set-up system comprising;

image capture functionality configured to capture local camera information that represents the local participant associated with the set-up system;

a local processing system configured to;

receive the local camera information;

generate local three-dimensional object information based on the local camera information;

transfer local input information to at least one remote processing system associated with at least one remote participant that is local to a remote set-up system, the local input information corresponding to information associated with objects identified by the local processing system;

receive remote input information from said at least one remote processing system, the remote input information corresponding to information associated with objects identified by said at least one remote processing system; and

compose three-dimensional scene information based on the local three-dimensional object information and the remote input information; and

mirror functionality configured to present a three-dimensional virtual space based on the three-dimensional scene information, the three-dimensional virtual space showing multiple remote participants of the plurality of participants without the local participant as if the multiple remote participants were physically present at a same location and looking into a mirror.

2. The tele-immersive environment of claim 1, wherein the image capture functionality is configured to provide information for use in constructing a depth image of the local participant.

3. The tele-immersive environment of claim 1, wherein the mirror functionality comprises:

a physical semi-transparent mirror for presenting a virtual image of the local participant at the set-up system; and

a display device configured to receive the three-dimensional scene information, and to present a virtual image of at least one remote participant.

4. The tele-immersive environment of claim 1, wherein the mirror functionality comprises a display device configured to receive the three-dimensional scene information, and to present a virtual image of the local participant and at least one remote participant.

21. The tele-immersive environment of claim 1, wherein the local processing system is configured to collect supplemental position information of an object identified by the local processing system, the supplemental position information generated by at least one of a position-determination device of the object and a tag affixed to the object in a real space.

22. The tele-immersive environment of claim 1, wherein the mirror functionality is configured to allow any participant of the tele-immersive session to interact with writing information.

5. A local set-up system in a tele-immersive environment, the tele-immersive environment for providing interaction among participants of a tele-immersive session, the local set-up system comprising:

image capture functionality configured to capture local camera information that represents a local participant associated with the local set-up system;

a local processing system configured to;

compose three-dimensional scene information based on the local camera information and remote input information, the remote input information being received from at least one remote processing system associated with at least one remote participant that is local to a remote set-up system operating in a different geographical location from the local set-up system, the remote input information corresponding to information associated with objects identified by said at least one remote processing system; and

transfer at least some of local input information to said at least one remote processing system, the local input information corresponding to information associated with objects identified by the local processing system; and

mirror functionality configured to present a three-dimensional virtual space based on the three-dimensional scene information, the three-dimensional virtual space showing at least some of the participants as if the participants were physically present at a same location and looking into a mirror.

6. The local set-up system of claim 5, wherein the image capture functionality is configured to provide information for use in constructing a depth image of the local participant.

7. The local set-up system of claim 5, wherein the mirror functionality comprises:

a physical semi-transparent mirror for presenting a virtual image of the local participant at the local set-up system; and

a display device configured to receive the three-dimensional scene information, and to present a virtual image of at least one remote participant.

8. The local set-up system of claim 5, wherein the mirror functionality comprises a display device configured to receive the three-dimensional scene information, and to present a virtual image of the local participant and at least one remote participant.

9. The local set-up system of claim 5, wherein the local processing system includes functionality that enables the local participant to interact with a virtual object presented by the mirror functionality.

10. The local set-up system of claim 9, wherein the local set-up system includes a physical workspace in which the local participant may place a physical object, and wherein the virtual object is a virtual counterpart of the physical object.

11. The local set-up system of claim 10, wherein the physical workspace includes a workspace table on which the physical object is placed.

12. The local set-up system of claim 9, wherein the virtual object has no counterpart physical object in a real space associated with any set-up system.

13. The local set-up system of claim 9, wherein interaction with the virtual object includes pulling or pushing the virtual object in a direction that is approximately orthogonal to the mirror functionality.

14. The local set-up system of claim 5, further comprising a management module configured to receive information specified by the local participant during the tele-immersive session, to provide participant-specified information,

wherein the local processing system is configured to present the participant-specified information using the mirror functionality.

15. The local set-up system of claim 5, further comprising a management module configured to provide a control feature and to manage interaction with the control feature,

wherein the local processing system is configured to present the control feature using the mirror functionality.

16. The local set-up system of claim 5, wherein the mirror functionality includes a display device having a curved display surface for displaying the participants of the tele-immersive session.

17. The local set-up system of claim 5, wherein the mirror functionality uses a handheld display device for presenting virtual images of two or more participants of the tele-immersive session.

18. The local set-up system of claim 5, wherein the three-dimensional scene information composed by the local processing system represents:

a virtual-reflected space that includes one or more virtual-reflected objects that are projected from a perspective of reflections in a mirror surface; and

a virtual-actual space that includes one or more virtual-actual objects that are projected from a perspective of entities that are placed before the mirror surface.

19. The local set-up system of claim 18, wherein the local processing system includes functionality that enables the local participant to interact with any virtual-reflected object and any virtual-actual object.

20. A method implemented by a local set-up system comprising one or more computer devices, the method comprising:

receiving local camera information from local image capture functionality provided at the local set-up system;

generating local three-dimensional object information based on the local camera information;

transferring local input information to at least one remote set-up system operating in a different geographical location than the local set-up system, the remote set-up system being associated with at least one remote participant of a tele-immersive session, the local input information corresponding to information associated with objects identified by the local set-up system;

receiving remote input information from said at least one remote set-up system, the remote input information corresponding to information associated with objects identified by said at least one remote set-up system;

composing three-dimensional scene information based on the local three-dimensional object information and the remote input information; and

providing the three-dimensional scene information to mirror functionality, the mirror functionality being configured to present a three-dimensional virtual space based on the three-dimensional scene information, the virtual space showing at least some of the participants as if the participants were physically present at a same location and looking into a mirror.