PROVIDING A TELE-IMMERSIVE EXPERIENCE USING A MIRROR METAPHOR

• US 20140232816A1
• Filed: 02/20/2013
• Published: 08/21/2014
• Est. Priority Date: 02/20/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Wilson, Andrew D., Chou, Philip A., Gillett, Donald M., Benko, Hrvoje, Zhang, Zhengyou, Fishman, Neil S.
• Granted Patent: US 9,325,943 B2
• Time in Patent Office: Days
• US Class Current: 348/14.08
• CPC Class Codes: A47G 1/02 Mirrors used as equipment hand, pocket, or shaving mirrors A45D42/00; optical viewing means for vehicles B60R1/00
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06T 17/20 Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827

Abstract

A tele-immersive environment is described that provides interaction among participants of a tele-immersive session. The environment includes two or more set-ups, each associated with a participant. Each set-up, in turn, includes mirror functionality for presenting a three-dimensional virtual space for viewing by a local participant. The virtual space shows at least some of the participants as if the participants were physically present at a same location and looking into a mirror. The mirror functionality can be implemented as a combination of a semi-transparent mirror and a display device, or just a display device acting alone. According to another feature, the environment may present a virtual object in a manner that allows any of the participants of the tele-immersive session to interact with the virtual object.

Claims

1. A tele-immersive environment for providing interaction among participants of a tele-immersive session, the tele-immersive environment comprising:

two or more set-ups, each set-up providing an interactive experience to a local participant of the tele-immersive session, and each set-up comprising;

image capture functionality configured to capture local camera information that represents the local participant associated with the set-up;

a local processing system configured to;

receive the local camera information;

generate local three-dimensional object information based on the local camera information;

transfer local input information to at least one remote processing system associated with at least one remote participant, the local input information corresponding to any information associated with objects identified by the local processing system;

receive remote input information from said at least one remote processing system, the remote input information corresponding to any information associated with objects identified by said at least one remote processing system; and

compose three-dimensional scene information based on the local three-dimensional object information and the remote input information; and

mirror functionality configured to present a three-dimensional virtual space based on the three-dimensional scene information, the three-dimensional virtual space showing at least some of the participants as if the participants were physically present at a same location and looking into a mirror.

2. The tele-immersive environment of claim 1, wherein the image capture functionality is configured to provide information for use in constructing a depth image of the local participant.

3. The tele-immersive environment of claim 1, wherein the mirror functionality comprises:

a physical semi-transparent mirror for presenting a virtual image of the local participant at the set-up; and

a display device configured to receive the three-dimensional scene information, and based thereon, present a virtual image of at least one remote participant.

4. The tele-immersive environment of claim 1, wherein the mirror functionality comprises a display device configured to receive the three-dimensional scene information, and based thereon, present a virtual image of the local participant and at least one remote participant.

5. A local set-up in a tele-immersive environment, the tele-immersive environment for providing interaction among participants of a tele-immersive session, the local set-up comprising:

image capture functionality configured to capture local camera information that represents a local participant associated with the local set-up;

a local processing system configured to;

compose three-dimensional scene information based on the local camera information and remote input information, the remote input information being provided by at least one remote processing system associated with at least one remote participant, the remote input information corresponding to any information associated with objects identified by said at least one remote processing system; and

transfer at least some of local input information to said at least one remote processing system, the local input information corresponding to any information associated with objects identified by the local processing system; and

mirror functionality configured to present a three-dimensional virtual space based on the three-dimensional scene information, the three-dimensional virtual space showing at least some of the participants as if the participants were physically present at a same location and looking into a mirror.

6. The local set-up of claim 5, wherein the image capture functionality is configured to provide information for use in constructing a depth image of the local participant.

7. The local set-up of claim 5, wherein the mirror functionality comprises:

a physical semi-transparent mirror for presenting a virtual image of the local participant at the local set-up; and

a display device configured to receive the three-dimensional scene information, and based thereon, present a virtual image of at least one remote participant.

8. The local set-up of claim 5, wherein the mirror functionality comprises a display device configured to receive the three-dimensional scene information, and based thereon, present a virtual image of the local participant and at least one remote participant.

9. The local set-up of claim 5, wherein the local processing system includes functionality that enables the local participant to interact with a virtual object presented by the mirror functionality.

10. The local set-up of claim 9, wherein the local set-up includes a physical workspace in which the local participant may place a physical object, and wherein the virtual object is a virtual counterpart of the physical object.

11. The local set-up of claim 10, wherein the physical workspace includes a workspace table on which the physical object is placed.

12. The local set-up of claim 9, wherein the virtual object has no counterpart physical object in a real space associated with any set-up.

13. The local set-up of claim 9, wherein interaction with the virtual object includes pulling or pushing the virtual object in a direction that is approximately orthogonal to the mirror functionality.

14. The local set-up of claim 5, further comprising a management module configured to receive information specified by the local participant during the tele-immersive session, to provide participant-specified information,

wherein the local processing module is configured to present the participant-specified information using the mirror functionality.

15. The local set-up of claim 5, further comprising a management module configured to provide a control feature and to manage interaction with the control feature,

wherein the local processing module is configured to present the control feature using the mirror functionality.

16. The local set-up of claim 5, wherein the mirror functionality includes a display device having a curved display surface for displaying the participants of the tele-immersive session.

17. The local set-up of claim 5, wherein the mirror functionality uses a handheld display device for presenting virtual images of two or more participants of the tele-immersive session.

18. The local set-up of claim 5, wherein the three-dimensional scene information generated by the local processing module represents:

a virtual-reflected space that includes one or more virtual-reflected objects that are projected from a perspective of reflections in a mirror surface; and

a virtual-actual space that includes one or more virtual-actual objects that are projected from a perspective of entities that are placed before the mirror surface.

19. The local set-up of claim 18, wherein the local processing system includes functionality that enables the local participant to interact with any virtual-reflected object and any virtual-actual object.

20. A method, implemented by one or more computer devices, comprising:

receiving local camera information from local image capture functionality provided at a local set-up, the local set-up comprising one of a set of set-ups for providing an interactive experience to participants of a tele-immersive session;

generating local three-dimensional object information based on the local camera information;

transferring local input information to at least one remote set-up associated with at least one remote participant of the tele-immersive session, the local input information corresponding to any information associated with objects identified by the local set-up;

receiving remote input information from said at least one remote set-up, the remote input information corresponding to any information associated with objects identified by said at least one remote set-up;

composing three-dimensional scene information based on the local three-object dimensional information and the remote input information; and

providing the three-dimensional scene information to mirror functionality, the mirror functionality being configured to present a three-dimensional virtual space based on the three-dimensional scene information, the virtual space showing at least some of the participants as if the participants were physically present at a same location and looking into a mirror.