FUSING VIRTUAL CONTENT INTO REAL CONTENT

• US 20120092328A1
• Filed: 10/15/2010
• Published: 04/19/2012
• Est. Priority Date: 10/15/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Flaks, Jason, Bar-Zeev, Avi, Margolis, Jeffrey Neil, Kipman, Alex Aben-Athar, Fuller, Andrew John, Miles, Chris, Crocco, Bob Jr.
• Granted Patent: US 8,884,984 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G02B 2027/0127 comprising devices increasing the depth of field
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59

Abstract

A system that includes a head mounted display device and a processing unit connected to the head mounted display device is used to fuse virtual content into real content. In one embodiment, the processing unit is in communication with a hub computing device. The system creates a volumetric model of a space, segments the model into objects, identifies one or more of the objects including a first object, and displays a virtual image over the first object on a display (of the head mounted display) that allows actual direct viewing of at least a portion of the space through the display.

Claims

1. A method for fusing virtual content into real content, comprising:

creating a volumetric model of a space;

segmenting the model into objects;

identifying one or more of the objects including a first object; and

automatically displaying a virtual image over the first object on a display that allows actual direct viewing of at least a portion of the space through the display.

2. The method of claim 1, wherein the creating a volumetric model comprises:

receiving depth images from multiple perspectives; and

using data from the depth images to create and store a three dimensional description of the space.

3. The method of claim 1, further comprising:

specifying that the first object is to be replaced; and

identifying the virtual image to replace the first object.

4. The method of claim 1, wherein the segmenting the model into objects comprises:

accessing one or more depth images;

accessing one or more visual images;

detecting one or more persons using the one or more depth images and the one or more visual images;

detecting edges based on the one or more depth images and the one or more visual images;

detecting objects based on the edges; and

updating the model to store information about the detection of the objects and person.

5. The method of claim 4, wherein the identifying one or more of the objects comprises:

matching at least one of the detected objects to a user identity;

accessing a data structure of shapes;

matching one or more of the detected objects to shapes in the data structure; and

updating the model to reflect the matching of the at least one of the detected objects to the user identity and the matching of the one or more of the detected objects to shapes in the data structure.

6. The method of claim 5, wherein the identifying one or more of the objects comprises:

visually displaying an unmatched object to the user and requesting the user to identify the unmatched object;

receiving a new identification for the unmatched object;

updating the data structure to reflect the new identification; and

updating the model to store information about the new identification for the unmatched object.

7. The method of claim 1, wherein the automatically displaying the virtual image over the first object further comprises:

identifying a light source; and

adjusting the virtual image based on the light source.

8. The method of claim 1, wherein the automatically displaying the virtual image over the first object further comprises:

identifying a shadow area; and

darkening a view of a real object corresponding to the shadow area.

9. The method of claim 1, wherein:

the displaying of the virtual image over the first object on the display comprises determining point of view of a user of the display; and

the determining field of view of the user of the display comprises

automatically determining location and orientation of the display using one or more depth images, automatically refining the determined orientation of the display using inertial sensors and automatically determining position of an eye of the user of the display to identify a subset of the space viewed at the determined location and orientation.

10. The method of claim 1, wherein the displaying of the virtual image over the first object on the display further comprises:

automatically determining that the first object is moving;

automatically tracking movement of the first object;

automatically determining a change in shape of the first object; and

blurring the virtual image in response to movement of the first object.

11. The method of claim 1, wherein the displaying of the virtual image over the first object on the display further comprises:

identifying a portion of the display that potentially shows the first object; and

selectively blocking light from passing through the portion of the display that potentially shows the first object, the virtual image is projected on the portion of the display that potentially shows the first object.

12. The method of claim 1, wherein:

the displaying of the virtual image over the first object on the display further comprises automatically determining location and orientation of the display using one or more depth images, automatically refining the determined orientation of the display using inertial sensors, automatically determining position of an eye of the user of the display to identify a subset of the space viewed at the determined location and orientation;

the creating the volumetric model comprises receiving depth images from multiple perspectives and using data from the depth images to create and store a three dimensional description of the space; and

the segmenting the model into objects comprises accessing one or more depth images, accessing one or more visual images, detecting one or more persons using the one or more depth images and the one or more visual images, detecting edges based on the one or more depth images and the one or more visual images, detecting objects based on the edges and updating the model to store information about the detection of the objects.

13. The method of claim 1, wherein:

the displaying of the virtual image includes projecting the virtual image on the display without projecting the portion of the space on the display; and

the first object is a portion of a person.

14. An apparatus for fusing virtual content into real content, comprising:

a head mounted display device, the head mounted display device comprises;

a display that allows actual direct view of a first object through the display,

an opacity filter aligned with the display so that light passing through the opacity filer also passes through the display,

inertial sensors that sense orientation information for the head mounted display device,

a first communication interface, and

one or more control circuits in communication with the display, opacity filter, inertial sensors and the first communication interface; and

a processing unit, the processing unit comprises a second communication interface that is in communication with the first communication interface and one or more control circuits in communication with the head mounted display device via the first communication interface and the second communication interface to automatically display a virtual image over the first object on the display.

15. The apparatus of claim 14, wherein the display comprises:

an image projector;

a lightguide optical element aligned with the opacity filter so that light passing through the opacity filer also passes through the lightguide optical element, additional light entering the lightguide optical element at a first end will exit the lightguide optical element along a surface in front of an eye of a user of the head mounted display device; and

a lens that is aligned with the image projector to direct light from the image projector into the lightguide optical element.

16. The apparatus of claim 14, wherein the head mounted display device further comprises:

an illumination source for projecting light onto an eye of a user of the head mounted display device; and

a sensor that senses light reflected back from the eye of the user, the sensor is in communication with the one or more control circuits of the head mounted display so that the apparatus can determine the direction of the eye of the user.

17. The apparatus of claim 14, wherein:

the second communication interface wirelessly communicates with a hub computing device to receive a three dimensional model of a space that identifies one or more objects; and

the one or more control circuits of the processing unit determine a field of view of a user of the head mounted display device, determine if the first object is in the field of view, determine a position of the first object in the display, adjusting a size and orientation of the virtual image based size and orientation of the first object in the display and send a signal to the display to cause the display to project the adjusted virtual image on the display to replace a view of the first object.

18. The apparatus of claim 17, wherein:

the one or more control circuits of the processing unit identify a portion of the display that potentially shows the first object and sends a signal to the opacity filter to selectively block light from passing through the portion of the display that potentially shows the first object, the virtual image is projected on the portion of the display that potentially shows the first object.

19. A method for fusing virtual content into real content, comprising:

creating a volumetric model of a space using one or more depth images;

identifying a first object using the model;

automatically tracking the first object as the first object moves;

automatically determining a change in shape of the first object as it moves; and

automatically displaying a virtual image over the first object on a display that allows actual direct viewing of at least a portion of the space through the display such that the first object is no longer visible.

20. The method of claim 19, wherein:

the automatically displaying the warped virtual image over the first object on the display further comprises identifying a portion of the display that potentially shows the first object and selectively blocking light from passing through the portion of the display that potentially shows the first object, the virtual image is projected on the portion of the display that potentially shows the first object; and

the creating the volumetric model includes mapping lighting and reflectivity of surfaces for reflections and shadows.