Holographic bird's eye view camera

• US 9,846,968 B2
• Filed: 06/02/2015
• Issued: 12/19/2017
• Est. Priority Date: 01/20/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Tomlin, Arthur Charles, Keibler, Evan Michael, Fajt, Nicholas Gervase, Mount, Brian J., Alt, Gregory Lowell, Tosar, Jorge, Lyons, Jonathan Michael, Ambrus, Anthony J., Egbert, Cameron Quinn, Guyman, Will, McGlynn, Jeff W., Hance, Jeremy, Sylvan, Roger Sebastian-Kevin, Pfaffe, Alexander Georg, Kroymann, Dan, Saltwell, Erik Andrew, Word, Chris
• Primary Examiner(s): Yang, Ryan R
• Application Number:
  US14/728,650
• Publication Number:
  US 20160210783A1
• Time in Patent Office: 931 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/00 Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 23/51 Housings

Abstract

A system and method are disclosed for capturing views of a mixed reality environment from various perspectives which can be displayed on a monitor. The system includes one or more physical cameras at user-defined positions within the mixed reality environment. The system renders virtual objects in the mixed reality environment from the perspective of the one or more cameras. Real and virtual objects from the mixed reality environment may then be displayed from the perspective of the one or more cameras on one or more external, 2D monitor for viewing by others.

Claims

1. A system for presenting a mixed reality environment including real and virtual objects, the system comprising:

a head mounted display device including a display unit for displaying a three-dimensional virtual object in the virtual environment and a first set of one or more cameras;

one or more camera assemblies, remote from the head mounted display device, for capturing video of real objects in the mixed reality environment; and

a processing unit operatively coupled to the display device and the one or more camera assemblies, the processing unit generating a scene map comprising a three-dimensional coordinate space in which the head mounted display, the three-dimensional virtual object and the one or more camera assemblies are registered, the processing unit and a camera assembly of the one or more camera assemblies generating an image of the mixed reality environment including video of real objects and the virtual object for display from a perspective of the camera assembly.

2. The system of claim 1, the camera assembly including an image capture device for capturing a color or black and white image of real objects in the mixed reality environment.

3. The system of claim 1, the camera assembly including an image sensor for capturing depth data of the mixed reality environment, the depth data used to generate a depth map of the mixed reality environment from a perspective of the image sensor.

4. The system of claim 1, the camera assembly and the head mounted display device calibrated to a common coordinate system by the camera assembly and the head mounted display device capturing one or more images of a calibration cue.

5. The system of claim 4, wherein, the processing unit determining poses of the camera assembly and head mounted display device based on depth imaging sensor data from the one or more images of the calibration target from perspectives of the camera assembly and the head mounted display device.

6. The system of claim 1, the camera assembly calibrated in the three-dimensional coordinate space of the scene map using at least one of a depth map and an RGB image generated at least in part by the camera assembly.

7. The system of claim 1, the camera assembly including a position sensor including an inertial measurement unit for assisting in registering the camera assembly within the three-dimensional coordinate space of the scene map.

8. The system of claim 1, the camera assembly being a first camera assembly, the one or more camera assemblies further comprising a second camera assembly, and the system further comprising a selection device for selecting a display of the mixed reality environment from a perspective of either of the first and second camera assemblies.

9. The system of claim 8, further comprising at least first and second two-dimensional monitors, the first monitor displaying the image of the mixed reality environment from the perspective of the first camera assembly and the second monitor displaying the image of the mixed reality environment from the perspective of the second camera assembly.

10. A system for presenting a mixed reality environment including real and virtual objects, the system comprising:

a plurality of image capture devices for capturing images of real world objects in the mixed reality environment from different perspectives; and

a processing unit operatively coupled to the plurality of image capture devices, the processing unit identifying poses of the plurality of image capture devices with respect to each other, the processing unit further generating a virtual object in the mixed reality environment and identifying a position of the virtual object with respect to the plurality of image capture devices, the processing unit and an image capture device of the plurality of image capture devices generating an image of the mixed reality environment for display on a two-dimensional monitor from a perspective of the image capture device.

11. The system of claim 10, wherein the image capture device comprises a camera on a head mounted display device, the head mounted display device further comprising a display unit for displaying the virtual object to optics in the head mounted display device.

12. The system of claim 10, wherein the image capture device comprises a camera assembly.

13. The system of claim 12, the camera assembly comprising a color image capture device for capturing a color image of real objects in the mixed reality environment, and an image sensor for capturing depth data of the mixed reality environment, from which depth data a depth map of the mixed reality environment may be generated from a perspective of the image sensor.

14. The system of claim 13, the processing unit identifying the pose of the camera assembly relative to other image capture devices using at least one of the RGB image and the depth map.

15. The system of claim 12, the camera assembly including a position sensor including an inertial measurement unit for assisting in identifying the pose of the camera assembly relative to other image capture devices.

16. A method of presenting a mixed reality environment including real and virtual objects, the method comprising:

(a) capturing a first image of the mixed reality environment from a first perspective via a first camera incorporated into a head mounted display device;

(b) capturing a second image of the mixed reality environment from a second perspective via a second camera incorporated into a camera assembly;

(c) generating a virtual object at a position in the mixed reality environment;

(d) displaying the virtual object onto optics on the head mounted display device, the virtual object being displayed from a perspective of the head mounted display device; and

(e) displaying the second image from the camera assembly together with the virtual object onto a two-dimensional display device, the virtual object being displayed from a perspective of the camera assembly.

17. The method of claim 16, further comprising the step of changing a display on the two-dimensional display device to displaying the first image from the head mounted display device together with the virtual object, the virtual object being displayed from a perspective of the head mounted display device.

18. The method of claim 16, further comprising the step of registering poses of the head mounted display device, the camera assembly and the virtual object to each other using at least objects in common in the first and second images.

19. The method of claim 16, further comprising the step of capturing the second image by camera assembly from one of the sides, below and above the position of the virtual object.

20. The method of claim 16, further comprising the step of a wearer of the head mounted display device constructing the virtual object using virtual object tools, the second image from the camera assembly displayed on the two-dimensional monitor comprising a video feed showing the virtual tools and the virtual object as it is being constructed.