Mixed-reality image capture

• US 9,824,499 B2
• Filed: 06/23/2015
• Issued: 11/21/2017
• Est. Priority Date: 06/23/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Wright, Shawn Crispin, Osborn, Dan, Cerriteno, Alberto E., Hsu, Bofan, Jones, Evan, Thompson, Joe, Gouin, Forest Woodcroft, Saunders, Megan
• Primary Examiner(s): Mushambo, Martin
• Application Number:
  US14/747,997
• Publication Number:
  US 20160379408A1
• Time in Patent Office: 882 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0141 characterised by the informative content of the display
• CPC Class Codes: G02B 2027/0174 holographic
• CPC Class Codes: G02B 2027/0198 System for aligning or maintaining alignment of an image in a predetermined direction
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A head-mounted display includes a visible-light camera configured to collect a visible-light image of a physical space, a surface sensor configured to measure one or more surface parameters of the physical space, a see-through display configured to visually present an augmentation image while light from the physical space passes through the see-through display to a user eye, and an augmented-reality engine. The augmented-reality engine may be configured to identify a surface of the physical space from the one or more measured surface parameters, compose a mixed-reality image that includes the augmentation image overlaid on the visible-light image, and visually present, via the see-through display, the mixed-reality image in alignment with the identified surface.

Claims

1. A head-mounted display device comprising:

a visible-light camera configured to collect a visible-light image of a physical space;

a surface sensor configured to measure one or more surface parameters of the physical space;

a see-through display configured to visually present an augmentation image while light from the physical space passes through the see-through display to a user eye; and

an augmented-reality engine configured to;

identify a surface of the physical space from the one or more measured surface parameters,

determine an angle of the identified surface relative to a perspective of the see-through display;

compose a mixed-reality image that includes the augmentation image overlaid on the visible-light image, and

visually present, via the see-through display, the mixed-reality image in alignment with the identified surface at a non-parallel angle relative to the identified surface if the angle is less than a threshold angle.

2. The head-mounted display device of claim 1, further comprising:

a gaze sensor configured to measure one or more gaze parameters; and

wherein the augmented-reality engine is configured to recognize a gaze vector from the one or more gaze parameters, and wherein the identified surface is a surface nearest the gaze vector.

3. The head-mounted display device of claim 1, wherein the surface sensor is a depth camera, and wherein the one or more surface parameters include a relative surface depth measured by the depth camera.

4. The head-mounted display device of claim 1, wherein the mixed-reality image is aligned at an angle that is parallel with the identified surface.

5. The head-mounted display device of claim 1, wherein the mixed-reality image is automatically visually presented via the see-through display responsive to a user request during use of the head-mounted display device.

6. The head-mounted display device of claim 1, wherein the mixed-reality image is automatically visually presented via the see-through display responsive to the visible-light camera collecting the visible light image.

7. The head-mounted display device of claim 1, wherein the see-through display is configured to visually present an animation in which the mixed-reality image appears to move away from the user eye into alignment with the identified surface of the physical space.

8. The head-mounted display device of claim 1, wherein the augmented-reality engine is configured to visually present, via the see-through display, a virtual adjustment control for adjusting a size or position of the mixed-reality image in the physical space.

9. A method for controlling an augmented-reality device, comprising:

collecting, with a visible-light camera of the augmented-reality device, a visible-light image of a physical space;

measuring, with a surface sensor of the augmented-reality device, one or more surface parameters of the physical space;

identifying a surface of the physical space from the one or more measured surface parameters;

determining an angle of the identified surface relative to a perspective of a user of the augmented-reality device; and

visually presenting, via a display of the augmented-reality device, the visible-light image in alignment with the identified surface at a non-parallel angle relative to the identified surface if the angle is less than a threshold angle from a perspective of a user of the augmented-reality device.

10. The method of claim 9, further comprising

visually presenting, via the display of the augmented-reality device, one or more virtual objects overlaid on the physical space;

composing a mixed-reality image that includes the one or more virtual objects overlaid on the visible-light image of the physical space; and

visually presenting, via the display of the augmented-reality device, the mixed-reality image in alignment with the identified surface from the perspective of the user of the augmented-reality device.

11. The method of claim 9, further comprising:

measuring, with a gaze sensor of the augmented-reality device, one or more gaze parameters of a user eye; and

recognizing a gaze vector of the user eye from the one or more gaze parameters, and wherein the identified surface is a surface nearest the gaze vector.

12. The method of claim 9, wherein the visible-light image is aligned at an angle that is parallel with the identified surface.

13. The method of claim 9, wherein the visible-light image is automatically visually presented via the display responsive to the visible-light camera collecting the visible light image.

14. The method of claim 9, further comprising:

visually presenting, via the display of the augmented-reality device, an animation in which the visible-light image appears to move away from a perspective of the user of the augmented-reality device into alignment with the identified surface of the physical space.

15. A method for controlling an augmented-reality device, comprising:

measuring, with a surface sensor of the augmented-reality device, one or more surface parameters of a physical space;

identifying one or more surfaces of the physical space from the one or more measured surface parameters;

visually presenting, via a see-through display of the augmented-reality device, an augmentation image while light from the physical space passes through the see-through display to a user eye;

collecting, with a visible-light camera of the augmented-reality device, a visible-light image of the physical space;

composing a mixed-reality image that includes the augmentation image overlaid on the visible-light image;

measuring, with a gaze sensor of the augmented-reality device, one or more gaze parameters of a user eye;

recognizing a gaze vector from the one or more gaze parameters;

determining an angle of the identified surface relative to a perspective of the user eye; and

automatically visually presenting, via the see-through display of the augmented-reality device, the mixed-reality image in alignment with an identified surface nearest the gaze vector at a non-parallel angle relative to the identified surface if the angle is less than a threshold angle.

16. The method of claim 15, wherein the mixed-reality image is automatically visually presented via the see-through display responsive to the visible-light camera collecting the visible light image.

17. The method of claim 15, further comprising:

visually presenting, via the see-through display of the augmented-reality device, an animation in which the mixed-reality image appears to move away from a perspective of the user of the augmented-reality device into alignment with the identified surface of the physical space.