Calibrating a near-eye display

• US 10,630,965 B2
• Filed: 10/02/2015
• Issued: 04/21/2020
• Est. Priority Date: 10/02/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Held, Robert Thomas, Gavriliuc, Anatolie, Giraldi, Riccardo, Stachniak, Szymon P., Muehlhausen, Andrew Frederick, Ouellet, Maxime
• Primary Examiner(s): Dang, Philip P.
• Application Number:
  US14/873,928
• Publication Number:
  US 20170099481A1
• Time in Patent Office: 1,663 Days
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0174 holographic
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/327 Calibration thereof
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
• CPC Class Codes: H04N 13/398 Synchronisation thereof; Control thereof

Abstract

Examples are disclosed herein that relate to calibrating a user'"'"'s eye for a stereoscopic display. One example provides, on a head-mounted display device including a see-through display, a method of calibrating a stereoscopic display for a user'"'"'s eyes, the method including for a first eye, receiving an indication of alignment of a user-controlled object with a first eye reference object viewable via the head-mounted display device from a perspective of the first eye, determining a first ray intersecting the user-controlled object and the first eye reference object from the perspective of the first eye, and determining a position of the first eye based on the first ray. The method further includes repeating such steps for a second eye, determining a position of the second eye based on a second ray, and calibrating the stereoscopic display based on the position of the first eye and the position of the second eye.

Claims

1. On a head-mounted display device comprising a see-through stereoscopic display and memory storing instructions executable by a logic subsystem, a method of calibrating the see-through stereoscopic display for a user'"'"'s eyes, the method comprising:

for a first eye,

receiving first eye image data from one or more image sensors, the first eye image data comprising an image of a user-controlled object for the first eye, the user-controlled object for the first eye comprising a body part of the user or an object held by the user,

receiving a user input indicating an alignment of the user-controlled object for the first eye with a first eye reference object viewable via the head-mounted display device from a perspective of the first eye,

determining via the first eye image data a first ray intersecting the user-controlled object for the first eye and the first eye reference object from the perspective of the first eye, and

determining a position of the first eye based on a location at which the first ray intersects a plane passing through the first eye;

separately, for a second eye,

receiving second eye image data from the one or more image sensors, the second eye image data comprising an image of a user-controlled object for the second eye, the user-controlled object for the second eye comprising the body part of the user, another body part of the user, or the object held by the user,

receiving a user input indicating an alignment of the user-controlled object for the second eye with a second eye reference object viewable via the head-mounted display device from a perspective of the second eye,

determining via the second eye image data a second ray intersecting the user-controlled object for the second eye and the second eye reference object from the perspective of the second eye, and

determining a position of the second eye based on a location at which the second ray intersects a plane passing through the second eye;

calibrating the see-through stereoscopic display based on the position determined for the first eye and the position determined for the second eye; and

displaying a stereoscopic image comprising a first eye image and a second eye image based on calibrating.

2. The method of claim 1, wherein the position determined for the first eye and the position determined for the second eye each comprises an x-position, a y-position, and a z-position.

3. The method of claim 1, wherein the user-controlled object for the first eye and the user-controlled object for the second eye each comprises a finger of the user.

4. The method of claim 1, wherein one or more of the first eye reference object and the second eye reference object comprises a hologram displayed via the see-through stereoscopic display.

5. The method of claim 1, wherein one or more of the first eye reference object and the second eye reference object comprises a real-world object.

6. The method of claim 1, further comprising,

receiving an additional user input indicating an alignment of a user-controlled object with an additional reference object for the first eye,

determining an additional ray intersecting the user-controlled object and the additional reference object to form a plurality of rays from the perspective of the first eye, and

determining the position of the first eye based on a location of convergence of the plurality of rays.

7. The method of claim 1, further comprising isolating each of the first eye and the second eye by instructing the user to close the other of the first eye and the second eye.

8. The method of claim 1, further comprising isolating each of the first eye and the second eye by occluding a view of the other of the first eye and the second eye via the see-through stereoscopic display.

9. A head-mounted display device, comprising:

a depth image sensor;

a see-through stereoscopic display configured to display virtual content and through which a real-world background scene is viewable;

a logic subsystem; and

a storage subsystem comprising instructions executable by the logic subsystem to,

for a first eye,

receive first eye image data from the depth image sensor, the first eye image data comprising an image of a user-controlled object for the first eye, the user-controlled object for the first eye comprising a body part of the user or an object held by the user,

for each reference object of a first plurality of reference objects viewable via the head-mounted display device, receive a user input indicating an alignment of the user-controlled object for the first eye with the reference object, and determine a ray intersecting the user-controlled object for the first eye and the reference object, to form a first plurality of rays based on a perspective of the first eye,

estimate a location of convergence of the first plurality of rays, and

determine a position of the first eye based on the location of convergence of the first plurality of rays;

separately, for a second eye,

receive second eye image data from the depth image sensor, the second eye image data comprising an image of a user-controlled object for the second eye, the user-controlled object for the second eye comprising the body part of the user, another body part of the user, or the object held by the user,

for each reference object of a second plurality of reference objects viewable via the head-mounted display device, receive a user input indicating an alignment of the user-controlled object for the second eye with the reference object, and determine a ray intersecting the user-controlled object for the second eye and the reference object to form a second plurality of rays based on a perspective of the second eye,

estimate a location of convergence of the second plurality of rays, and

determine a position of the second eye based on the location of convergence of the second plurality of rays;

calibrate the see-through stereoscopic display for the user'"'"'s eyes based on the position determined for the first eye and the position determined for the second eye; and

display a stereoscopic image comprising a first eye image and a second eye image based on calibrating the see-through stereoscopic display.

10. The head-mounted display device of claim 9, wherein the position determined for the first eye and the position determined for the second eye each comprises an x-position, a y-position, and a z-position.

11. The head-mounted display device of claim 9, wherein the user-controlled object for the first eye and the user-controlled object for the second eye each comprises a finger of the user.

12. The head-mounted display device of claim 9, wherein one or more of the first plurality of reference objects and the second plurality of reference objects comprises a hologram displayed via the see-through stereoscopic display.

13. The head-mounted display device of claim 12, wherein the instructions are executable to display each hologram of the first plurality of reference objects sequentially and each hologram of the second plurality of reference objects sequentially.

14. The head-mounted display device of claim 9, wherein one or more of the first plurality of reference objects and the second plurality of reference objects comprises a real-world object.

15. The head-mounted display device of claim 9, wherein the instructions are executable to isolate each of the first eye and the second eye by occluding a view of the other of the first eye and the second eye via the see-through stereoscopic display.

16. A head-mounted display device, comprising:

a depth image sensor;

a see-through stereoscopic display configured to display virtual content and through which a real-world background scene is viewable;

a logic subsystem; and

a storage subsystem comprising instructions executable by the logic subsystem to

for a first eye,

sequentially display a first plurality of holograms via the see-through display,

receive first eye image data from the depth image sensor, the first eye image data comprising an image of a user-controlled object for the first eye, the user-controlled object for the first eye comprising a body part of the user or an object held by the user,

for each hologram of the first plurality of holograms, receive a user input indicating an alignment of the user-controlled object for the first eye with the hologram, and determine a ray intersecting the user-controlled object for the first eye and the hologram to form a first set of rays based on a perspective of the first eye,

estimate a location of convergence of the first set of rays, and

determine a position of the first eye based on the location of convergence of the first set of rays;

separately, for a second eye,

sequentially display a second plurality of holograms via the see-through display,

receive second eye image data from the depth image sensor, the second eye image data comprising an image of a user-controlled object for the second eye, the user-controlled object for the second eye comprising the body part of the user, another body part of the user, or the object held by the user,

for each hologram of the second plurality of holograms, receive a user input indicating an alignment of the user-controlled object for the second eye with the hologram, and determine a ray intersecting the user-controlled object for the second eye and the hologram to form a second set of rays based on a perspective of the second eye,

estimate a location of convergence of the second set of rays, and

determine a position of the second eye based on the location of convergence of the second set of rays;

calibrate the see-through stereoscopic display for the user'"'"'s eyes based on the position determined for the first eye and the position determined for the second eye; and

display a stereoscopic image comprising a first eye image and a second eye image based on calibrating the see-through stereoscopic display.

17. The head-mounted display device of claim 16, wherein the position determined for the first eye and the position determined for the second eye each comprises one or more of an x-position, a y-position, and a z-position.

18. The head-mounted display device of claim 16, wherein the user-controlled object for the first eye and the user-controlled object for the second eye each comprises a finger of the user.

19. The head-mounted display device of claim 18, wherein the instructions are executable to display each hologram by displaying a ring, and detect the finger aligning with the hologram by detecting a tip of the finger positioned within the ring.

20. The head-mounted display device of claim 16, wherein the instructions are executable to calibrate the see-through stereoscopic display via an interpupillary distance based on the position determined for the first eye and the position determined for the second eye.