Aligning inter-pupillary distance in a near-eye display system

• US 9,213,163 B2
• Filed: 08/30/2011
• Issued: 12/15/2015
• Est. Priority Date: 08/30/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Lewis, John R., Wei, Yichen, Crocco, Robert L., Vaught, Benjamin I., Perez, Kathryn Stone, Kipman, Alex Aben-Athar
• Primary Examiner(s): PINKNEY, DAWAYNE
• Application Number:
  US13/221,662
• Publication Number:
  US 20130050642A1
• Time in Patent Office: 1,568 Days
• Field of Search: 359630-634, 359 13- 14, 310/49.R, 310/156.32, 310/156.33, 310/156.34, 310/156.35, 310266-268, 310/156.02, 353 11- 12, 353/28, 353/119, 345/7, 345/9, 340/438, 340/980, 340/995.1, 340/815.47, 340/815.74, 351/200, 351205-206, 351/210, 351/221
• US Class Current: 1/1
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: G02B 2027/0127 comprising devices increasing the depth of field
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 7/12 Adjusting pupillary distance of binocular pairs

Abstract

The technology provides for automatic alignment of a see-through near-eye, mixed reality device with an inter-pupillary distance (IPD). A determination is made as to whether a see-through, near-eye, mixed reality display device is aligned with an IPD of a user. If the display device is not aligned with the IPD, the display device is automatically adjusted. In some examples, the alignment determination is based on determinations of whether an optical axis of each display optical system positioned to be seen through by a respective eye is aligned with a pupil of the respective eye in accordance with an alignment criteria. The pupil alignment may be determined based on an arrangement of gaze detection elements for each display optical system including at least one sensor for capturing data of the respective eye and the captured data. The captured data may be image data, image and glint data, and glint data only.

Claims

1. A method for automatically aligning a near-eye, mixed reality display device with an inter-pupillary distance (IPD) for a wearer, comprising:

automatically determining whether the near-eye, mixed reality display device, including a respective display optical system for each eye of the wearer which is positioned to be seen through by the respective eye, is aligned with a wearer IPD in accordance with an alignment criteria based on determining whether each optical axis of the respective display optical system positioned in the display device satisfies the alignment criteria for alignment with a respective pupil of the wearer; and

responsive to the display device not being aligned with the wearer IPD in accordance with the alignment criteria, automatically adjusting a position of one or both of the display optical systems within the display device for bringing the display device into alignment with the wearer IPD, wherein the automatically adjusting comprises

determining at least one adjustment value for a respective display adjustment mechanism based on a mapping criteria of the at least one sensor for each display optical system not satisfying the alignment criteria; and

moving the respective display optical system based on the at least one adjustment value.

2. The method of claim 1 further comprising:

responsive to the display device being aligned with the user IPD in accordance with the alignment criteria, storing an IPD data set for the user comprising a position of each optical axis of each display optical system of the display device.

3. The method of claim 1 wherein automatically determining whether the near-eye, mixed reality display device, including a respective display optical system for each eye which is positioned to be seen through by the respective eye, is aligned with the wearer IPD in accordance with the alignment criteria based on determining whether each optical axis of a respective display optical system positioned in the display device satisfies alignment criteria for alignment with a respective pupil of the user further comprises:

identifying an object in the user field of view at a distance and a direction for determining an IPD;

performing processing for drawing the wearer'"'"'s focus to the object;

capturing data of each eye during an observation period for the object by at least one sensor in an arrangement of gaze detection elements for the respective display optical system; and

determining whether each pupil is aligned with its respective optical axis in accordance with alignment criteria based on the captured data and the arrangement of gaze detection elements.

4. The method of claim 1 wherein automatically determining whether the near-eye, mixed reality display device, including a respective display optical system for each eye which is positioned to be seen through by the respective eye, is aligned with the wearer IPD in accordance with the alignment criteria based on determining whether each optical axis of a respective display optical system positioned in the display device satisfies alignment criteria for alignment with a respective pupil of the user further comprises:

identifying an object in the wearer field of view at a distance and a direction for determining an IPD;

performing processing for drawing the wearer'"'"'s focus to the object;

determining a reference gaze vector for each eye to the real object which passes through the optical axis of the respective display optical system for each eye based on an arrangement of gaze detection elements for the display optical system;

capturing data of each eye by at least one sensor of the arrangement during an observation period for the real object;

determining a current gaze vector for each eye based on the captured data and the arrangement; and

determining whether each pupil is aligned with its respective optical axis in accordance with an alignment criteria be determining whether the current gaze vector for each eye matches the reference gaze vector for each eye.

5. The method of claim 1 wherein automatically determining whether the near-eye, mixed reality display device, including a respective display optical system for each eye which is positioned to be seen through by the respective eye, is aligned with the wearer IPD in accordance with the alignment criteria based on determining whether each optical axis of a respective display optical system positioned in the display device satisfies alignment criteria for alignment with a respective pupil of the wearer further comprises:

identifying a real object in the wearer field of view at a distance and a direction for determining an IPD;

performing processing for drawing the wearer focus to the real object;

capturing image data of each eye in an image format during an observation period for the real object by at least one sensor aligned with the optical axis of the respective display optical system of the display device, each display optical system being positioned to be seen through by a respective eye;

determining a respective pupil position from the optical axis from the image data; and

determining whether each pupil aligned is aligned with its respective optical axis based on the pupil position in the image format in accordance with alignment criteria.

6. The method of claim 1 wherein determining at least one adjustment value for a respective display adjustment mechanism based on a mapping criteria of the at least one sensor for each display optical system not satisfying the alignment criteria further comprises:

based on the mapping criteria for the at least one sensor, determining a horizontal pupil position difference vector;

based on the mapping criteria for the at least one sensor, determining a vertical pupil position difference vector;

correlating the horizontal pupil position difference vector to a horizontal adjustment value; and

correlating the vertical pupil position difference vector to a vertical adjustment value.

7. The method of claim 6 further comprising:

determining whether any of the horizontal or vertical adjustment values are out of range; and

responsive to any horizontal or vertical adjustment value being out of range, determining a depth adjustment value for bringing any out of range adjustment value closer to being within range.

8. A system for automatically aligning a near-eye, mixed reality display with an inter-pupillary distance (IPD) of a user comprising:

a near-eye, mixed reality display device having a support structure including independently movable support structures, and having attached to a respective independently movable support structure a respective display optical system for each eye positioned to be seen through by the respective eye, each display optical system having an optical axis;

a display adjustment mechanism attached to at least one independently movable support structure for automatically adjusting a position of the respective display optical system;

a respective arrangement of gaze detection elements positioned with respect to each display optical system on the respective independently movable support structure for forming a spatial relationship between the gaze detection elements and the respective eye;

each respective arrangement including at least one respective sensor having a detection area for capturing data of the respective eye of the respective display optical system;

a memory for storing software and data including the captured data of each eye; and

one or more processors having access to the memory and being communicatively coupled to the display adjustment mechanism for automatically re-positioning at least one display optical system within the display device for alignment of the optical axes of the display optical systems with the inter-pupillary distance based on the captured data and the arrangement of gaze detection elements.

9. The system of claim 8 further comprising an environment facing camera positioned on the display device to have a front facing field of view and being communicatively coupled to the one or more processors for providing image data from which the one or more processors determine a user field of view including at least a real object.

10. The system of claim 8 wherein the at least one respective sensor is an infra-red (IR) sensor;

and the arrangement of gaze detection elements includes IR illuminators for generating glints, the IR illuminators being at fixed positions on the respective independently movable support structure with respect to a detection area of the IR sensor;

the IR sensor capturing data of glints on the eye associated with the respective display optical system; and

the processor determining a gaze vector for the eye based on the data of glints and determining whether a pupil is aligned with the optical axis of the respective display optical system based on the gaze vector.

11. The system of claim 8 wherein the at least one respective sensor is an infra-red (IR) camera;

and the arrangement of gaze detection elements includes IR illuminators for generating glints, the IR illuminators being at fixed positions on the respective independently movable support structure with respect to a detection area of the IR camera;

the IR camera capturing image data of the respective eye of the respective display optical system including any glints reflecting off the eye;

the processor determining a gaze vector from an inner part of the eye to an object in a user field of view based on determining a center of a cornea based on the image data of the glints and on determining a center of a pupil based on the image data of the eye; and

the processor determining whether a pupil is aligned with the optical axis of the respective display optical system based on the gaze vector.

12. The system of claim 8 wherein the at least one respective sensor is an infra-red (IR) camera having a detection area aligned with an optical axis of the respective display optical system and capturing image data of the respective eye of the respective display optical system;

and

the processor determines whether a pupil of the respective eye is aligned with the optical axis of the respective display optical system based on a pupil position in the image data.

13. The system of claim 10 wherein the detection area of the IR camera is centered on the optical axis of the respective display optical system.

14. A method for automatically aligning a near-eye, mixed reality display device with an inter-pupillary distance (IPD) of a wearer, comprising:

automatically determining whether the near-eye, mixed reality display device, including a respective display optical system for each eye of the wearer which is positioned to be seen through by the respective eye, is aligned with a wearer IPD in accordance with an alignment criteria based on determining whether each optical axis of the respective display optical system positioned in the display device satisfies the alignment criteria for alignment with a respective pupil of the wearer; and

responsive to the display device not being aligned with the wearer IPD in accordance with the alignment criteria, automatically adjusting a position of one or both of the display optical systems, each display optical system supported by an independently movable support structure, within the display device for bringing the display device into alignment with the wearer IPD;

detecting a change indicating an alignment with the wearer IPD no longer satisfies the alignment criteria; and

automatically re-adjusting the one or both of the display optical systems positioned to be seen through by a respective eye for satisfying the alignment criteria.

15. The method of claim 14 wherein automatically determining whether the near-eye, mixed reality display device, including a respective display optical system for each eye which is positioned to be seen through by the respective eye, is aligned with the wearer IPD in accordance with an alignment criteria based on determining whether each optical axis of a respective display optical system positioned in the display device satisfies alignment criteria for alignment with a respective pupil of the wearer further comprises:

displaying a stereo image of a virtual object in the wearer field of view at a distance and a direction for determining an IPD by projecting a separate image in each display optical system of the display positioned to be seen through by a respective eye;

during an observation period, moving at least one of the separate images in the wearer field of view for at least one of the display optical systems to one or more positions expected to be viewable if each pupil were aligned with its respective optical axis;

capturing data of each eye during the observation period by at least one sensor of an arrangement of gaze detection elements for the respective display optical system;

determining a gaze pattern for each respective eye during the observation period based on the captured data and the arrangement of gaze detection elements for each display optical system; and

determining whether the gaze pattern indicates each optical axis is aligned with its respective pupil in accordance with an alignment criteria.

16. The method of claim 15 wherein responsive to the display device not being aligned with the wearer IPD in accordance with the alignment criteria, automatically adjusting a position of the one or both of the display optical systems within the display device for bringing the display device into alignment with the wearer IPD further comprises:

automatically determine one or more adjustment values for at least one display adjustment mechanism for each display optical system not satisfying the alignment criteria based on the gaze pattern based on a distance vector between each gaze vector of the gaze pattern which failed to intersect the virtual object of the stereo image and the position of the virtual object at the time period of expected intersection.

17. The method of claim 14 wherein detecting a change indicating an alignment with the wearer IPD no longer satisfies the alignment criteria further comprises:

determining a distance of a point of gaze based on gaze data; and

selecting as the current IPD a near IPD value or a far IPD value based on the distance of the point of gaze.

18. The method of claim 14 wherein detecting a change indicating the alignment with the wearer IPD no longer satisfies the alignment criteria further comprises determining whether the distance of the point of gaze has fallen within a distance boundary for the near IPD;

and

automatically re-adjusting the one or both of the display optical systems for satisfying the alignment criteria further comprises re-adjusting the the one or both display optical systems for satisfying the alignment criteria for the near IPD.

19. The method of claim 14 wherein detecting a change indicating the alignment with the wearer IPD no longer satisfies the alignment criteria further comprises:

determining a change in a depth dimension between the the one or both display optical systems and the respective eye based on a vector from a fixed center of eyeball rotation of each eye to the position of at least one gaze detection element of the the one or both display optical systems.