Periocular test for mixed reality calibration

• US 10,573,042 B2
• Filed: 09/27/2017
• Issued: 02/25/2020
• Est. Priority Date: 10/05/2016
• Status: Active Grant

• Current Assignee: Magic Leap, Inc.
• Original Assignee: Magic Leap, Inc.
• Inventors: Kaehler, Adrian, Bradski, Gary, Badrinarayanan, Vijay
• Primary Examiner(s): Xiao, Ke
• Assistant Examiner(s): Tran, Kim Thanh T
• Application Number:
  US15/717,747
• Publication Number:
  US 20180096503A1
• Time in Patent Office: 881 Days
• Field of Search: 345633
• CPC Class Codes: G02B 2027/0127 comprising devices increasing the depth of field
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0181 Adaptation to the pilot/driver
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0179 Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 3/20 Linear translation of whole images or parts thereof, e.g. panning
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/166 using acquisition arrangements
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships
• CPC Class Codes: G06V 40/19 Sensors therefor
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction
• CPC Class Codes: G06V 40/197 Matching; Classification

Abstract

A wearable device can include an inward-facing imaging system configured to acquire images of a user'"'"'s periocular region. The wearable device can determine a relative position between the wearable device and the user'"'"'s face based on the images acquired by the inward-facing imaging system. The relative position may be used to determine whether the user is wearing the wearable device, whether the wearable device fits the user, or whether an adjustment to a rendering location of virtual object should be made to compensate for a deviation of the wearable device from its normal resting position.

Claims

1. A method comprising:

under control of a head-mounted device (HMD) comprising a hardware processor, a computer processor, a display system configured to render a virtual object, and an inward-facing imaging system configured to image a periocular region of a user'"'"'s face;

receiving an image from the inward-facing imaging system;

analyzing the image to identify periocular features of the user by an object recognizer;

performing a first calculation to determine a first measurement of the relative position between the HMD and a head of the user based at least partly on the periocular features;

performing a second calculation different from the first calculation to determine a second measurement of the relative position between the HMD and the head of the user based at least partly on the periocular features, wherein performing the first calculation or performing the second calculation comprises;

receiving a first image and a second image from the inward-facing imaging system, wherein the first image and the second image are acquired in sequence;

analyzing the first image and the second image to identify periocular features specific to the user'"'"'s periocular region in the first image and the second image;

determining whether the periocular region is in the first image or the second image based at least partly on the periocular features;

determining the first measurement or the second measurement of the relative position based at least partly on the appearance of the periocular features in the first and the second image;

determining a relative position between the HMD and the head of the user based on the first and second measurements;

determining a degree of fit by comparing the relative position with a threshold condition; and

causing the HMD to perform at least one of the following;

adjusting a display of the HMD to compensate for the relative position, or

providing an indication associated with the degree of fit to the user;

in response to a determination that the periocular region is in neither the first image nor the second image, providing an indication that the HMD has been removed from the head of the user; and

in response to a determination that the periocular region is in either the first image or the second image, indicating a change of a state for the HMD.

2. The method of claim 1, wherein performing the first calculation or performing the second calculation comprises at least one of:

calculating a distance with the HMD to eyes of the user;

calculating an interpupillary distance between the eyes of the user;

determining a relative position between the HMD and the eyes of the user;

determining an asymmetry of the eyes of the user;

or

determining a tilt of the HMD, wherein the tilt comprises at least one of a forward tilt or a side tilt.

3. The method of claim 1, wherein adjusting the display of the HMD to compensate for the relative position comprises:

determining a normal resting position of an eye of the user and a normal rendering position of the virtual object which corresponds to the normal resting position of the eye of the user;

calculating an adjustment to the normal rendering position of the virtual object based at least partly on the relative position between the HMD and the head of the user; and

determining the rendering location of the virtual object based on the adjustment to the normal rendering position.

4. The method of claim 1, wherein determining the degree of fit comprises:

accessing an image of the periocular region of the user;

accessing a mapping between images of the periocular region and the degree of fit of the HMD on the user; and

applying the mapping to determine the degree of fit of the HMD.

5. The method of claim 1, wherein the indication to the user associated with the degree of fit comprises:

providing instructions to the user on improving the degree of fit;

accessing an additional eye image;

applying the mapping to the additional eye image to determine an updated degree of fit; and

accessing additional eye images and determine updated degrees of fit until an updated degree of fit passes a threshold.

6. A head-mounted device (HMD) comprising:

an inward-facing imaging system configured to image a periocular region of a user'"'"'s face, wherein the inward-facing imaging system is configured to acquire at least a first image of the periocular region and is further configured to mask a portion of the periocular region captured in the first image; and

a hardware processor programmed to;

receive an image from the inward-facing imaging system;

analyze the image to identify periocular features of the user by an object recognizer;

perform a first calculation to determine a first measurement of the relative position between the HMD and a head of the user based at least partly on the periocular features;

perform a second calculation different from the first calculation to determine a second measurement of the relative position between the HMD and the head of the user based at least partly on the periocular features, wherein to perform the first calculation or performing the second calculation, the hardware processor is programmed to;

receive a first image and a second image from the inward-facing imaging system, wherein the first image and the second image are acquired in sequence;

analyze the first image and the second image to identify periocular features specific to the user'"'"'s periocular region in the first image and the second image;

determine whether the periocular region is in the first image or the second image based at least partly on the periocular features;

determine the first measurement or the second measurement of the relative position based at least partly on the appearance of the periocular features in the first and the second image;

determine a relative position between the HMD and the head of the user based at least partly on the periocular features based on the first and second measurements; and

determine a degree of fit by comparing the relative position with a threshold condition;

in response to a determination that the periocular region is in neither the first image nor the second image, provide an indication that the HMD has been removed from the head of the user; and

in response to a determination that the periocular region is in either the first image or the second image, indicate a change of a state for the HMD.

7. The HMD of claim 6, wherein to determine the degree of fit, the hardware processor is programmed to:

access an image of the periocular region of the user;

access a mapping between images of the periocular region and the degree of fit of the HMD on the user, wherein the mapping was trained using a machine learning model; and

apply the mapping to determine the degree of fit of the HMD.

8. The HMD of claim 6, wherein the degree of fit comprises at least one of:

fit well, fit adequately, or fit poorly.

9. The HMD of claim 6, wherein the hardware processor is further programmed to adjust a rendering location of a virtual object based on the degree of fit.

10. The HMD of claim 6, wherein the hardware processor is further programmed to provide an indication to the user on the degree of fit at an initialization phase of the HMD.

11. The HMD of claim 10, wherein the hardware processor is further programmed to:

provide instructions to the user on improving the degree of fit;

access an additional eye image;

apply the mapping to the additional eye image to determine an updated degree of fit; and

access additional eye images and determine updated degrees of fit until an updated degree of fit passes a threshold.