Determination of position of a head-mounted device on a user

• US 11,551,376 B2
• Filed: 10/25/2019
• Issued: 01/10/2023
• Est. Priority Date: 10/29/2018
• Status: Active Grant

• Current Assignee: Tobii Technology AB
• Original Assignee: Tobii Technology AB
• Inventors: Zachrisson, Joakim, Rosell, Mikael, Pedreira, Carlos, Ryan, Mark, Johansson, Simon
• Primary Examiner(s): Conner, Sean M
• Application Number:
  US16/663,672
• Publication Number:
  US 20200134867A1
• Time in Patent Office: 1,173 Days
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/0068 having means for controlling the degree of correction, e.g. using phase modulators, movable elements controlling the phase of light using moving or deformable elements G02B26/06
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris

Abstract

There is provided a method and system for determining if a head-mounted device for extended reality (XR) is correctly positioned on a user, and optionally performing a position correction procedure if the head-mounted device is determined to be incorrectly positioned on the user. Embodiments include: performing eye tracking by estimating, based on a first image of a first eye of the user, a position of a pupil in two dimensions; determining whether the estimated position of the pupil of the first eye is within a predetermined allowable area in the first image; and, if the determined position of the pupil of the first eye is inside the predetermined allowable area, concluding that the head-mounted device is correctly positioned on the user; or, if the determined position of the pupil of the first eye is outside the predetermined allowable area, concluding that the head-mounted device is incorrectly positioned on the user.

Claims

1. A method for determining if a head-mounted device for extended reality (XR) is correctly positioned on a user, the method comprising:

performing eye tracking, using processing circuitry, by estimating, based on a first image of a first eye of the user, the position of the pupil of the first eye in two dimensions;

determining, using the processing circuitry, if the head-mounted device is correctly positioned on the user, by;

determining whether the estimated position of the pupil of the first eye is within a predetermined allowable area in the first image; and

(i) if the determined position of the pupil of the first eye is inside the predetermined allowable area of the first image;

concluding that the head-mounted device is correctly positioned on the user, or

(ii) if the determined position of the pupil of the first eye is outside the predetermined allowable area of the first image;

concluding that the head-mounted device is incorrectly positioned on the user

further comprising using the processing circuitry to;

perform eye tracking by estimating, based on a second image of a second eye of the user, the position of the pupil of the second eye in two dimensions, wherein each of the positions of the pupils of the first and second eyes, respectively, are defined according to a common coordinate system;

determining an eye separation distance (d_EYES) as the difference between the position of the pupil of the first eye and the position of the pupil of the second eye,

wherein determining, using the processing circuitry, if the head-mounted device is correctly positioned on the user, further comprises;

if the absolute value of the determined eye separation distance (d_EYES) is above a preset eye separation threshold value (T_EYES);

concluding that the head-mounted device is incorrectly positioned on the user.

2. The method of claim 1, wherein the step of determining, using the processing circuitry, if the head-mounted device is correctly positioned on the user, further comprises:

a) obtaining a gaze angle (β

) for the first eye;

b) comparing the obtained gaze angle (β

) to a preset gaze angle threshold value (T_GA); and

if the obtained gaze angle (β

) is below the preset gaze angle threshold value (T_GA), continuing with the steps of performing eye tracking and determining if the head-mounted device is correctly positioned on the user;

or

if the obtained gaze angle (β

) is not below the preset gaze angle threshold value (T_GA), returning to step a).

3. The method of claim 2, wherein the obtained gaze angle (β

) is defined as the angle between an optical axis of the head mounted device and a gaze direction of the first eye of the user, wherein the optical axis of the head mounted device is defined as a vector passing through a focal point of the first eye and an origin of coordinates in an internal coordinate system of the head mounted device, and wherein the gaze direction of the first eye of the user is defined as a vector passing through the focal point of the first eye and a gaze point of the first eye at the display.

4. The method of claim 1, wherein the step of performing eye tracking, using processing circuitry, further comprises:

obtaining an eye rotation center for the first eye wherein the eye rotation center defines a point where all gaze directions of the first eye converge behind the cornea of the first eye; and

for a given gaze direction, estimating the position of the pupil of the first eye in three dimensions, based on the obtained eye rotation center,

wherein estimating the position of the pupil of the first eye, in two dimensions is further based on the estimated position of the pupil of the first eye in three dimensions.

5. The method of claim 1, further comprising:

estimating a lens cup separation distance (d_{LENS_CUP}) as the difference between the position of the center of a first lens cup of the head-mounted device and the position of the center of a second lens cup of the head-mounted device in three dimensions;

performing eye tracking by estimating the eye separation distance (d_EYES) as the difference between the position of the pupil of the first eye of the user and the position of the pupil of the second eye of the user, in three dimensions, wherein each of the positions of the centers of the first and second lens cups, respectively, and each of the pupils of the first and second eyes of the user, respectively, are defined according to the common coordinate system; and

calculating a difference (d_DIFF) between the lens cup separation distance (d_{LENS_CUP}) and the eye separation distance (d_EYES);

wherein determining, using the processing circuitry, if the head-mounted device is correctly positioned on the user, further comprises;

comparing the calculated difference (d_DIFF) between the lens cup separation distance (d_{LENS_CUP}) and the eye separation distance (d_EYES) to a preset difference threshold value (T_DIFF); and

if the calculated difference (d_DIFF) is above the preset difference threshold value (T_DIFF);

concluding that the head-mounted device is incorrectly positioned on the user.

6. The method of claim 1, further comprising, if the head-mounted device is determined to be incorrectly positioned on the user:

performing a position correction procedure.

7. The method of claim 6, wherein performing the position correction procedure comprises, if a lens cup separation distance (d_{LENS_CUP}) has been determined to be too small or too large, performing motorized position correction of the lens cup separation distance (d_{LENS_CUP}).

8. The method of claim 6, wherein performing the position correction procedure comprises notifying the system or the user about the determined incorrect position of the head-mounted device.

9. The method of claim 6, wherein performing the position correction procedure comprises initiating a positioning guide, which positioning guide provides guidance on steps to be taken for manual correction of the position of the head-mounted device.

10. A system for determining if a head-mounted device for extended reality (XR) is incorrectly positioned on a user, the system comprising processing circuitry configured to:

perform eye tracking by estimating, based on a first image of a first eye of the user, a position of a pupil of the first eye; and

determine if the head-mounted device is correctly positioned on the user, by;

determining whether the estimated position of the pupil of the first eye is within a predetermined allowable area in the first image; and

(i) if the determined position of the pupil of the first eye is inside the predetermined allowable area of the first image;

concluding that the head-mounted device is correctly positioned on the user, or

(ii) if the determined position of the pupil of the first eye is outside the predetermined allowable area of the first image;

concluding that the head-mounted device is incorrectly positioned on the user;

wherein the processing circuitry is further configured to;

perform eye tracking by estimating, based on a second image of a second eye of the user, the position of the pupil of the second eye in two dimensions, wherein each of the positions of the pupils of the first and second eyes, respectively, are defined according to a common coordinate system;

determine an eye separation distance (d_EYES) as the difference between the position of the pupil of the first eye and the position of the pupil of the second eye,

wherein the processing circuitry is configured to determine if the head-mounted device is correctly positioned on the user by;

if the absolute value of the determined eye separation distance (d_EYES) is above a preset eye separation threshold value (T_EYES);

concluding that the head-mounted device is incorrectly positioned on the user.

11. The system of claim 10, wherein the processing circuitry is further configured to:

a) obtain a gaze angle (β

) for the first eye;

b) compare the obtained gaze angle (β

) to a preset gaze angle threshold value (T_GA); and

if the obtained gaze angle (β

) is below the preset gaze angle threshold value (T_GA), continue with the steps of performing eye tracking and determining if the head-mounted device is correctly positioned on the user;

or

if the obtained gaze angle (β

) is not below the preset gaze angle threshold value (T_GA), return to step a).

12. The system of claim 10, wherein the processing circuitry is further configured to:

perform the eye tracking by obtaining an eye rotation center for the first eye, wherein the eye rotation center defines a point where all gaze directions of the first eye converge behind the cornea of the first eye; and

for a given gaze direction, estimating the position of the pupil of the first eye in three dimensions, based on the obtained eye rotation center,

wherein estimating the position of the pupil of the first eye in two dimensions, is further based on the estimated position of the pupil of the first eye in three dimensions.

13. The system of claim 10, wherein the processing circuitry is further configured to:

estimate a lens cup separation distance (d_{LENS_CUP}) as the difference between the position of the center of a first lens cup of the head-mounted device and the position of the center of a second lens cup of the head-mounted device;

perform eye tracking by estimating the eye separation distance (d_EYES) as the difference between the position of the pupil of the first eye of the user and the position of the pupil of the second eye of the user, wherein each of the positions of the centers of the first and second lens cups, respectively, and each of the pupils of the first and second eyes of the user, respectively, are defined according to the common coordinate system;

calculate a difference (d_DIFF) between the lens cup separation distance (d_{LENS_CUP}) and the eye separation distance (d_EYES); and

compare the calculated difference (d_DIFF) between the lens cup separation distance (d_{LENS_CUP}) and the eye separation distance (d_EYES) to a preset difference threshold value (T_DIFF),

wherein the processing circuitry is further configured to determine if the head-mounted device is correctly positioned on the user by;

if the calculated difference (d_DIFF) is above the preset difference threshold value (T_DIFF);

concluding that the head-mounted device is incorrectly positioned on the user.

14. The system of claim 10, wherein the processing circuitry is further configured to perform a position correction procedure if the head-mounted device is determined to be incorrectly positioned on the user.

15. The system of claim 14, wherein the processing circuitry is configured to, in the position correction procedure:

perform motorized position correction of a lens cup separation distance (d_{LENS_CUP}), if the lens cup separation distance (d_{LENS_CUP}) has been determined to be too small or too large;

notify the system or the user about the determined incorrect position of the head-mounted device;

or

initiate a positioning guide, which positioning guide provides guidance on steps to be taken for manual correction of the position of the head-mounted device.

16. The system of claim 10 further comprising a head-mounted device.

17. A non-transitory computer-readable storage medium storing instructions which, when executed by processing circuitry of a system cause the system to:

perform eye tracking, by estimating, based on an image of a first eye of the user, the position of the pupil of the first eye; and

determine if the head-mounted device is correctly positioned on the user, by;

determining whether the estimated position of the pupil of the first eye is within a predetermined allowable area in the first image; and

(i) if the determined position of the pupil of the first eye is inside the predetermined allowable area of the first image;

concluding that the head-mounted device is correctly positioned on the user, or

(ii) if the determined position of the pupil of the first eye is outside the predetermined allowable area of the first image;

concluding that the head-mounted device is incorrectly positioned on the user

wherein the system is further caused to;

perform eye tracking by estimating, based on a second image of a second eye of the user, the position of the pupil of the second eye in two dimensions, wherein each of the positions of the pupils of the first and second eyes, respectively, are defined according to a common coordinate system;

determining an eye separation distance (d_EYES) as the difference between the position of the pupil of the first eye and the position of the pupil of the second eye,

wherein determining, using the processing circuitry, if the head-mounted device is correctly positioned on the user, further comprises;

if the absolute value of the determined eye separation distance (d_EYES) is above a preset eye separation threshold value (T_EYES);

concluding that the head-mounted device is incorrectly positioned on the user.

18. The non-transitory computer-readable storage medium of claim 17, further storing instructions which, when executed by processing circuitry of a system, cause the system to perform a position correction procedure if the head-mounted device is determined to be incorrectly positioned on the user.