Optical see-through display calibration

• US 10,070,120 B2
• Filed: 09/16/2015
• Issued: 09/04/2018
• Est. Priority Date: 09/17/2014
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Pedley, Christopher, Masato, Daniele, Everitt, Andrew John
• Primary Examiner(s): Edwards, Mark
• Application Number:
  US14/856,509
• Publication Number:
  US 20160080732A1
• Time in Patent Office: 1,084 Days
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0189 Sight systems
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 13/327 Calibration thereof
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays

Abstract

Disclosed are a system, apparatus, and method for calibrating an optical see-through display (OSD). The OSD virtual display size is estimated as a starting point for determining an eye to camera distance. In response to determining the eye to camera distance and comparing the result with a measured eye to camera distance an updated display size may be determined. A user'"'"'s head movement may be tracked in order to determine when a visual alignment based calibration routine is complete. When a user is not viewing a calibration target a request is sent to prompt the user to realign the OSD. When a user is still after a period of alignment with the OSD, the alignment and calibration procedure may be assumed as complete. A buffer may receive calibration values from before and after the calibration procedure is complete and the values may be averaged or filtered for accuracy.

Claims

1. A method of calibrating an optical see-through display (OSD), the method comprising:

guiding a user to align a first virtual target on the OSD to a real-world target;

collecting, while guiding the user, a first plurality of real-world target poses, each real-world target pose having a first plurality of point correspondences to the OSD;

receiving an indication of a correct alignment between the first virtual target and the real-world target;

guiding the user to align a second virtual target on the OSD to the real-world target, the second virtual target having a different position than the first virtual target;

collecting, while guiding the user, a second plurality of real-world target poses, each real-world target pose having a second plurality of point correspondences to the OSD;

receiving an indication of a correct alignment between the second virtual target and the real-world target;

iteratively refining an estimated eye position with respect to the OSD according to;

a set of positioning parameters, and

a set of point correspondences to the OSD selected from the first and second plurality of point correspondences to the OSD, wherein the iteratively refining includes a rigid geometric transformation between an image sensor and a position of an eye pupil; and

providing, in response to refining the estimated eye position, an initial OSD calibration profile comprising the estimated eye position.

2. The method of claim 1, wherein the iterative refining occurs until an iteration threshold is met, the iteration threshold comprising one or more of:

an error threshold, an iteration count, or a change in display diagonal value threshold.

3. The method of claim 1, wherein the set of positioning parameters includes a width to height ratio of the OSD, an eye to camera distance along a user'"'"'s eye optical axis, and an initial estimated eye position with respect to the OSD.

4. The method of claim 1, wherein the second virtual target has a different scale and center point on the OSD than the first virtual target.

5. The method of claim 1, further comprising:

determining an eye intrinsic matrix from at least the estimated eye position and the set of positioning parameters from the initial calibration profile; and

iteratively re-determining the eye intrinsic matrix by adjusting one or more of the set of positioning parameters until an iteration threshold is met, wherein the iteration threshold includes one or more of;

an iteration counter, error threshold, or change in error threshold.

6. The method of claim 1, further comprising:

determining an eye gaze vector from one or more of;

an eye tracker or assuming a user is looking at the virtual target;

determining an eye position offset from the eye gaze vector; and

adjusting the estimated eye position with respect to the OSD according to the eye position offset and the eye gaze vector.

7. The method of claim 1, wherein the set of point correspondences to the OSD are selected according to one or more of time or quality thresholds.

8. The method of claim 1, wherein the initial calibration profile is determined from a sample of a plurality of different users.

9. The method of claim 1, wherein a different initial calibration profile is determined for each of a plurality of different types of OSDs.

10. The method of claim 1, further comprising:

collecting point correspondences for one or both of;

a threshold amount of time after the indication of correct alignment, or a threshold amount of time before the indication of correct alignment.

11. A device comprising:

a processor;

a storage device coupled to the processor and configurable for storing instructions, which, when executed by the processor cause the processor to;

guide a user to align a first virtual target on the OSD to a real-world target;

collect, while guiding the user, a first plurality of real-world target poses, each real-world target pose having a first plurality of point correspondences to the OSD;

receive an indication of a correct alignment between the first virtual target and the real-world target;

guide the user to align a second virtual target on the OSD to the real-world target, the second virtual target having a different position than the first virtual target;

collect, while guiding the user, a second plurality of real-world target poses, each real-world target pose having a second plurality of point correspondences to the OSD;

receive an indication of a correct alignment between the second virtual target and the real-world target;

iteratively refine an estimated eye position with respect to the OSD according to;

a set of positioning parameters, and

a set of point correspondences to the OSD selected from the first and second plurality of point correspondences to the OSD, wherein the iteratively refining includes a rigid geometric transformation between an image sensor and a position of an eye pupil; and

provide, in response to refining the estimated eye position, an initial OSD calibration profile comprising the estimated eye position.

12. The device of claim 11, wherein the iterative refining occurs until an iteration threshold is met, the iteration threshold comprising one or more of:

an error threshold, an iteration count, or a change in display diagonal value threshold.

13. The device of claim 11, wherein the set of positioning parameters includes a width to height ratio of the OSD, an eye to camera distance along a user'"'"'s eye optical axis, and an initial estimated eye position with respect to the OSD.

14. The device of claim 11, wherein the second virtual target has a different scale and center point on the OSD than the first virtual target.

15. The device of claim 11, further comprising instructions to:

determine an eye intrinsic matrix from at least the estimated eye position and the set of positioning parameters from the initial calibration profile; and

iteratively re-determine the eye intrinsic matrix by adjusting one or more of the set of positioning parameters until a second iteration threshold is met, wherein the second iteration threshold includes one or more of;

an iteration counter, error threshold, or change in error threshold.

16. The device of claim 11, further comprising:

determine an eye gaze vector from one or more of;

an eye tracker or assuming a user is looking at the virtual target;

determine an eye position offset from the eye gaze vector; and

adjust the estimated eye position with respect to the OSD according to the eye position offset and the eye gaze vector.

17. The device of claim 11, wherein the set of point correspondences to the OSD are selected according to one or more of time or quality thresholds.

18. The device of claim 11, wherein the initial calibration profile is determined from a sample of a plurality of different users.

19. The device of claim 11, wherein a different initial calibration profile is determined for each of a plurality of different types of OSDs.

20. The device of claim 11, further configured to:

collect point correspondences for one or both of;

a threshold amount of time after the indication of correct alignment, or a threshold amount of time before the indication of correct alignment.

21. A machine readable non-transitory storage medium having stored therein program instructions that are executable by a processor to:

guide a user to align a first virtual target on the OSD to a real-world target;

collect, while guiding the user, a first plurality of real-world target poses, each real-world target pose having a first plurality of point correspondences to the OSD;

receive an indication of a correct alignment between the first virtual target and the real-world target;

guide the user to align a second virtual target on the OSD to the real-world target, the second virtual target having a different position than the first virtual target;

collect, while guiding the user, a second plurality of real-world target poses, each real-world target pose having a second plurality of point correspondences to the OSD;

receive an indication of a correct alignment between the second virtual target and the real-world target;

iteratively refine an estimated eye position with respect to the OSD according to;

a set of positioning parameters, and

a set of point correspondences to the OSD selected from the first and second plurality of point correspondences to the OSD, wherein the iteratively refining includes a rigid geometric transformation between an image sensor and a position of an eye pupil; and

provide, in response to refining the estimated eye position, an initial OSD calibration profile comprising the estimated eye position.

22. The medium of claim 21, wherein the iterative refinement is executed until an iteration threshold is met, the iteration threshold comprising one or more of:

an error threshold, an iteration count, or a change in display diagonal value threshold.

23. The medium of claim 21, wherein the set of positioning parameters includes a width to height ratio of the OSD, an eye to camera distance along a user'"'"'s eye optical axis, and an initial estimated eye position with respect to the OSD.

24. The medium of claim 21, further comprising instructions to:

determine an eye intrinsic matrix from at least the estimated eye position and the set of positioning parameters from the initial calibration profile; and

iteratively re-determine the eye intrinsic matrix by adjusting one or more of the set of positioning parameters until a second iteration threshold is met, wherein the second iteration threshold includes one or more of;

an iteration counter, error threshold, or change in error threshold.

25. The medium of claim 21, wherein the set of point correspondences to the OSD are selected according to one or more of time or quality thresholds.

26. An apparatus for calibrating an optical see-through display, the apparatus comprising:

means for guiding a user to align a first virtual target on the OSD to a real-world target;

means for collecting, while guiding the user, a first plurality of real-world target poses, each real-world target pose having a first plurality of point correspondences to the OSD;

means for receiving an indication of a correct alignment between the first virtual target and the real-world target;

means for guiding the user to align a second virtual target on the OSD to the real-world target, the second virtual target having a different position than the first virtual target;

means for collecting, while guiding the user, a second plurality of real-world target poses, each real-world target pose having a second plurality of point correspondences to the OSD;

means for receiving an indication of a correct alignment between the second virtual target and the real-world target;

means for iteratively refining an estimated eye position with respect to the OSD according to;

a set of positioning parameters, and

a set of point correspondences to the OSD selected from the first and second plurality of point correspondences to the OSD, wherein the means for iteratively refining includes means for utilizing a rigid geometric transformation between an image sensor and a position of an eye pupil; and

means for providing, in response to refining the estimated eye position, an initial OSD calibration profile comprising the estimated eye position.

27. The apparatus of claim 26, wherein the set of point correspondences to the OSD are selected according to one or more of time or quality thresholds.

28. The apparatus of claim 26, wherein the iterative refining occurs until an iteration threshold is met, the iteration threshold comprising one or more of:

an error threshold, an iteration count, or a change in display diagonal value threshold.

29. The apparatus of claim 26, wherein the set of positioning parameters includes a width to height ratio of the OSD, an eye to camera distance along a user'"'"'s eye optical axis, and an initial estimated eye position with respect to the OSD, and wherein the second virtual target has a different scale and center point on the OSD than the first virtual target.

30. The apparatus of claim 26, further comprising:

means for determining an eye intrinsic matrix from at least the estimated eye position and the set of positioning parameters from the initial calibration profile; and

means for iteratively re-determining the eye intrinsic matrix by adjusting one or more of the set of positioning parameters until an iteration threshold is met, wherein the iteration threshold includes one or more of;

an iteration counter, error threshold, or change in error threshold.