CALIBRATION OF AUGMENTED REALITY (AR) OPTICAL SEE-THROUGH DISPLAY USING SHAPE-BASED ALIGNMENT

• US 20150193980A1
• Filed: 03/25/2014
• Published: 07/09/2015
• Est. Priority Date: 01/06/2014
• Status: Abandoned Application

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: MITTAL, Arpit, MACIOCCI, Giuliano, PEDLEY, Christopher, WARD, Jonathan David
• Application Number:
  US14/225,042
• Publication Number:
  US 20150193980A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/20101 Interactive definition of point of interest, landmark or seed
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: H04N 13/327 Calibration thereof
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays

Abstract

Two-dimensional or three-dimensional augmented reality (AR) markers are provided for alignment with a target object in calibrating an optical see-through display, such as a head-mounted display (HMD), in an AR system. A calibrated projection matrix for calibration of the optical see-through display is computed based upon a user'"'"'s repeated alignments of the AR markers with the target object.

Claims

1. A method of calibrating an optical see-through display, comprising the steps of:

(a) repeating the steps of;

(i) receiving an input from a user indicating that the user has aligned an augmented reality (AR) marker with an object on the optical see-through display; and

(ii) obtaining a pose matrix based upon the user'"'"'s alignment of the AR marker with the object; and

(b) computing a calibrated projection matrix for calibration of the optical see-through display based upon a plurality of pose matrices.

2. The method of claim 1, wherein the AR marker comprises a two-dimensional marker.

3. The method of claim 2, wherein the two-dimensional marker comprises a rectangular marker.

4. The method of claim 1, wherein the AR marker is formed by at least a portion of a predefined three-dimensional marker.

5. The method of claim 4, wherein the three-dimensional marker comprises a truncated rectangular pyramid having at least five two-dimensional surfaces forming at least five separate AR markers.

6. The method of claim 5, wherein steps (a)(i) through (a)(ii) are repeated at least five times using said at least five separate AR markers.

7. The method of claim 1, wherein step (b) comprises computing screen coordinate matrices by multiplying a projection matrix, a model view matrix and vertices of the object.

8. The method of claim 7, wherein step (b) further comprises computing a concatenated screen coordinate matrix by concatenating said screen coordinate matrices.

9. The method of claim 8, wherein step (b) further comprises:

multiplying each of the pose matrices with the vertices of the object; and

concatenating products of the pose matrices with the vertices of the object to generate a concatenated product matrix.

10. The method of claim 9, wherein the calibrated projection matrix is computed by multiplying the concatenated screen coordinate matrix with an inverse of the concatenated product matrix.

11. An apparatus configured to perform operations to calibrate an optical see-through display, the apparatus comprising:

a memory; and

a processor for executing a set of instructions stored in the memory, the set of instructions for;

(a) repeating n times, n being more than one, and each loop in which steps are repeated is an ith loop, i being less than or equal to n, the steps of;

(i) receiving an input from a user indicating that the user has aligned an augmented reality (AR) marker with an object on the optical see-through display; and

(ii) obtaining an ith pose matrix based upon the user'"'"'s alignment of the AR marker with the object; and

(b) computing a calibrated projection matrix for calibration of the optical see-through display based upon n pose matrices.

12. The apparatus of claim 11, wherein the AR marker is formed by at least a portion of a predefined three-dimensional marker.

13. The apparatus of claim 12, wherein the three-dimensional marker comprises a truncated rectangular pyramid having at least five two-dimensional surfaces forming at least five separate AR markers.

14. The apparatus of claim 11, wherein step (b) comprises computing screen coordinate matrices by multiplying a projection matrix, a model view matrix and vertices of the object.

15. The apparatus of claim 14, wherein step (b) further comprises computing a concatenated screen coordinate matrix by concatenating said screen coordinate matrices.

16. The apparatus of claim 15, wherein step (b) further comprises:

multiplying each of the pose matrices with the vertices of the object; and

concatenating products of the pose matrices with the vertices of the object to generate a concatenated product matrix.

17. The apparatus of claim 16, wherein the calibrated projection matrix is computed by multiplying the concatenated screen coordinate matrix with an inverse of the concatenated product matrix.

18. An apparatus configured to perform operations to calibrate an optical see-through display, the apparatus comprising:

(a) means for repeating the steps of;

(i) receiving an input from a user indicating that the user has aligned an augmented reality (AR) marker with an object on the optical see-through display; and

(ii) obtaining a pose matrix based upon the user'"'"'s alignment of the AR marker with the object; and

(b) means for computing a calibrated projection matrix for calibration of the optical see-through display based upon a plurality of said pose matrices.

19. The apparatus of claim 18, wherein the AR marker is formed by at least a portion of a predefined three-dimensional marker.

20. The apparatus of claim 19, wherein the three-dimensional marker comprises a truncated rectangular pyramid having at least five two-dimensional surfaces forming at least five separate AR markers.

21. The apparatus of claim 18, wherein the means for computing the calibrated projection matrix comprises means for computing screen coordinate matrices by multiplying a projection matrix, a model view matrix and vertices of the object.

22. The apparatus of claim 21, wherein the means for computing the calibrated projection matrix further comprises means for computing a concatenated screen coordinate matrix by concatenating said screen coordinate matrices.

23. The apparatus of claim 22, wherein the means for computing the calibrated projection matrix further comprises:

means for multiplying each of the pose matrices with the vertices of the object; and

means for concatenating products of the pose matrices with the vertices of the object to generate a concatenated product matrix.

24. The apparatus of claim 23, wherein the calibrated projection matrix is computed by multiplying the concatenated screen coordinate matrix with an inverse of the concatenated product matrix.

25. A machine-readable storage medium encoded with instructions executable to perform operations to calibrate an optical see-through display, the operations comprising:

(a) repeating the steps of;

(i) receiving an input from a user indicating that the user has aligned an augmented reality (AR) marker with an object on the optical see-through display; and

(ii) obtaining a pose matrix based upon the user'"'"'s alignment of the AR marker with the object; and

(b) computing a calibrated projection matrix for calibration of the optical see-through display based upon a plurality of said pose matrices.

26. The machine-readable storage medium of claim 25, wherein the AR marker is formed by at least a portion of a predefined three-dimensional marker.

27. The machine-readable storage medium of claim 25, wherein step (b) comprises computing screen coordinate matrices by multiplying a projection matrix, a model view matrix and vertices of the object.

28. The machine-readable storage medium of claim 25, wherein step (b) further comprises computing a concatenated screen coordinate matrix by concatenating said screen coordinate matrices.

29. The machine-readable storage medium of claim 28, wherein step (b) further comprises:

multiplying each of the pose matrices with the vertices of the object; and

concatenating products of the pose matrices with the vertices of the object to generate a concatenated product matrix.

30. The machine-readable storage medium of claim 29, wherein the calibrated projection matrix is computed by multiplying the concatenated screen coordinate matrix with an inverse of the concatenated product matrix.