AUTOMATIC CALIBRATION OF SCENE CAMERA FOR OPTICAL SEE-THROUGH HEAD MOUNTED DISPLAY

• US 20150049201A1
• Filed: 01/09/2014
• Published: 02/19/2015
• Est. Priority Date: 08/19/2013
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: LIU, Kexi, RAHMAN, Md Sazzadur, RENSCHLER, Martin H.
• Granted Patent: US 9,264,702 B2
• Time in Patent Office: Days
• US Class Current: 348/189
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/00 Details not covered by groups G06F3/00 - G06F13/00 and G06F21/00 architectures of general purpose stored program computers G06F15/76
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 2207/30208 Marker matrix
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04L 12/1827 Network arrangements for conference optimisation or adaptation
• CPC Class Codes: H04N 17/00 Diagnosis, testing or measuring for television systems or their details
• CPC Class Codes: H04N 5/7491 of head mounted projectors

Abstract

An apparatus for calibrating an augmented reality (AR) device having an optical see-through head mounted display (HMD) obtains eye coordinates in an eye coordinate system corresponding to a location of an eye of a user of the AR device, and obtains object coordinates in a world coordinate system corresponding to a location of a real-world object in the field of view of the AR device, as captured by a scene camera having a scene camera coordinate system. The apparatus calculates screen coordinates in a screen coordinate system corresponding to a display point on the HMD, where the calculating is based on the obtained eye coordinates and the obtained object coordinates. The apparatus calculates calibration data based on the screen coordinates, the object coordinates and a transformation from the target coordinate system to the scene camera coordinate system. The apparatus then derives subsequent screen coordinates for the display of AR in relation to other real-world object points based on the calibration data.

Claims

1. A method of calibrating an augmented reality (AR) device having an optical see-through head mounted display (HMD), said method comprising:

obtaining eye coordinates in an eye coordinate system corresponding to a location of an eye of a user of the AR device;

obtaining object coordinates in a world coordinate system corresponding to a location of a real-world object in the field of view of the AR device, as captured by a scene camera having a scene camera coordinate system;

calculating screen coordinates in a screen coordinate system corresponding to a display point on the HMD, the calculating being based on the eye coordinates and the object coordinates; and

calculating calibration data based on the screen coordinates, the object coordinates and a transformation from the target coordinate system to the scene camera coordinate system.

2. The method of claim 1, further comprising deriving subsequent screen coordinates for the display of AR in relation to other real-world object points based on the calibration data.

3. The method of claim 1, wherein the calibration data corresponds to a projection matrix (P).

4. The method of claim 1, wherein the transformation from the target coordinate system to the scene camera coordinate system is provided by a model view matrix (M).

5. The method of claim 1, wherein calculating screen coordinates comprises:

defining a ray that originates from the eye coordinates, intersects the HMD and terminates at object coordinates; and

calculating an intersection point of the ray and the HMD, the intersection point being the display point.

6. An apparatus for calibrating an augmented reality (AR) device having an optical see-through head mounted display (HMD), said apparatus comprising:

means for obtaining eye coordinates in an eye coordinate system corresponding to a location of an eye of a user of the AR device;

means for obtaining object coordinates in a world coordinate system corresponding to a location of a real-world object in the field of view of the AR device, as captured by a scene camera having a scene camera coordinate system;

means for calculating screen coordinates in a screen coordinate system corresponding to a display point on the HMD, the calculating being based on the eye coordinates and the object coordinates; and

means for calculating calibration data based on the screen coordinates, the object coordinates and a transformation from the target coordinate system to the scene camera coordinate system.

7. The apparatus of claim 6, further comprising means for deriving subsequent screen coordinates for the display of AR in relation to other real-world object points based on the calibration data.

8. The apparatus of claim 6, wherein the calibration data corresponds to a projection matrix (P).

9. The apparatus of claim 6, wherein the transformation from the target coordinate system to the scene camera coordinate system is provided by a model view matrix (M).

10. The apparatus of claim 6, wherein the means for calculating screen coordinates is configured to:

define a ray that originates from the eye coordinates, intersects the HMD and terminates at object coordinates; and

calculate an intersection point of the ray and the HMD, the intersection point being the display point.

11. An apparatus for calibrating an augmented reality (AR) device having an optical see-through head mounted display (HMD), said apparatus comprising:

a memory; and

at least one processing module coupled to the memory and configured to;

obtain eye coordinates in an eye coordinate system corresponding to a location of an eye of a user of the AR device;

obtain object coordinates in a world coordinate system corresponding to a location of a real-world object in the field of view of the AR device, as captured by a scene camera having a scene camera coordinate system;

calculate screen coordinates in a screen coordinate system corresponding to a display point on the HMD, the calculating being based on the eye coordinates and the object coordinates; and

calculate calibration data based on the screen coordinates, the object coordinates and a transformation from the target coordinate system to the scene camera coordinate system.

12. The apparatus of claim 11, the at least one processor further configured to derive subsequent screen coordinates for the display of AR in relation to other real-world object points based on the calibration data.

13. The apparatus of claim 11, wherein the calibration data corresponds to a projection matrix (P).

14. The apparatus of claim 11, wherein the transformation from the target coordinate system to the scene camera coordinate system is provided by a model view matrix (M).

15. The apparatus of claim 11, wherein the at least one processor calculates screen coordinates by being further configured to:

define a ray that originates from the eye coordinates, intersects the HMD and terminates at object coordinates; and

calculate an intersection point of the ray and the HMD, the intersection point being the display point.

16. A computer program product for calibrating an augmented reality (AR) device having an optical see-through head mounted display (HMD), said product comprising:

a computer-readable medium comprising code for;

obtaining eye coordinates in an eye coordinate system corresponding to a location of an eye of a user of the AR device;

obtaining object coordinates in a world coordinate system corresponding to a location of a real-world object in the field of view of the AR device, as captured by a scene camera having a scene camera coordinate system;

calculating screen coordinates in a screen coordinate system corresponding to a display point on the HMD, the calculating being based on the eye coordinates and the object coordinates; and

calculating calibration data based on the screen coordinates, the object coordinates and a transformation from the target coordinate system to the scene camera coordinate system.