System and method for calibrating a stereo optical see-through head-mounted display system for augmented reality

• US 20020113756A1
• Filed: 09/25/2001
• Published: 08/22/2002
• Est. Priority Date: 09/25/2000
• Status: Active Grant

• Current Assignee: Siemens Medical Solutions USA Incorporated (Siemens AG)
• Original Assignee: Siemens Corporate Research Incorporated (Siemens AG)
• Inventors: Genc, Yakup, Navab, Nassir, Tuceryan, Mihran
• Granted Patent: US 6,753,828 B2
• Time in Patent Office: Days
• US Class Current: 345/8
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/286 having separate monoscopic and stereoscopic modes
• CPC Class Codes: H04N 13/327 Calibration thereof
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/373 for tracking forward-backward translational head movements, i.e. longitudinal movements
• CPC Class Codes: H04N 13/376 for tracking left-right translational head movements, i.e. lateral movements
• CPC Class Codes: H04N 13/38 for tracking vertical translational head movements

Abstract

A system and method for calibrating a stereo optical see-through HMD (head-mounted display). A preferred method integrates measurement for an optical see-through HMD and a six degrees of freedom tracker that is fixedly attached to the HMD to perform calibration. Calibration is based on the alignment of a stereoscopically fused marker image, which is perceived in depth, with a single 3D reference point in a world coordinate system from various viewpoints. The user interaction to perform the calibration is extremely easy compared to conventional methods and does not require keeping the head static during the calibration process. In one aspect, a method for calibrating a stereo HMD comprises the steps of displaying a 2-dimensional marker image on each eye display of the stereo HMD for view by a user, wherein the 2-dimensional marker images are displayed with an offset so as to induce a 3-dimensional marker that is perceived by the user as being at a distance away from the user; aligning the 3-dimensional marker image with a preselected reference point; collecting calibration data associated with the alignment; and computing a model of the HMD using the collected calibration data.

Claims

1. A method for calibrating a stereo see-through HMD (head-mounted display) for augmented reality, the method comprising the steps of:

displaying a 2-dimensional marker image on each eye display of a stereo HMD for view by a user, wherein the 2-dimensional marker images are displayed with an offset so as to induce a 3-dimensional marker that is perceived by the user as being at a distance away from the user;

aligning the 3-dimensional marker image with a preselected reference point;

collecting calibration data associated with the alignment; and

computing a model of the HMD using the collected calibration data.

2. The method of claim 1, wherein the distance at which the 3-dimensional marker is perceived is proportional to the offset between the 2-dimensional marker images for both eyes.

3. The method of claim 1, wherein the model is defined with respect to a coordinate system of a tracker sensor fixedly attached to the HMD.

4. The method of claim 3, wherein the model comprises a first projection matrix that defines a transformation between the HMD and coordinates of the tracker sensor for the left eye of the user and a second projection matrix that defines a transformation between the HMD and coordinates of the tracker sensor for the right eye of the user.

5. The method of claim 3, wherein the calibration data for a given alignment comprises 2-dimensional image coordinates of the image marker for each eye display of the stereo HMD and 3-dimensional coordinates of the preselected reference point in the coordinate system of the tracker sensor.

6. The method of claim 1, wherein the preselected reference point is located on a tracker transmitter.

7. The method of claim 1, wherein the 2-dimensional image marker comprises a crosshair.

8. The method of claim 1, wherein the 2-dimensional image marker comprises a solid disk.

9. The method of claim 1, wherein the 2-dimensional image marker comprises a rhombus with a cross inside the rhombus.

10. The method of claim 1, wherein the reference point comprises a center of a circular hole in a planar surface.

11. The method of claim 1, wherein the steps of aligning and collecting are performed for each of a plurality of different locations at the same approximate distance from the preselected reference point.

12. The method of claim 11, wherein calibration data is collected for at least six different locations.

14. The program storage device of claim 13, wherein the distance at which the 3-dimensional marker is perceived is proportional to the offset between the 2-dimensional marker images for both eyes.

15. The program storage device of claim 13, wherein the model is defined with respect to a coordinate system of a tracker sensor fixedly attached to the HMD.

16. The program storage device of claim 15, wherein the model comprises a first projection matrix that defines a transformation between the HMD and coordinates of the tracker sensor for the left eye of the user and a second projection matrix that defines a transformation between the HMD and coordinates of the tracker sensor for the right eye of the user.

17. The program storage device of claim 15, wherein the calibration data for a given alignment comprises 2-dimensional image coordinates of the image marker for each eye display of the stereo HMD and 3-dimensional coordinates of the preselected reference point in the coordinate system of the tracker sensor.

18. The program storage device of claim 13, wherein the preselected reference point is located on a tracker transmitter.

19. The program storage device of claim 13, wherein the 2-dimensional image marker comprises a crosshair.

20. The program storage device of claim 13, wherein the 2-dimensional image marker comprises a solid disk.

21. The program storage device of claim 13, wherein the 2-dimensional image marker comprises a rhombus with a cross inside the rhombus.

22. The program storage device of claim 13, wherein the reference point comprises a center of a circular hole in a planar surface.

24. The system of claim 23, wherein the computer readable program code comprises program code for:

displaying a 2-dimensional marker image on each eye display of the stereo HMD for view by a user, wherein the 2-dimensional marker images are displayed with an offset so as to induce a 3-dimensional marker that is perceived by the user as being at a distance away from the user;

receiving as input, calibration data associated with an alignment of the 3-dimensional marker image with a preselected reference point; and

computing a model of the HMD using the collected calibration data.

25. The system of claim 24, wherein the calibration data for a given alignment comprises 2-dimensional image coordinates of the image marker for each eye display of the stereo HMD and 3-dimensional coordinates of the preselected reference point in the coordinate system of the tracker sensor.

13. A program storage device readable by a machine, tangibly embodying a program of instructions executable by the machine to perform method steps for calibrating a stereo see-through HMD (head-mounted display) for augmented reality, the method steps comprising:

displaying a 2-dimensional marker image on each eye display of a stereo HMD for view by a user, wherein the 2-dimensional marker images are displayed with an offset so as to induce a 3-dimensional marker that is perceived by the user as being at a distance away from the user;

receiving as input, calibration data associated with an alignment of the 3-dimensional marker image with a preselected reference point; and

computing a model of the HMD using the collected calibration data.

23. An augmented reality system, comprising:

a stereo see-through HMD (head-mounted display) comprising a first and second eye display;

a tracker system comprising a transmitter and at least one sensor, wherein the sensor is fixedly attached to the HMD; and

a workstation, operatively connected to the tracker system and HMD, comprising computer readable program code embodied therein for calibrating the HMD, wherein the computer readable program code comprises program code for simultaneously collecting calibration data for a first virtual camera and second virtual camera to compute a model that defines a relation between the first virtual camera and the sensor and a relation between the second virtual camera and the sensor, wherein the first virtual camera comprises a combination of the left eye of the individual and a left eye display of the HMD and wherein the second virtual camera comprises a combination of the right eye of the individual and a right eye display of the HMD.

26. A method for calibrating a stereo see-through HMD (head-mounted display) for augmented reality, the method comprising the steps of:

collecting calibration data by aligning a 3-dimensional maker image, which is perceived by a user through a first and second eye display of the HMD, to a fixed location, which is viewed by an individual through the HMD, for each of a plurality of different viewpoints within a same approximate distance away from the fixed location; and

computing a model using the calibration data, wherein the model defines a relation between a coordinate system of a virtual camera and a coordinate system of a tracker sensor fixedly attached to the HMD, wherein the virtual camera comprises a combination of the first and second eye displays of the HMD and both eyes of an individual.