Position and orientation determining method and apparatus and storage medium

• US 6,834,250 B2
• Filed: 03/28/2001
• Issued: 12/21/2004
• Est. Priority Date: 11/30/2000
• Status: Expired due to Fees

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Satoh, Kiyohide, Uchiyama, Shinji
• Primary Examiner(s): Barlow, John
• Assistant Examiner(s): Lau, Tung S
• Application Number:
  US09/818,602
• Publication Number:
  US 20020103617A1
• Time in Patent Office: 1,364 Days
• Field of Search: 702/150, 382/107, 382/103, 382/184, 375/240, 463/1, 345/420, 345/157
• US Class Current: 702/150
• CPC Class Codes: G06T 7/74 involving reference images or patches

Abstract

A viewing transform matrix indicating a position and a orientation of an HMD is generated according to a signal indicating the position and the orientation of the HMD. On the other hand, a landmark and its position are detected based on a captured picture, and a correction matrix ΔMc is generated based on the detected position of the landmark. The position and orientation of the HMD is corrected by using a viewing transform matrix and a correction matrix ΔMc generated in the above mentioned processes, a picture of a virtual object is generated according to an external parameter indicating the corrected position and the orientation of the HMD, and a picture of augmented reality is generated. The generated picture of the augmented reality is displayed on a display unit.

Claims

1. A position and orientation determination apparatus which identifies a parameter indicating a position and orientation of a capture unit adapted to capture a picture in a real space containing a plurality of feature points whose positions are known in a three-dimensional array, comprising:

a position and orientation measurement unit adapted to measure the position and orientation of the capture unit in a method other than using a captured picture;

a detection unit adapted to detect the plurality of feature points and their positions in a two-dimensional array on an image pickup screen using the picture in the real space captured by the capture unit;

a prediction unit adapted to predict the positions of the feature points in the two-dimensional array on the image pickup screen based on the position and orientation of the capture unit measured by said position and orientation measurement unit;

a correction unit adapted to correct a parameter of the measured position and orientation of the capture unit based on an average value or an average weighting value of the positions of the feature points on the image pickup screen of the capture unit obtained by said prediction unit, and based on an average value or an average weighting value of the positions of the feature points obtained by said detection unit; and

an image generation unit adapted to generate an image of a virtual object based on the corrected parameter, and to superimpose the image of the virtual object on the picture in the real space captured by the capture unit.

2. The position and orientation determination apparatus according to claim 1, wherein an artificially applied marker is used as the plurality of feature points whose three-dimensional positions are known.

3. The position and orientation determination apparatus according to claim 1, wherein a point originally existing in the real space as the plurality of feature points whose three-dimensional positions are known, and whose two-dimensional positions can be detected on the image pickup screen by said detection unit.

4. The position and orientation determination apparatus according to claim 1, wherein said position and orientation measurement unit is a sensor for measuring the position and orientation of the capture unit.

5. The position and orientation determination apparatus according to claim 1, wherein said correction unit corrects the parameter such that the capture unit either rotates or translates.

6. The position and orientation determination apparatus according to claim 1, wherein said correction unit corrects the parameter of the capture unit by combining rotation transform with translation transform.

7. The position and orientation determination apparatus according to claim 1, wherein when the parameter of the capture unit is processed plural times alternately by rotation transform and translation transform, said correction unit repeats the process until an average value or an average weighting value of an error between the position of the feature point corrected by said prediction unit and the position of the feature point on the image pickup screen is equal to or less than a predetermined value or until the error cannot be smaller.

8. The position and orientation determination apparatus according to claim 1, wherein when said correction unit corrects the parameter such that the capture unit can rotate, the feature point whose position predicted by said prediction unit is associated with the feature point detected by said detection unit, a rotation axis and a rotation angle with which the position of the feature point on the image pickup screen of the capture unit using the position of the feature point obtained by said prediction unit matches the position of the feature point on the image pickup screen of the capture unit obtained by said detection unit are obtained for each feature point, and the parameter can be corrected by using average values of the rotation axis and the rotation angle obtained for each feature point which is dealt with.

9. The position and orientation determination apparatus according to claim 1, wherein when said correction unit corrects the parameter such that the capture unit can rotate, the feature point whose position predicted by said prediction unit is associated with the feature point detected by said detection unit, a rotation axis and a rotation angle with which the position of the feature point on the image pickup screen of the capture unit using the position of the feature point obtained by said prediction unit matches the position of the feature point on the image pickup screen of the capture unit obtained by said detection unit are obtained for each feature point which is dealt with, and the parameter can be corrected using average weighting values of the rotation axis and the rotation angle obtained for each feature point which is dealt with.

10. The position and orientation determination apparatus according to claim 1, wherein when said correction unit corrects the parameter such that the capture unit can translate, the feature point whose position predicted by said prediction unit is associated with the feature point detected by said detection unit, a difference between the position of the feature point obtained by said prediction unit and the position of the feature point obtained by said detection unit is obtained for each feature point which is dealt with, and the parameter can be corrected using average values of the difference obtained for each feature point which is dealt with.

11. The position and orientation determination apparatus according to claim 1, wherein when said correction unit corrects the parameter such that the capture unit can translate, the feature point whose position predicted by said prediction unit is associated with the feature point detected by said detection unit, a difference between the position of the feature point obtained by said prediction unit and the position of the feature point obtained by said detection unit is obtained for each feature point which is dealt with, and the parameter can be corrected using average weighting values of the difference obtained for each feature point which is dealt with.

12. The position and orientation determination apparatus according to claim 4, wherein said sensor is a magnetic sensor for measuring the three-dimensional position and orientation of the capture unit.

13. The position and orientation determination apparatus according to claim 6, wherein said correction unit corrects the parameter of the capture unit by combining rotation transform with translation transform alternately and plural times.

14. The position and orientation determination apparatus according to claim 6, wherein said correction unit corrects the parameter such that the capture unit can rotate, and then re-predicts the two-dimensional position of the feature point on the image pickup screen based on the position of the feature point in the real space and the position and orientation of the camera after the correction, and the capture unit can translate.

15. The position and orientation determination apparatus according to claim 6, wherein said correction unit corrects the parameter such that the capture unit can translate, and then re-predicts the two-dimensional position of the feature point on the image pickup screen based on the position of the feature point in the real space and the position and orientation of the camera after the correction, and the capture unit can rotate.

16. A position and orientation determination method which identifies a parameter indicating a position and orientation of a capture unit adapted to capture a picture in a real space containing a plurality of feature points whose positions are known in a three-dimensional array, said method comprising:

a position and orientation measurement step of measuring the position and orientation of the capture unit with a method other than using a captured picture;

a detection step of detecting the plurality of feature points and their positions in a two-dimensional array on an image pickup screen using the picture in the real space captured by the capture unit;

a prediction step of predicting the positions of the feature points in the two-dimensional array on the image pickup screen based on the position and orientation of the capture unit measured in said position and orientation measurement step;

a correction step of correcting a parameter of the measured position and orientation of the capture unit based on an average value or an average weighting value of the positions of the feature points on the image pickup screen of the capture unit obtained in said prediction step, and based on an average value or an average weighting value of the positions of the feature points obtained in said detection step; and

an image generation step of generation an image of a virtual object based on the corrected parameter, and superimposing the image of the virtual object on the picture in the real space captured by the capture unit.