Position and orientation determining method and apparatus and storage medium

• US 20020103617A1
• Filed: 03/28/2001
• Published: 08/01/2002
• Est. Priority Date: 11/30/2000
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Satoh, Kiyohide, Uchiyama, Shinji
• Granted Patent: US 6,834,250 B2
• Time in Patent Office: Days
• US Class Current: 702/150
• CPC Class Codes: G06T 7/74 involving reference images or patches

Abstract

A viewing transform matrix indicating a position and a orientation of an HMD is generated according to a signal indicating the position and the orientation of the HMD. On the other hand, a landmark and its position are detected based on a captured picture, and a correction matrix ΔMc is generated based on the detected position of the landmark. The position and orientation of the HMD is corrected by using a viewing transform matrix and a correction matrix ΔMc generated in the above mentioned processes, a picture of a virtual object is generated according to an external parameter indicating the corrected position and the orientation of the HMD, and a picture of augmented reality is generated. The generated picture of the augmented reality is displayed on a display unit.

Claims

1. A position and orientation determination apparatus which identifies a parameter indicating a position and orientation of capture means for capturing a picture in real a space containing a plurality of feature points whose positions are known in a three-dimensional array, comprising:

position and orientation measurement means for measuring the position and orientation of the capture means in a method other than using a captured picture;

detection means for detecting the plurality of feature points and their positions in a two-dimensional array on an image pickup screen using the picture in the real space captured by said capture means;

prediction means for predicting the positions of the feature points in the two-dimensional array on the image pickup screen based on the position and orientation of said capture means measured by said position and orientation measurement means; and

correction means for correcting the parameter indicating the position and orientation of said capture means based on the positions of the feature points on the image pickup screen of said capture means obtained by said prediction means, and based on the positions of the feature points obtained by said detection means,

wherein said parameter indicating the position and orientation of said capture means for capturing the real space is identified by said correction means correcting the parameter.

2. The position and orientation determination apparatus according to claim 1, wherein an artificially applied marker is used as the plurality of feature points whose three-dimensional positions are known.

3. The position and orientation determination apparatus according to claim 1, wherein a point originally existing in the real space as the plurality of feature points whose three-dimensional positions are known, and whose two-dimensional positions can be detected on the image pickup screen by said detection means.

4. The position and orientation determination apparatus according to claim 1, wherein on the picture in the real space captured by said capture means, another image is superimposed and displayed.

5. The position and orientation determination apparatus according to claim 4, wherein said another image is a drawn image of a virtual object.

6. The position and orientation determination apparatus according to claim 1, wherein said position and orientation measurement means is a sensor for measuring the position and orientation of said capture means.

7. The position and orientation determination apparatus according to claim 6, wherein said sensor is a magnetic sensor for measuring the three-dimensional position and orientation of said capture means.

8. The position and orientation determination apparatus according to claim 1, wherein said correction means corrects the parameter such that said capture means either rotates or translates.

9. The position and orientation determination apparatus according to claim 1, wherein said correction means corrects the parameter of said capture means by combining rotation transform with translation transform.

10. The position and orientation determination apparatus according to claim 9, wherein said correction means corrects the parameter of said capture means by combining rotation transform with translation transform alternately and plural times.

11. The position and orientation determination apparatus according to claim 9, wherein said correction means corrects the parameter such that said capture means can rotate, and then re-predicts the two-dimensional position of the feature point on the image pickup screen based on the position of the feature point in the real space and the position and orientation of the camera after the correction, and said capture means can translate.

12. The position and orientation determination apparatus according to claim 9, wherein said correction means corrects the parameter such that said capture means can translate, and then re-predicts the two-dimensional position of the feature point on the image pickup screen based on the position of the feature point in the real space and the position and orientation of the camera after the correction, and said capture means can rotate.

13. The position and orientation determination apparatus according to claim 1, wherein said correction means computes a first average value on the image pickup screen of said capture means using the position of the feature point obtained by said prediction means, and a second average value on the image pickup screen of said capture means of the feature point detected by said detection means, and corrects the parameter such that the first average value matches the second average value.

14. The position and orientation determination apparatus according to claim 1, wherein said correction means computes a first average weighting value on the image pickup screen of said capture means using the position of the feature point obtained by said prediction means, and a second average weighting value on the image pickup screen of said capture means of the feature point detected by said detection means, and corrects the parameter such that the first average weighting value matches the second average weighting value.

15. The position and orientation determination apparatus according to claim 13, wherein when said parameter of said capture means is processed plural times alternately by rotation transform and translation transform, said correction means repeats the process until an average value or a average weighting value of an error between the position of the feature point corrected by said prediction means and the position of the feature point on the image pickup screen can be considerably small or until the error cannot be smaller.

16. The position and orientation determination apparatus according to claim 14, wherein when said parameter of said capture means is processed plural times alternately by rotation transform and translation transform, said correction means repeats the process until an average value or a average weighting value of an error between the position of the feature point corrected by said prediction means and the position of the feature point on the image pickup screen can be considerably small or until the error cannot be smaller.

17. The position and orientation determination apparatus according to claim 13, wherein when said correction means corrects the parameter such that said capture means can rotate, the feature point whose position predicted by said prediction means is associated with the feature point detected by said detection means, a rotation axis and a rotation angle with which the position of the feature point on the image pickup screen of said capture means using the position of the feature point obtained by said prediction means matches the position of the feature point on the image pickup screen of said capture means obtained by said detection means are obtained for each feature point, and said parameter can be corrected by using average values of the rotation axis and the rotation angle obtained for each feature point which is dealt with.

18. The position and orientation determination apparatus according to claim 14, wherein when said correction means corrects the parameter such that said capture means can rotate, the feature point whose position predicted by said prediction means is associated with the feature point detected by said detection means, a rotation axis and a rotation angle with which the position of the feature point on the image pickup screen of the capture means using the position of the feature point obtained by said prediction means matches the position of the feature point on the image pickup screen of said capture means obtained by said detection means are obtained for each feature point which is dealt with, and said parameter can be corrected using average weighting values of the rotation axis and the rotation angle obtained for each feature point which is dealt with.

19. The position and orientation determination apparatus according to claim 13, wherein when said correction means corrects the parameter such that said capture means can translate, the feature point whose position predicted by said prediction means is associated with the feature point detected by said detection means, a difference between the position of the feature point obtained by said prediction means and the position of the feature point obtained by said detection means is obtained for each feature point which is dealt with, and said parameter can be corrected using average values of the difference obtained for each feature point which is dealt with.

20. The position and orientation determination apparatus according to claim 14, wherein when said correction means corrects the parameter such that said capture means can translate, the feature point whose position predicted by said prediction means is associated with the feature point detected by said detection means, a difference between the position of the feature point obtained by said prediction means and the position of the feature point obtained by said detection means is obtained for each feature point which is dealt with, and said parameter can be corrected using average weighting values of the difference obtained for each feature point which is dealt with.

21. A position and orientation determining method which identifies a parameter indicating a position and orientation of capture means for capturing an picture in real space containing a plurality of feature points whose positions are known in a three-dimensional array, comprising:

a position and orientation measuring step of measuring the position and orientation of the capture means in a method other than using a captured picture;

a detecting step of detecting the plurality of feature points and their positions in a two-dimensional array on an image pickup screen using the picture in the real space captured by said capture means;

a predicting step of predicting the positions of the feature points in the two-dimensional array on the image pickup screen based on the position and orientation of said capture means measured in said position and orientation measuring step; and

a correcting step of correcting the parameter indicating the position and orientation of said capture means based on the positions of the feature points on the image pickup screen of said capture means obtained in said predicting step, and based on the positions of the feature points obtained in said detecting step,

wherein said parameter indicating the position and orientation of said capture means for capturing the real space is identified by correcting the parameter in said correcting step.

22. The program code which performs the position and orientation determining method according to claim 21.

23. The storage medium storing the program code according to claim 22.