Image processing device and method therefor and program codes, storing medium

• US 20040090444A1
• Filed: 08/21/2003
• Published: 05/13/2004
• Est. Priority Date: 03/06/2001
• Status: Active Grant

• Current Assignee: Canon Ayutthaya Limited (Canon Inc.)
• Original Assignee: Canon Ayutthaya Limited (Canon Inc.)
• Inventors: Satoh, Kiyohide
• Granted Patent: US 7,193,636 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G06T 7/74 involving reference images or patches

Abstract

Firstly, a template image is created (S501). Then, an image I^t is photographed (S502). And the sensor output is acquired (S503). A model view matrix M^t is calculated on the basis of the sensor output (S504). And a correction matrix ΔM^t is calculated (S505). And the model view matrix M^t is corrected employing the calculated correction matrix ΔM^t (S506). Then CG is drawn and displayed using the corrected model view matrix (S507).

Claims

1. An image processing device comprising:

an image pick-up device having the fixed positional relation with a measurement object;

an attitude sensor for measuring the attitude at an image pick-up visual point of said image pick-up device;

storage means for storing the calculation information to calculate the attitude and/or position of said measurement object on the basis of an output from said attitude sensor;

target image setting means for setting a target image that is an object for detecting a predetermined index on the basis of a picked-up image picked up by said image pick-up device;

detecting means for detecting the position of said index in said target image by performing a template matching process between a template image of said index and said target image;

updating means for updating said calculation information stored in said storage means on the basis of a detected position of said index detected by said detecting means; and

calculation means for calculating the attitude and/or position of said measurement object on the basis of said measured value and said calculation information updated by said updating means.

2. The image processing device according to claim 1, wherein said target image setting means obtains a prediction position of the index in said picked-up image employing said measured value and said calculation information stored in said storage means, creates an image with a peripheral area around said prediction position in said picked-up image subjected to a rotational process on the basis of a rotational angle in a roll direction of said image pick-up device derived from said measured value, and outputs said image as a target image.

3. The image processing device according to claim 2, wherein said calculation information is the correction information to correct for an error in the measured value of attitude measured by said attitude sensor, and said calculation means calculates the attitude of said measurement object on the basis of said measured value and said correction information.

4. The image processing device according to claim 1, wherein said calculation information is the correction information to correct for an error in the measured value of attitude measured by said attitude sensor, and said calculation means calculates the attitude of said measurement object on the basis of said measured value and said correction information.

5. The image processing device according to claim 1 or 2, wherein said calculation information is the correction information to correct for an error in the measured value of attitude measured by said attitude sensor and the position information of the image pick-up visual point of said image pickup device, and said calculation means calculates the position and attitude of said measurement object on the basis of said measured value, said correction information and said position information.

6. The image processing device according to claim 1 or 2, wherein said calculation information is the position information of the image pick-up visual point of said image pickup device, and said calculation means calculates the position and attitude of said measurement object on the basis of said measured value and said position information.

7. The image processing device according to claim 5 or 6, wherein said updating means updates the position information in the two directions except for a depth direction in the camera coordinate system of said image pickup device, even when an index of only a single point is detected in said detecting means.

8. The image processing device according to any one of claims 3 to 5, wherein said correction information is the information to correct for an error in the azimuth direction among the measured values of the attitude measured by said attitude sensor.

9. The image processing device according to any one of claims 1 to 8, wherein said updating means updates said calculation information on the basis of the detected position of said index in said picked-up image.

10. The image processing device according to claim 3, 4 or 6, wherein said updating means updates said calculation information on the basis of a typical value of the updated value of said calculation information obtained for each index when a plurality of indices are detected in said detecting means.

11. The image processing device according to claim 3, wherein said updating means updates said calculation information on the basis of a dislocation between the prediction position and said detected position of said index in said target image.

12. The image processing device according to claim 11, wherein said updating means updates said calculation information on the basis of a typical value of said dislocation obtained for each index when a plurality of indices are detected in said detecting means.

13. The image processing device according to any one of claims 5 to 7, wherein said updating means updates the position information in three directions in the camera coordinate system of said image pick-up device, when two or more indices are detected in said detecting means.

14. The image processing device according to any one of claims 1 to 13, wherein said index is a projected image of a landmark in the real space onto said picked-up image.

15. The image processing device according to claim 14, wherein the position of said landmark in the real space is known.

16. The image processing device according to claim 4, 11 or 12, wherein said index is an image feature on said picked-up image.

17. The image processing device according to any one of claims 1 to 16, wherein said measurement object is an image pick-up visual point of said image pick-up device.

18. The image processing device according to claim 17, further comprising display means for displaying said picked-up image with the image in the virtual space superposed thereon on the basis of the attitude, or position and attitude of said image pick-up device calculated by said calculation means.

19. The image processing device according to any one of claims 1 to 16, wherein said measurement object is a visual point of the observer, and said image processing device further comprises display means for displaying the image in the virtual space drawn on the basis of the attitude or position and attitude of the observer calculated by said calculation means on said display screen, while optically transmitting the image in the real space through said display screen observed by the observer.

21. The image processing device according to any one of claims 1 to 20, further comprising template image creating means for creating a template image for detecting said index from the image, wherein said template image creating means specifies the position of the index in the picked-up image when said image pickup device is located at a predetermined position and attitude and creates a template image of the index with a peripheral area around the position of the index in said picked-up image subjected to a rotational process on the basis of the rotational angle in a roll direction from said predetermined positional attitude.

22. The image processing device according to any one of claims 1 to 21, wherein said attitude sensor measures the attitude of an image pick-up visual point of said image pick-up device in a state where there is an accumulated error in the measured value in the azimuth direction.

23. The image processing device according to claim 22, wherein said attitude sensor is constituted of a gyro sensor.

25. The image processing method according to claim 23, wherein said target image setting step comprises creating a target image with a peripheral area around a prediction position in said picked-up image subjected to a rotational process on the basis of the rotational angle in a roll direction of said image pick-up device derived from said measured value by obtaining said prediction position of the index in said picked-up image, employing the measured value and the calculation information stored at said storage step.

20. An image processing device in which the position of an index in a picked-up image picked up by an image pick-up device is detected by template matching employing a template image of said index, comprising:

an attitude sensor for measuring the attitude at an image pick-up visual point of said image pick-up device;

target image creating means for creating a target image having a peripheral area around a prediction position in said picked-up image subjected to a rotational process on the basis of the rotational angle in a roll direction from said measured values by obtaining said prediction position of the index in said picked-up image, employing the measured value of said attitude measured by said attitude sensor, and outputting the target image; and

detecting means for detecting the position of said index in said picked-up image by performing a template matching process between said template image and said target image.

24. An image processing method comprising:

an image pick-up step of picking up an image with an image pick-up device having the fixed positional relation with a measurement object;

an attitude measuring step of measuring the attitude at an image pick-up visual point of said image pick-up device;

a storage step of storing the calculation information to calculate the attitude and/or position of said measurement object on the basis of the measured value measured at said attitude measuring step;

a target image setting step of setting a target image that is an object for detecting a predetermined index on the basis of said picked-up image;

a detecting step of detecting the position of said index in said target image by performing a template matching process between a template image of said index and said target image;

an updating step of updating said calculation information stored at said storage step on the basis of a detected position of said index detected at said detecting step; and

a calculating step of calculating the attitude and/or position of said measurement object on the basis of said measured value and said calculation information updated at said updating step.

26. A program code for executing the image processing method according to claim 24 or 25.

27. A storage medium storing the program code according to claim 26.