Information processing method and apparatus for calculating information regarding measurement target on the basis of captured images

• US 8,320,709 B2
• Filed: 06/21/2007
• Issued: 11/27/2012
• Est. Priority Date: 06/23/2006
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Aratani, Shinichi, Kotake, Daisuke, Uchiyama, Shinji
• Primary Examiner(s): Le, Vu
• Assistant Examiner(s): TSAI, TSUNG YIN
• Application Number:
  US11/766,573
• Publication Number:
  US 20070297695A1
• Time in Patent Office: 1,986 Days
• Field of Search: None
• US Class Current: 382/291
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning

Abstract

From a sequence of images captured by an image pickup unit, images necessary for measuring placement information regarding markers and/or a sensor are automatically determined and obtained. To this end, using position and orientation information regarding the image pickup unit at the time the image pickup unit has captured an obtained image and placement information regarding detected markers, whether to use the captured image corresponding to the position and orientation is determined. Using the captured image determined to be used, the marker placement information, placement information regarding a measurement target, or the position and orientation of the image pickup unit serving as an unknown parameter is obtained so as to minimize the error between the measured image coordinates and theoretical image coordinates of each marker, which are estimated on the basis of a rough value of the parameter.

Claims

1. An information processing method of determining whether to calibrate information regarding a calibration target using images of markers existing in a real space captured by a movable image pickup unit, the method comprising the steps of:

obtaining a first image captured by the image pickup unit from a first viewpoint;

extracting markers from the first image;

obtaining a position of the first viewpoint;

calibrating the information regarding the calibration target using the markers extracted from the first image;

determining a representative position of the markers extracted from the first image based on placement information of the markers;

obtaining a second image captured by the image pickup unit from a second viewpoint after the image pickup unit has been moved from the first viewpoint to the second viewpoint such that the second viewpoint is different from the first viewpoint;

obtaining a position of the second viewpoint;

determining, on the basis of a positional relationship between the position of the first viewpoint, the representative position of the markers and the position of the second viewpoint, whether to calibrate the information regarding the calibration target using the second image; and

calibrating the information regarding the calibration target using the second image in the case that it is determined to calibrate the information regarding the calibration target using the second image.

2. The information processing method according to claim 1, wherein the information regarding the calibration target includes at least one of the placement information regarding the markers, the orientation of the image pickup unit, a parameter of the image pickup unit, an orientation of an orientation sensor which is mounted on the image pickup unit in a camera coordinate system of the image pickup unit, and a position and orientation of a position and orientation sensor which is mounted on the image pickup unit in the camera coordinate system of the image pickup unit.

3. The information processing method according to claim 1, wherein the markers are identifiable markers.

4. The information processing method according to claim 1, wherein the representative position is a barycentric position of the markers extracted from the first image.

5. The information processing method according to claim 1, further comprising the steps of:

calculating, on the basis of the position information of the image pickup unit at the first viewpoint, area information regarding an image capturing area in which the second image should be captured; and

presenting the calculated area information.

6. The information processing method according to claim 5, wherein an area subtending an angle greater than or equal to a predetermined angle relative to a straight line extending from a position represented by the position information of the image pickup unit at the first viewpoint to a representative position of the markers extracted from the captured image is calculated.

7. The information processing method according to claim 1, wherein the determining step includes:

calculating an evaluation value including a value of an angular difference between a vector given from the position of the first viewpoint to the representative position and a vector given from the position of the second viewpoint to the representative position; and

determining to use the second image for calibrating the information regarding the calibration target, in the case that the evaluation value is more than a threshold value set in advance.

8. The information processing method according to claim 1, wherein the determining whether to calibrate further comprises:

calculating an evaluation value including a value of a magnitude ratio between a vector given from the position of the first viewpoint to the representative position and a vector given from the position of the second viewpoint to the representative position; and

determining to use the second image for calibrating the information regarding the calibration target, in the case that the evaluation value is more than a threshold value set in advance.

9. The information processing method according to claim 1, wherein, in the step of determining the representative position based on placement information of the markers extracted from the first image, the markers are weighted in accordance with a result of the extracted markers.

10. The information processing method according to claim 1, wherein the representative position is on a visual axis of the image pickup unit.

11. The information processing method according to claim 1, wherein obtaining the position information of the image pickup unit at the first and second viewpoints by using a position and orientation sensor.

12. The information processing method according to claim 1, wherein obtaining the position information of the image pickup unit at the first and second viewpoints by using the placement information of the markers.

13. The information processing method according to claim 1, further comprising obtaining orientation information of the image pickup unit at the first and second viewpoints by using an orientation sensor mounted onto the image pickup unit,

wherein the information regarding the calibration target includes an orientation of the orientation sensor in a camera coordinate system of the image pickup unit.

14. The information processing method according to claim 1, further comprising obtaining orientation information of the image pickup unit at the first and second viewpoints by using an orientation sensor mounted on the image pickup unit,

wherein the information regarding the calibration target includes an orientation of the orientation sensor in a camera coordinate system of the image pickup unit.

15. The information processing method according to claim 1, further comprising obtaining orientation information of the image pickup unit at the first and second viewpoints by using a position and orientation sensor mounted on the image pickup unit,

wherein the information regarding the calibration target includes an orientation of the orientation sensor in a reference coordinate system.

16. The information processing method according to claim 1,

wherein the representative position is on a visual axis of the image pickup unit;

the method further comprising;

calculating an evaluation value including a value of an angular difference between a vector given from a visual axis of the image pickup unit at the first viewpoint and a vector given from a visual axis of the image pickup unit at the second viewpoint; and

wherein, in the calibration determining step, the second image is determined for use in calibrating the information regarding the calibration target, in the case that the evaluation value is more than a threshold value set in advance.

17. The information processing method according to claim 1, further comprising:

obtaining orientation information of the image pickup unit at the first viewpoint,

calibrating the information regarding the calibration target using the markers extracted from the first image and the obtained position and orientation information of the image pickup unit at the first viewpoint;

obtaining orientation information of the image pickup unit at the second viewpoint; and

calibrating the information regarding the calibration target using the second image and the obtained position and orientation information of the image pickup unit at the first viewpoint, in the case that it is determined to calibrate the information regarding the calibration target using the second image.

22. A non-transitory computer-readable storage medium storing a program which causes a computer to execute an information processing method according to claim 1.

18. An information processing apparatus for determining whether to calibrate information of a calibration target using images of markers existing in a real space captured by a movable image pickup unit, the apparatus comprising:

a captured image obtaining unit configured to obtain a first image captured by the image pickup unit from a first viewpoint, and to obtain a second image captured by the image pickup unit from a second viewpoint;

an extraction unit configured to extract markers from the first image, and to extract markers from the second image;

an obtaining unit configured to obtain a position of the first viewpoint, and to obtain a position of the second viewpoint after the image pickup unit has been moved from the first viewpoint to the second viewpoint such that the second viewpoint is different from the first viewpoint;

a representative position determination unit configured to calculate a representative position of the markers extracted from the first image based on placement information of the markers;

a calibration determination unit configured to determine, on the basis of a positional relationship between the position of the first viewpoint, the representative position of the markers, and the position of the second viewpoint, whether to calibrate the information regarding the calibration target using the second image; and

a calibration unit configured to calibrate the information regarding the calibration target using the markers extracted from the first image, and to calibrate the information regarding the calibration target using the second image in the case that the calibration determination unit determines to calibrate the information regarding the calibration target using the second image.

19. An information processing method of determining whether to calibrate information regarding a calibration target using images of markers existing in a real space captured by a movable image pickup unit, the method comprising the steps of:

obtaining a first image captured by the image pickup unit from a first viewpoint with a first visual axis;

extracting markers from the first image;

obtaining an orientation of the first visual axis;

calibrating the information regarding the calibration target using the markers extracted from the first image;

obtaining a second image captured by the image pickup unit from the second viewpoint with a second visual axis after the image pickup unit has been moved from the first viewpoint to the second viewpoint such that the second viewpoint is different from the first viewpoint;

obtaining an orientation of the second visual axis;

determining, based on an orientational relationship between the orientation of the first visual axis and the orientation of the second visual axis, whether to calibrate the information regarding the calibration target using the second image; and

calibrating the information regarding the calibration target using the second image in the case that it is determined to calibrate the information regarding the calibration target using the second image.

20. The information processing method according to claim 19, further comprising:

calculating an angular difference between a vector given from a visual axis of the image pickup unit at the first viewpoint and a vector given from a visual axis of the image pickup unit at the second viewpoint, on the basis of the position and orientation of the image pickup nit at first viewpoint and the position and orientation of the image pickup unit at the second viewpoint;

wherein, in the step of determining whether to calibrate, the second image is determined to use for calibrating the information regarding the calibration target, in the case that the angular difference is more than a threshold value set in advance.

23. A non-transitory computer-readable storage medium storing a program which causes a computer to execute an information processing method according to claim 19.

21. An information processing apparatus for determining whether to calibrate information regarding a calibration target using images of markers existing in a real space captured by a movable image pickup unit, the apparatus comprising:

a captured image obtaining unit configured to obtain a first image captured by the image pickup unit from a first viewpoint with a first visual axis, and to obtain a second image captured by the image pickup unit from a second viewpoint with a second visual axis;

an extraction unit configured to extract markers from the first image, and to extract markers from the second image;

an obtaining unit configured to obtain an orientation of the first visual axis, and to obtain an orientation of the second visual axis after the image pickup unit has been moved from the first viewpoint to the second viewpoint such that the second viewpoint is different from the first viewpoint;

a determination unit configured to determine, based on an orientational relationship between the orientation of the first visual axis and the orientation of the second visual axis, whether to calibrate the information regarding the calibration target using the second image; and

a calibration unit configured to calibrate the information regarding the calibration target using the markers extracted from the first image, and to calibrate the information regarding the calibration target using the second image in the case that the determination unit determines to calibrate the information regarding the calibration target using the second image.