Information processing apparatus and calibration processing method

• US 8,830,304 B2
• Filed: 04/21/2010
• Issued: 09/09/2014
• Est. Priority Date: 05/21/2009
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Morita, Kenji, Fujiki, Masakazu, Miyatani, Sonoko
• Primary Examiner(s): Perungavoor, Sath V
• Assistant Examiner(s): Anderson, Matthew J
• Application Number:
  US12/764,591
• Publication Number:
  US 20100295924A1
• Time in Patent Office: 1,602 Days
• Field of Search: 382100-300, 348 42- 60
• US Class Current: 348/47
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays

Abstract

An information processing apparatus, which provides images for stereoscopic viewing by synthesizing images obtained by capturing an image of real space by a main image sensing device and sub image sensing device to a virtual image, measures the position and orientation of the main image sensing device, calculates the position and orientation of the sub image sensing device based on inter-image capturing device position and orientation held in a holding unit and the measured position and orientation of the main image sensing device. Then the information processing apparatus calculates an error using the measured position and orientation of the main image sensing device, the calculated position and orientation of the sub image sensing device, and held intrinsic parameters of the main image sensing device and sub image sensing device. The information processing apparatus calibrates the held information based on the calculated error.

Claims

1. An information processing apparatus, which provides images for stereoscopic viewing by synthesizing images, obtained by capturing an image of a real space by a stereoscopic viewing apparatus that mounts both a first image capturing device and a second image capturing device, with a virtual image, said apparatus comprising:

a first acquisition unit configured to acquire a first captured image obtained by capturing an index on the real space by the first image capturing device;

a first index detection unit configured to detect the index from the first captured image;

a position and orientation measurements unit configured to measure a first position and orientation of the first capturing device based on the index detected by the first index detection unit;

a holding unit configured to hold information including at least relative position and orientation information that indicates a relative position and orientation relationship between the first capturing device and the second capturing device;

a first calculation unit configured to calculate a position and orientation of the second image capturing device, based on the relative position and orientation information held in the holding unit, and the first position and orientation of the first image capturing device measured by said position and orientation measurement unit;

a second calculation unit configured to calculate a position of a projected index in a projection plane, by projecting the index detected from the first captured image on the projection plane based on the position and orientation of the second image capturing device calculated by the first calculation unit;

a second acquisition unit configured to acquire a second captured image obtained by capturing an index on the real space by the second image capturing device;

a second index detection unit configured to detect the index from the second captured image;

an error detection unit configured to detect an error in the relative position and orientation information held in the holding unit by comparing positions of the projected index with the index detected by said second index detection unit; and

an output unit configured to output the error when the error detection unit detects the error.

2. The apparatus according to claim 1,

wherein the information held in the holding unit further includes an instrinsic parameter of the first and second image capturing devices, and

said apparatus further comprises:

a calibration target measurement unit configured to measure calibration target information including at least one of the relative position and orientation information, and the intrinsic parameter when said error detection unit detects an error;

a determination unit configured to determine an error to be calibrated of the calibration target information by comparing the information held in the holding unit and the calibration target information measured by said calibration target information measurement unit; and

a calibration unit configured to calibrate the information held in the holding unit, based on a determination result of said determination unit.

3. The apparatus according to claim 2, wherein said calibration target measurement unit measures the intrinsic parameters of the first and second image capturing devices from the first and second captured images, and

when a difference between each of the intrinsic parameters measured by said calibration target measurement unit and each of the intrinsic parameters held in the holding unit exceeds a predetermined value, said determination unit determines to calibrate the intrinsic parameter which exceeds the predetermined value.

4. The apparatus according to claim 2, wherein said calibration target measurement unit measures the intrinsic parameters of the first and second image capturing devices from the first and second captured images, and

when a difference between each of the intrinsic parameters measured by said calibration target measurement unit and each of the intrinsic parameters held in the holding unit does not exceed a predetermined value, said determination unit determines to calibrate the inter-device position and orientation.

5. The apparatus according to claim 2, further comprising a second measurement unit configured to measure a position and orientation of the second image capturing device based on the index detected by said second index detection unit,

wherein said calibration target measurement unit measures relative position and orientation information based on the position and orientation of the first image capturing device measured by said position and orientation measurement unit and the position and orientation of the second image capturing device measured by said second measurement unit, and

when a difference between the calculated relative position and orientation information and relative position and orientation information held in the holding unit exceeds a predetermined value, said determination unit determines to calibrate the inter-device position and orientation information.

6. The apparatus according to claim 2, further comprising a second measurement unit configured to measure a position and orientation of the second image capturing device based on the index detected by said second index detection unit,

wherein said calibration target information measurement unit measures relative position and orientation information based on the position and orientation of the first image capturing device measured by said position and orientation measurement unit and the position and orientation of the second image capturing device measured by said second measurement unit, and

when a difference between the calculated inter-device position and orientation information and inter-device position and orientation information held in the holding unit does not exceed a predetermined value, said determination unit determines to calibrate the intrinsic parameter of one of the first image capturing device and the second image capturing device.

7. The apparatus according to claim 1, wherein the index is a marker, and

said error detection unit detects the error based on a magnitude of a deviation between a marker as the projected index and a marker as the index detected by said second index detection unit.

8. The apparatus according to claim 1, wherein each of said first index detection unit and said second index detection unit detects the index as an edge, and

said error detection unit detects the error based on whether or not an edge as the projected index is parallel to an edge as the index detected by said second index detection unit.

9. The apparatus according to claim 2, wherein said calibration unit calibrates the intrinsic parameters of the first and second image capturing devices, and then calibrates the relative position and orientation information.

10. The apparatus according to claim 2,

further comprising a notification unit configured to notify the error in the held relative position and orientation information to a user when the error detection unit detects the error,

wherein said notification unit notifies a judgment result of said error detection unit and judges again whether or not an error is included after said calibration unit performs the calibration.

11. The apparatus according to claim 2, wherein the intrinsic parameter includes at least one of a focal length, a principal point position, and a distortion of the first or second image capturing device.

12. The apparatus according to claim 1, wherein the output unit outputs the error to a display device.

13. The apparatus according to claim 1, further comprising a notification unit configured to notify the error in the held relative position and orientation information to a user when the error detection unit detects the error.

14. The apparatus according to claim 1, further comprising a second calibration unit configured to calibrate the relative position and orientation information held in the holding unit when the error detection unit detects the error.

15. A calibration processing method by an information processing apparatus, which provides images for stereoscopic viewing by synthesizing images, obtained by capturing an image of a real space by a stereoscopic viewing apparatus that mounts both a first image capturing device and a second image capturing device, with a virtual image, the method comprising:

a first acquisition step of acquiring a first captured image obtained by capturing an index on the real space by the first image capturing device;

a first index detection step of detecting the index from the first captured image;

a position and orientation measurement step of measuring a first position and orientation of the first image capturing device based on the index detected in the first index detection step;

a holding step of holding, in a holding unit, information including at least relative position and orientation information that indicates a relative position and orientation relationship between the first capturing device and the second capturing device;

a first calculation step of calculating a position and orientation of the second image capturing device, based on the relative position and orientation information held in the holding unit, and the first position and orientation of the first image capturing device measured in the position and orientation measurement step;

a second calculation step of calculating a position of a projected index in a projection plane, by projecting the index detected from the first captured image on the projection plane based on the position and orientation of the second image capturing device calculated by the first calculation step;

a second acquisition step of acquiring a second captured image obtained by capturing an index on the real space by the second image capturing device;

a second index detection step of detecting the index from the second captured image;

an error detection step of detecting an error in the relative position and orientation information held in the holding step by comparing positions of the projected index with the index detected in the second index detection step; and

an output step of outputting the error when the error is detected in the error detection step.

16. A non-transitory computer-readable storage medium storing a computer program for making a computer execute respective steps of a calibration processing method according to claim 15.