INFORMATION PROCESSING APPARATUS AND CALIBRATION PROCESSING METHOD

• US 20100295924A1
• Filed: 04/21/2010
• Published: 11/25/2010
• Est. Priority Date: 05/21/2009
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Miyatani, Sonoko, Fujiki, Masakazu, Morita, Kenji
• Granted Patent: US 8,830,304 B2
• Time in Patent Office: Days
• US Class Current: 348/46
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays

Abstract

An information processing apparatus, which provides images for stereoscopic viewing by synthesizing images obtained by capturing an image of real space by a main image sensing device and sub image sensing device to a virtual image, measures the position and orientation of the main image sensing device, calculates the position and orientation of the sub image sensing device based on inter-image capturing device position and orientation held in a holding unit and the measured position and orientation of the main image sensing device. Then the information processing apparatus calculates an error using the measured position and orientation of the main image sensing device, the calculated position and orientation of the sub image sensing device, and held intrinsic parameters of the main image sensing device and sub image sensing device. The information processing apparatus calibrates the held information based on the calculated error.

Claims

1. An information processing apparatus, which provides images for stereoscopic viewing by synthesizing images, obtained by capturing an image of a real space by a stereoscopic viewing apparatus that mounts both a first image capturing device and a second image capturing device, with a virtual image, said apparatus comprising:

a position and orientation measurement unit which measures a position and orientation of the first image capturing device;

a first acquisition unit which acquires a first captured image obtained by capturing an index on the real space by the first image capturing device;

a first index detection unit which detects the index from the first captured image;

a calculation unit which calculates a projected index which may be included in an image captured from the second image capturing device, based on inter-device position and orientation information that indicates a position and orientation relationship between the first image capturing device and the second image capturing device and is held in a holding unit, intrinsic parameters of the first and second image capturing devices that are held in the holding unit, the position and orientation of the first image capturing device measured by said position and orientation measurement unit, and the index detected by said first index detection unit;

a second acquisition unit which acquires a second captured image obtained by capturing an index on the real space by the second image capturing device;

a second index detection unit which detects the index from the second captured image;

a judgment unit which judges based on the projected index and the index detected by said second index detection unit whether or not an error is included in at least one of pieces of calibration target information including the inter-device position and orientation information held in the holding unit, the intrinsic parameter of the first image capturing device, and the intrinsic parameter of the second image capturing device;

a calibration target measurement unit which measures, when said judgment unit judges that an error is included in one of the pieces of calibration target information, that calibration target information;

a determination unit which determines an error to be calibrated of the calibration target information by comparing the calibration target information held in the holding unit and calibration target information measured by said calibration target information measurement unit; and

a calibration unit which calibrates the calibration target information held in the holding unit, based on a determination result of said determination unit.

2. The apparatus according to claim 1, wherein said calibration target measurement unit measures the intrinsic parameters of the first and second image capturing devices from the first and second captured images, and

when a difference between each of the intrinsic parameters measured by said calibration target measurement unit and each of the intrinsic parameters held in the holding unit exceeds a predetermined value, said determination unit determines to calibrate the intrinsic parameter which exceeds the predetermined value.

3. The apparatus according to claim 1, wherein said calibration target measurement unit measures the intrinsic parameters of the first and second image capturing devices from the first and second captured images, and

when a difference between each of the intrinsic parameters measured by said calibration target measurement unit and each of the intrinsic parameters held in the holding unit does not exceed a predetermined value, said determination unit determines to calibrate the inter-device position and orientation.

4. The apparatus according to claim 1, further comprising a second measurement unit which measures a position and orientation of the second image capturing device based on the index detected by said second index detection unit,

wherein said calibration target measurement unit measures inter-device position and orientation information based on the position and orientation of the first image capturing device measured by said position and orientation measurement unit and the position and orientation of the second image capturing device measured by said second measurement unit, and

when a difference between the calculated inter-device position and orientation information and inter-device position and orientation information held in the holding unit exceeds a predetermined value, said determination unit determines to calibrate the inter-device position and orientation information.

5. The apparatus according to claim 1, further comprising a second measurement unit which measures a position and orientation of the second image capturing device based on the index detected by said second index detection unit,

wherein said calibration target information measurement unit measures inter-device position and orientation information based on the position and orientation of the first image capturing device measured by said position and orientation measurement unit and the position and orientation of the second image capturing device measured by said second measurement unit, and

when a difference between the calculated inter-device position and orientation information and inter-device position and orientation information held in the holding unit does not exceed a predetermined value, said determination unit determines to calibrate the intrinsic parameter of one of the first image capturing device and the second image capturing device.

6. The apparatus according to claim 1, wherein the index is a marker, and

said judgment unit judges the error based on a magnitude of a deviation between a marker as the projected index and a marker as the index detected by said second index detection unit.

7. The apparatus according to claim 1, wherein each of said first index detection unit and said second index detection unit detects the index as an edge, and

said judgment unit judges the error based on as to whether or not an edge as the projected index is parallel to an edge as the index detected by said second index detection unit.

8. The apparatus according to claim 1, wherein said calibration unit calibrates the intrinsic parameters of the first and second image capturing devices, and then calibrates the inter-device position and orientation information.

9. The apparatus according to claim 1, further comprising a notification unit which notifies, when said judgment unit judges that an error is included in one of the pieces of calibration target information, a user of the judgment result.

10. The apparatus according to claim 1, characterized by further comprising a notification unit which notifies a judgment result of said judgment unit which judges again whether or not an error is included after said calibration unit performs the calibration.

11. The apparatus according to claim 1, wherein the intrinsic parameter includes at least one of a focal length, a principal point position, and a distortion of the first or second image capturing device.

12. A calibration processing method by an information processing apparatus, which provides images for stereoscopic viewing by synthesizing images, obtained by capturing an image of a real space by a stereoscopic viewing apparatus that mounts both a first image capturing device and a second image capturing device, with a virtual image, the method comprising:

a position and orientation measurement step of measuring a position and orientation of the first image capturing device;

a first acquisition step of acquiring a first captured image obtained by capturing an index on the real space by the first image capturing device;

a first index detection step of detecting the index from the first captured image;

a calculation step of calculating a projected index which may be included in an image captured from the second image capturing device, based on inter-device position and orientation information that indicates a position and orientation relationship between the first image capturing device and the second image capturing device and is held in a holding unit, intrinsic parameters of the first and second image capturing devices that are held in the holding unit, the position and orientation of the first image capturing device measured in the position and orientation measurement step, and the index detected in the first index detection step;

a second acquisition step of acquiring a second captured image obtained by capturing an index on the real space by the second image capturing device;

a second index detection step of detecting the index from the second captured image;

a judgment step of judging based on the projected index and the index detected in the second index detection step whether or not an error is included in at least one of pieces of calibration target information including the inter-device position and orientation information held in the holding unit, the intrinsic parameter of the first image capturing device, and the intrinsic parameter of the second image capturing device;

a calibration target measurement step of measuring, when it is judged in the judgment step that an error is included in one of the pieces of calibration target information, that calibration target information;

a determination step of determining an error to be calibrated of the calibration target information by comparing the calibration target information held in the holding unit and calibration target information measured in the calibration target information measurement step; and

a calibration step of calibrating the calibration target information held in the holding unit, based on a determination result in the determination step.

13. A computer-readable storage medium storing a computer program for making a computer execute respective steps of a calibration processing method according to claim 12.