IMAGE CORRECTION DEVICE, METHOD, PROGRAM, INTEGRATED CIRCUIT, AND SYSTEM

• US 20100007751A1
• Filed: 02/23/2007
• Published: 01/14/2010
• Est. Priority Date: 02/23/2006
• Status: Active Grant

• Current Assignee: Piece Future Pte., Ltd.
• Original Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Inventors: Ohmiya, Jun, Icho, Keiji
• Granted Patent: US 8,085,308 B2
• Time in Patent Office: Days
• US Class Current: 348/222.100
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof

Abstract

An image correction device trims an image, which has been captured by a camera provided in a moving object, in accordance with a predetermined image frame to generate a corrected image that is enclosed by the predetermined image frame, and then outputs the corrected image as an output image. Even when there is an error between an actual camera position in which the camera is actually positioned and a scheduled camera position in which the camera is scheduled to be positioned, the image correction device (i) decomposes the error into a first component that extends along an imaging direction of the camera and a second component that extends along a plane orthogonal to the imaging direction, (ii) shifts the image frame in accordance with the first component, and (iii) enlarges or shrinks a size of the image frame in accordance with the second component.

Claims

1. An image correction device for correcting at least one image that has been captured by a camera provided in a moving object, comprising:

an image acquisition unit operable to acquire the image captured by the camera;

a difference acquisition unit operable to acquire positional difference information indicating a difference between (i) a scheduled camera position in which the camera is scheduled to be positioned at a scheduled time to capture the image and (ii) an actual camera position in which the camera is positioned at the scheduled time;

a determination unit operable to determine an output image frame, by correcting a standard image frame in accordance with the acquired positional difference information; and

a corrected image output unit operable to (i) trim the captured image in accordance with the determined output image frame to generate a corrected image that is enclosed by the determined output image frame, and (ii) output the corrected image.

2. The image correction device of claim 1, further comprising:

a camera information acquisition unit operable to acquire imaging direction information indicating at least one of (i) an actual imaging direction that is in range with an optical axis of the camera along which the camera captures the image from the actual camera position, and (ii) a scheduled imaging direction that is in range with an optical axis of the camera along which the camera is scheduled to perform an image capturing from the scheduled camera position, wherein

the standard image frame is located within the captured image in a position that is determined by a first reference point, and

the determination unit includes;

a decomposition subunit operable to, in accordance with the acquired imaging direction information, decompose the difference indicated by the positional difference information into (i) a first component that extends along the imaging direction indicated by the imaging direction information, and (ii) a second component that extends along a plane orthogonal to the imaging direction;

a size calculation subunit operable to determine a size of the output image frame, by correcting a size of the standard image frame in accordance with the first component; and

a point calculation subunit operable to determine a second reference point that determines a position in which the output image frame is located within the captured image, by shifting the first reference point of the standard image frame in accordance with the second component.

3. The image correction device of claim 2, wherein

the difference indicated by the positional difference information is a vector whose initial point is the scheduled camera position and whose terminal point is the actual camera position at the scheduled time,

the second component is a projection vector that is projected on the orthogonal plane, and

the point calculation subunit is operable to (i) calculate, in accordance with a magnitude of the projection vector, a shift amount by which the first reference point should be shifted, and (ii) shift the first reference point of the standard image frame by the calculated shift amount in a direction opposite to a direction pointed by the projection vector extending along the orthogonal plane.

4. The image correction device of claim 3, wherein

the camera information acquisition unit is further operable to acquire (i) angle-of-view information indicating at least one of (a) an angle of view of the camera positioned in the actual camera position at the scheduled time and (b) an angle of view that the camera is scheduled to have when performing the image capturing from the scheduled camera position, and (ii) distance information indicating at least one of (c) a distance between the actual camera position and a subject to be captured in the image and (d) a distance between the scheduled camera position and the subject to be captured,

the point calculation subunit is operable to (i) calculate a size of an imaging range in accordance with the acquired angle-of-view information and distance information, and (ii) calculate the shift amount in view of a ratio of the size of the imaging range to the magnitude of the projection vector.

5. The image correction device of claim 2, wherein

the difference is expressed by a vector whose initial point is the scheduled camera position and whose terminal point is the actual camera position at the scheduled time, and

the size calculation subunit determines the output image frame by (i) enlarging the size of the standard image frame when the first component of the vector is pointing in the same direction as the imaging direction, or (ii) shrinking the size of the standard image frame when the first component of the vector is pointing in a direction opposite to the imaging direction.

6. The image correction device of claim 2, wherein

the size calculation subunit (i) includes a scale determination subunit operable to, in accordance with a magnitude of the first component, determine a scale according to which the size of the standard image frame should be corrected, and (ii) corrects the size of the standard image frame according to the determined scale.

7. The image correction device of claim 6, wherein

the difference indicated by the positional difference information is a vector whose initial point is the scheduled camera position and whose terminal point is the actual camera position at the scheduled time,

the magnitude of the first component of the vector is a, while a distance between the scheduled camera position and a subject to be captured in the image is b,

the camera information acquisition unit is further operable to acquire distance information indicating the distance between the scheduled camera position and the subject to be captured, and

the scale determination subunit determines the scale to be (i) b/(b−

a) when the first component of the vector is pointing in the same direction as the imaging direction, and (ii) b/(b+a) when the first component of the vector is pointing in a direction opposite to the imaging direction.

8. The image correction device of claim 6, wherein

the corrected image output unit (i) includes an output image size adjustment subunit operable to (a) trim the captured image in accordance with the output image frame to generate the corrected image that is enclosed by the determined output image frame and (b) enlarge or shrink the corrected image by multiplying a size of the corrected image by a multiplicative inverse of the determined scale, and (ii) outputs the corrected image whose size has been multiplied by the multiplicative inverse of the determined scale.

9. The image correction device of claim 1, wherein

the camera can be rotated in roll about an imaging direction that is in range with an optical axis of the camera along which the camera captures the image at the scheduled time, the imaging direction serving as a rotation axis,

the image correction device further comprises a roll angle information acquisition unit operable to acquire actual roll angle information indicating an actual rotation amount by which an imaging range has been rotated at the scheduled time, the actual rotation amount being expressed by means of an angle, and

the determination unit determines the output image frame, by correcting the standard image frame in accordance with the positional difference information in view of the acquired actual roll angle information.

10. The image correction device of claim 9, wherein

the roll angle information acquisition unit is further operable to acquire scheduled roll angle information indicating a scheduled rotation amount, by which the imaging range is scheduled to be rotated when the camera is in the scheduled camera position,

the determination unit includes a rotation amount difference calculation subunit operable to calculate a rotation amount difference indicating a difference between (i) the scheduled rotation amount indicated by the acquired scheduled roll angle information and (ii) the actual rotation amount indicated by the acquired actual roll angle information, and

the determination unit determines the output image frame, by rotating the corrected standard image frame according to the rotation amount difference calculated by the rotation amount difference calculation subunit.

11. The image correction device of claim 10, wherein

the standard image frame is located within the captured image in a position that is determined by a first reference point,

the determination unit further includes:

a point calculation subunit operable to determine, in accordance with the positional difference information, a direction in which and a shift amount by which the first reference point of the standard image frame should be shifted; and

a point correction subunit operable to (i) shift the first reference point by the determined shift amount in the determined direction, and (ii) rotate the shifted first reference point about the original first reference point of the standard image frame, by the actual rotation amount indicated by the actual roll angle information, in a direction opposite to a rotation direction suggested by the actual roll angle information, and

the determination unit determines the output image frame by (i) positionally determining a new image frame within the captured image in accordance with the shifted and rotated first reference point, and (ii) rotating the new image frame about the shifted and rotated first reference point, by the rotation amount difference calculated by the rotation amount difference calculation subunit, in a direction opposite to a rotation direction suggested by the rotation amount difference.

12. The image correction device of claim 1, further comprising:

an imaging direction error information acquisition unit operable to acquire imaging direction error information indicating an error between (i) an actual imaging direction that is in range with an optical axis of the camera along which the camera captures the image from the actual camera position, and (ii) a scheduled imaging direction that is in range with an optical axis of the camera along which the camera is scheduled to perform an image capturing from the scheduled camera position, wherein

the determination unit determines the output image frame, by correcting the standard image frame in accordance with the positional difference information in view of the acquired imaging direction error information.

13. The image correction device of claim 12, further comprising:

a camera information acquisition unit operable to acquire imaging direction information indicating at least one of the actual imaging direction and the scheduled imaging direction, wherein

the standard image frame is located within the captured image in a position that is determined by a first reference point, and

the determination unit includes;

a decomposition subunit operable to, in accordance with the acquired imaging direction information, decompose (i) the difference indicated by the positional difference information into (a) a first component that extends along the imaging direction indicated by the imaging direction information and (b) a second component that extends along a plane orthogonal to the imaging direction, and (ii) the error indicated by the imaging direction error information into (c) a first component that extends along the imaging direction indicated by the imaging direction information and (d) a second component that extends along a plane orthogonal to the imaging direction;

a size calculation subunit operable to determine a size of the output image frame, by correcting a size of the standard image frame in accordance with the first component of the difference and the first component of the error; and

a point calculation subunit operable to determine a second reference point that determines a position in which the output image frame is located within the captured image, by shifting the first reference point of the standard image frame in accordance with the second component of the difference and the second component of the error.

14. The image correction device of claim 1, further comprising:

a movement error information acquisition unit operable to acquire moving object movement error information indicating at least one of (i) an actual roll angle and a scheduled roll angle, (ii) an actual pitch angle and a scheduled pitch angle, and (iii) an actual yaw angle and a scheduled yaw angle, where the actual roll, pitch and yaw angles are of the moving object at the scheduled time, whereas the scheduled roll, pitch and yaw angles are of the moving object when the camera is in the scheduled camera position, wherein

the determination unit determines the output image frame, by correcting the standard image frame in accordance with the positional difference information in view of the acquired moving object movement error information.

15. The image correction device of claim 14, further comprising:

a camera information acquisition unit operable to acquire imaging direction information indicating at least one of (i) an actual imaging direction that is in range with an optical axis of the camera along which the camera captures the image from the actual camera position, and (ii) a scheduled imaging direction that is in range with the optical axis of the camera along which the camera is scheduled to perform an image capturing from the scheduled camera position, wherein

the determination unit includes;

a decomposition subunit operable to, in accordance with the acquired imaging direction information, decompose (i) the difference indicated by the positional difference information into (a) a first component that extends along the imaging direction indicated by the imaging direction information and (b) a second component that extends along a plane orthogonal to the imaging direction, and (ii) the difference indicated by the moving object movement error information into (c) a first component that extends along the imaging direction indicated by the imaging direction information and (d) a second component that extends along a plane orthogonal to the imaging direction;

a size calculation subunit operable to determine a size of the output image frame, by correcting a size of the standard image frame in accordance with a sum of (i) the first component of the difference indicated by the positional difference information and (ii) the first component of the difference indicated by the moving object movement error information; and

a point calculation subunit operable to determine a second reference point that determines a position in which the output image frame is located within the captured image, by shifting the first reference point of the standard image frame in accordance with a sum of (i) the second component of the difference indicated by the positional difference information and (ii) the second component of the difference indicated by the moving object movement error information.

16. The image correction device of claim 1, wherein

the standard image frame is located within the captured image in a position that is determined by a first reference point, and

the determination unit includes a point calculation subunit operable to determine a second reference point that determines a position in which the output image frame is located within the captured image, by shifting the first reference point of the standard image frame in accordance with the difference indicated by the positional difference information.

17. The image correction device of claim 1, wherein

the determination unit is operable to determine a size of the output image frame, by enlarging or shrinking a size of the standard image frame in accordance with the difference indicated by the positional difference information.

18. The image correction device of claim 1, further comprising:

an instruction information acquisition unit operable to acquire instruction information indicating a path that the moving object is scheduled to follow, wherein

the moving object makes the camera capture the image while moving in accordance with the instruction information, and

the scheduled camera position is on the path indicated by the instruction information.

19. The image correction device of claim 1, further comprising:

an image correction information acquisition unit operable to acquire and store therein image correction information that includes a correction method for correcting the standard image frame, the image correction information and the correction method corresponding to the difference indicated by the positional difference information, wherein

the determination unit determines the output image frame in accordance with the correction method, by reading out the correction method from the stored image correction information.

20. The image correction device of claim 19, wherein

the standard image frame is located within the captured image in a position that is determined by a first reference point,

the image correction information includes, in one-to-one correspondence, (i) a size of the output image frame and (ii) a second reference point that determines a position in which the output image frame is located within the captured image, and

the determination unit determines the output image frame to be a frame specified by the size and the second reference point that are included in the image correction information.

21. The image correction device of claim 1, wherein

the moving object makes the camera capture the at least one image that is used for generating a video,

the image correction device makes the determination unit determine the output image frame for each of the images, and

the corrected image output unit includes a video output subunit operable to output the video by (i) trimming each image in accordance with a corresponding one of the determined output image frames to generate at least one corrected image each of which is enclosed by the corresponding one of the output image frames, and (ii) reconstructing the corrected images in a video format.

22. An image correction system for correcting at least one image that has been captured by a camera provided in a moving object, comprising:

a scheduled position reception unit operable to receive a scheduled camera position in which the camera is scheduled to be positioned at a scheduled time to capture the image;

a scheduled position information storage unit operable to store therein the received scheduled camera position as scheduled position information;

a trajectory information acquisition unit operable to acquire trajectory information indicating a trajectory that the moving object has actually followed;

a positional difference calculation unit operable to, in accordance with the scheduled position information and the acquired trajectory information, calculate a difference between (i) the scheduled camera position and (ii) an actual camera position in which the camera is positioned at the scheduled time;

a determination unit operable to determine an output image frame, by correcting a standard image frame in accordance with the difference calculated by the positional difference calculation unit; and

a corrected image output unit operable to (i) trim the captured image in accordance with the determined output image frame to generate a corrected image that is enclosed by the determined output image frame, and (ii) output the corrected image.

23. An image correction method for correcting at least one image that has been captured by a camera provided in a moving object, comprising the steps of:

acquiring the image captured by the camera;

acquiring positional difference information indicating a difference between (i) a scheduled camera position in which the camera is scheduled to be positioned at a scheduled time to capture the image and (ii) an actual camera position in which the camera is positioned at the scheduled time;

determining an output image frame by correcting a standard image frame in accordance with the calculated difference; and

trimming the captured image in accordance with the determined output image frame to generate a corrected image that is enclosed by the determined output image frame, and outputting the corrected image.

24. A control program that makes an image correction method perform image correction processing for correcting at least one image that has been captured by a camera provided in a moving object, comprising the steps of:

acquiring the image captured by the camera;

acquiring positional difference information indicating a difference between (i) a scheduled camera position in which the camera is scheduled to be positioned at a scheduled time to capture the image and (ii) an actual camera position in which the camera is positioned at the scheduled time;

determining an output image frame by correcting a standard image frame in accordance with the calculated difference; and

trimming the captured image in accordance with the determined output image frame to generate a corrected image that is enclosed by the determined output image frame, and outputting the corrected image.

25. An integrated circuit that performs image correction processing for correcting at least one image that has been captured by a camera provided in a moving object, comprising:

an image acquisition unit operable to acquire the image captured by the camera;

a difference acquisition unit operable to acquire positional difference information indicating a difference between (i) a scheduled camera position in which the camera is scheduled to be positioned at a scheduled time to capture the image and (ii) an actual camera position in which the camera is positioned at the scheduled time;

a determination unit operable to determine an output image frame by correcting a standard image frame in accordance with the calculated difference; and

a corrected image output unit operable to (i) trim the captured image in accordance with the determined output image frame to generate a corrected image that is enclosed by the determined output image frame, and (ii) output the corrected image.