Surroundings monitoring apparatus and surroundings monitoring method for reducing distortion caused by camera position displacement

• US 8,139,114 B2
• Filed: 10/03/2006
• Issued: 03/20/2012
• Est. Priority Date: 02/15/2005
• Status: Active Grant

• Current Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Original Assignee: Panasonic Corporation (Panasonic Holdings Corporation)
• Inventors: Nobori, Kunio, Sato, Masaki
• Primary Examiner(s): Patel, Jay Au
• Assistant Examiner(s): PONTIUS, JAMES M
• Application Number:
  US11/541,684
• Publication Number:
  US 20070041659A1
• Time in Patent Office: 1,995 Days
• Field of Search: 348/148, 348/222.1, 348/333.01, 348/373
• US Class Current: 348/148
• CPC Class Codes: B60R 1/23 with a predetermined field of view
• CPC Class Codes: B60R 2300/102 using 360 degree surveillance camera system
• CPC Class Codes: B60R 2300/105 using multiple cameras
• CPC Class Codes: B60R 2300/301 combining image information with other obstacle sensor information, e.g. using RADAR/LIDAR/SONAR sensors for estimating risk of collision
• CPC Class Codes: B60R 2300/305 merging camera image with lines or icons
• CPC Class Codes: B60R 2300/402 Image calibration
• CPC Class Codes: B60R 2300/607 from a bird's eye viewpoint
• CPC Class Codes: B60R 2300/806 for aiding parking
• CPC Class Codes: G07C 5/0891 Video recorder in combination with video camera
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A surrounding monitoring apparatus reduces distortion caused by camera position displacement. A position displacement information obtainment unit obtains position displacement information and outputs it as camera position displacement information, a moving object state detection unit detects vehicle information and outputs it as a moving object state, a coordinate transformation parameter selection unit selects a coordinate transformation parameter according to the camera position displacement information, an image synthesis parameter selection unit selects a reference image synthesis parameter and a graphics parameter according to the moving object state, a coordinate transformation unit performs coordinate transformation on image coordinates of a captured image which are included in the reference image synthesis parameter using the coordinate transformation parameter and outputs the image coordinates as a transformed image synthesis parameter, and an image synthesis unit generates and outputs a synthetic image according to the transformed image synthesis parameter.

Claims

1. A surroundings monitoring apparatus which generates a surrounding image using an image captured by a camera, the surroundings monitoring apparatus comprising:

a position displacement information obtainment unit operable to obtain position displacement information which is information regarding a displacement in a position of the camera;

a coordinate transformation parameter obtainment unit operable to obtain, based on the position displacement information obtained by the position displacement information obtainment unit, a coordinate transformation parameter for correcting a distortion in the surrounding image caused by the displacement in the position, the coordinate transformation parameter defining a function for transforming two-dimensional coordinates of one pixel into two-dimensional coordinates of another pixel;

an image synthesis parameter obtainment unit operable to obtain at least one reference image synthesis parameter which is used for generating the surrounding image using the captured image, the at least one reference image synthesis parameter including information of image coordinates indicating, on the captured image, a pixel position corresponding to each pixel in the surrounding image, and the image coordinates being two-dimensional coordinates;

a coordinate transformation unit operable to perform coordinate transformation on the two-dimensional image coordinates included in the at least one reference image synthesis parameter obtained by the image synthesis parameter obtainment unit, using the coordinate transformation parameter obtained by the coordinate transformation parameter obtainment unit, and to output the image coordinates as a transformed image synthesis parameter indicating transformed two-dimensional image coordinates; and

an image synthesis unit operable to generate the surrounding image by applying the transformed image synthesis parameter to the captured image, and to output the generated surrounding image,

wherein the image synthesis unit is operable to read, from among pixels included in the captured image, an image of a pixel located in the transformed two-dimensional image coordinates indicated by the transformed image synthesis parameter, and to output the read image as an image of a pixel located in the surrounding image and corresponding to the pixel located in the transformed two-dimensional image coordinates.

2. The surroundings monitoring apparatus according to claim 1,

wherein the coordinate transformation parameter obtainment unit includes:

a coordinate transformation parameter storage unit operable to store at least one coordinate transformation parameter beforehand, and

a coordinate transformation parameter selection unit operable to select a coordinate transformation parameter from the at least one coordinate transformation parameter based on the position displacement information obtained by the position displacement information obtainment unit.

3. The surroundings monitoring apparatus according to claim 1,

wherein the coordinate transformation parameter obtainment unit is operable to generate the coordinate transformation parameter based on the position displacement information obtained by the position displacement information obtainment unit, and obtain the generated coordinate transformation parameter as the coordinate transformation parameter for correcting distortion.

4. The surroundings monitoring apparatus according to claim 1, further comprising:

a graphics synthesis unit operable to superimpose graphics on the surrounding image generated by the image synthesis unit, and to output the image superimposed with graphics,

wherein the coordinate transformation parameter obtainment unit is operable to obtain the coordinate transformation parameter based on the position displacement information obtained by the position displacement information obtainment unit and the graphics superimposed by the graphics synthesis unit.

5. The surroundings monitoring apparatus according to claim 4,

wherein the coordinate transformation parameter obtainment unit is operable to obtain a coordinate transformation parameter in which a position error of the surrounding image after the coordinate transformation is minimal in a predetermined position of the graphics superimposed by the graphics synthesis unit.

6. The surroundings monitoring apparatus according to claim 1,

wherein the image synthesis parameter obtainment unit includes:

an image synthesis parameter storage unit operable to store the at least one reference image synthesis parameter beforehand, and

an image synthesis parameter selection unit operable to select a reference image synthesis parameter from the at least one reference image synthesis parameter.

7. The surroundings monitoring apparatus according to claim 6, further comprising:

a state detection unit operable to detect a state of a moving object including information regarding a location of the moving object which is equipped with the surroundings monitoring apparatus,

wherein the image synthesis parameter selection unit is operable to select one of the at least one reference image synthesis parameters based on the moving object state detected by the state detection unit.

8. The surroundings monitoring apparatus according to claim 6,

wherein the coordinate transformation parameter obtainment unit is operable to obtain the coordinate transformation parameter based on the position displacement information obtained by the position displacement information obtainment unit and the reference image synthesis parameter selected by the image synthesis parameter selection unit.

9. The surroundings monitoring apparatus according to claim 1,

wherein the reference image synthesis parameter includes information of the image coordinates for a part of the captured image corresponding to the surrounding image, and

the coordinate transformation parameter obtainment unit is operable to obtain a coordinate transformation parameter in which a position error of the surrounding image after the coordinate transformation is minimal in a predetermined position for the part of the captured image.

10. The surroundings monitoring apparatus according to claim 1,

wherein the reference image synthesis parameter includes plural pieces of information of image coordinates for a plurality of the captured images, the information being used for generating the surrounding image using a plurality of the images captured by plural cameras,

the position displacement information obtainment unit is operable to obtain plural pieces of the position displacement information which are respective pieces of information regarding displacements in the positions of the plural cameras,

the coordinate transformation parameter obtainment unit is operable to obtain a plurality of the coordinate transformation parameters based on the plural pieces of position displacement information obtained by the position displacement information obtainment unit,

the coordinate transformation unit is operable to perform coordinate transformation on the image coordinates included in the reference image synthesis parameter obtained by the image synthesis parameter obtainment unit, using the plurality of coordinate transformation parameters obtained by the coordinate transformation parameter obtainment unit, and to output the image coordinates as the transformed image synthesis parameter, and

the image synthesis unit is operable to generate the surrounding image by applying the plurality of coordinate transformation parameters to the plurality of captured images, the coordinate transformation parameters being obtained by the coordinate transformation parameter obtainment unit.

11. The surroundings monitoring apparatus according to claim 10,

wherein the reference image synthesis parameter includes, for each pixel of the surrounding image, information which identifies a captured image from among a plurality of the captured images and information of image coordinates indicating a pixel position on the captured image.

12. The surroundings monitoring apparatus according to claim 10,

wherein the coordinate transformation parameter obtainment unit is operable to obtain a coordinate transformation parameter in which a position error of the surrounding image after the coordinate transformation is minimal in a boundary position for a plurality of the captured images, the boundary position being included in the reference image synthesis parameter.

13. The surroundings monitoring apparatus according to claim 1,

wherein the image synthesis parameter obtainment unit is operable to obtain the reference image synthesis parameter which includes information of the image coordinates out of the area of the captured image.

14. The surroundings monitoring apparatus according to claim 1,

wherein the image synthesis parameter obtainment unit is operable to obtain the reference image synthesis parameter which includes plural pieces of information of the image coordinates indicating, per pixel on the surrounding image, positions of two or more pixels which are on the captured image, and

the coordinate transformation unit is operable to select a pixel after performing the coordinate transformation by applying the coordinate transformation parameter obtained by the coordinate transformation parameter obtainment unit to the image coordinates included in the reference image synthesis parameter obtained by the image synthesis parameter obtainment unit, and to output the image coordinates as the transformed image synthesis parameter.

15. The surroundings monitoring apparatus according to claim 1,

wherein the captured image is an image including lens distortion, and

the image synthesis unit is operable to generate the surrounding image using the captured image having a lens distortion parameter in which an error is minimal in the surrounding image generated by one or more of the predetermined reference image synthesis parameters and one or more of the predetermined coordinate transformation parameters.

16. The surroundings monitoring apparatus according to claim 1, further comprising:

a state detection unit operable to detect a state of a moving object including information regarding a location of the moving object which is equipped with the surroundings monitoring apparatus,

wherein the coordinate transformation parameter obtainment unit is operable to obtain the coordinate transformation parameter based on the position displacement information obtained by the position displacement information obtainment unit and a state of the moving object detected by the state detection unit.

17. The surroundings monitoring apparatus according to claim 1, further comprising:

a receiving unit operable to receive a correction degree from a user,

wherein the position displacement information obtainment unit is operable to obtain, as the position displacement information, the correction degree received by the receiving unit.

18. The surroundings monitoring apparatus according to claim 1, further comprising:

a position displacement degree detection unit operable to detect a position displacement degree of the camera,

the position displacement information obtainment unit is operable to obtain, as the position displacement information, the position displacement degree detected by the position displacement degree detection unit.

19. A surroundings monitoring method which generates a surrounding image using an image captured by a camera, the surroundings monitoring method comprising:

a position displacement information obtainment step of obtaining position displacement information which is information regarding a displacement in a position of the camera;

a coordinate transformation parameter obtainment step of obtaining, based on the position displacement information obtained by the position displacement information obtainment step, a coordinate transformation parameter for correcting a distortion in the surrounding image caused by the displacement in the position, the coordinate transformation parameter defining a function for transforming two-dimensional coordinates of one pixel into two-dimensional coordinates of another pixel;

an image synthesis parameter obtainment step of obtaining at least one reference image synthesis parameter which is used for generating the surrounding image using the captured image, the at least one reference image synthesis parameter including information of image coordinates indicating, on the captured image, a pixel position corresponding to each pixel in the surrounding image, and the image coordinates being two-dimensional coordinates;

a coordinate transformation step of performing coordinate transformation on the two-dimensional image coordinates included in the at least one reference image synthesis parameter obtained by the image synthesis parameter obtainment step, using the coordinate transformation parameter obtained by the coordinate transformation parameter obtainment step, and of outputting the image coordinates as a transformed image synthesis parameter indicating transformed two-dimensional image coordinates; and

an image synthesis step of generating the surrounding image by applying the transformed image synthesis parameter to the captured image, and of outputting the generated surrounding image,

wherein the image synthesis step reads, from among pixels included in the captured image, an image of a pixel located in the transformed two-dimensional image coordinates indicated by the transformed image synthesis parameter, and outputs the read image as an image of a pixel located in the surrounding image and corresponding to the pixel located in the transformed two-dimensional image coordinates.

20. A non-transitory computer-readable recording medium storing a program for generating a surrounding image using an image captured by a camera, the program causing a computer to execute steps comprising:

a position displacement information obtainment step of obtaining position displacement information which is information regarding a displacement in a position of the camera;

a coordinate transformation parameter obtainment step of obtaining, based on the position displacement information obtained by the position displacement information obtainment step, a coordinate transformation parameter for correcting a distortion in the surrounding image caused by the displacement in the position, the coordinate transformation parameter defining a function for transforming two-dimensional coordinates of one pixel into two-dimensional coordinates of another pixel;

an image synthesis parameter obtainment step of obtaining at least one reference image synthesis parameter which is used for generating the surrounding image using the captured image, the at least one reference image synthesis parameter including information of image coordinates indicating, on the captured image, a pixel position corresponding to each pixel in the surrounding image, and the image coordinates being two-dimensional coordinates;

a coordinate transformation step of performing coordinate transformation on the two-dimensional image coordinates included in the at least one reference image synthesis parameter obtained by the image synthesis parameter obtainment step, using the coordinate transformation parameter obtained by the coordinate transformation parameter obtainment step, and of outputting the image coordinates as a transformed image synthesis parameter indicating transformed two-dimensional image coordinates; and

an image synthesis step of generating the surrounding image by applying the transformed image synthesis parameter to the captured image, and of outputting the generated surrounding image,

wherein the image synthesis step reads, from among pixels included in the captured image, an image of a pixel located in the transformed two-dimensional image coordinates indicated by the transformed image synthesis parameter, and outputs the read image as an image of a pixel located in the surrounding image and corresponding to the pixel located in the transformed two-dimensional image coordinates.