IMAGE PROCESSING APPARATUS AND IMAGE PROCESSING METHOD

• US 20150139499A1
• Filed: 01/20/2015
• Published: 05/21/2015
• Est. Priority Date: 08/30/2012
• Status: Active Grant

• Current Assignee: Fujitsu Limited
• Original Assignee: Fujitsu Limited
• Inventors: Shimizu, Seiya
• Granted Patent: US 9,436,986 B2
• Time in Patent Office: Days
• US Class Current: 382/104
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/20208 High dynamic range [HDR] image processing
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/92 based on global image properties
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle

Abstract

An image processing apparatus that obtains a synthesized image of a subject viewed from an arbitrary viewpoint includes a calculator, a storage, and a synthesizer. The calculator calculates a correction value for a pixel value of a pixel that partially constitutes the picked-up images on the basis of a brightness of each of the plurality of picked-up images. The storage stores the correction value for each of the pixels. The synthesizer obtains a pixel value of a pixel in the synthesized image that corresponds to the pixel that partially constitutes the picked-up images by correcting a pixel value of the pixel that partially constitutes the picked-up images on the basis of a positional relationship of the pixel in the picked-up images, a correspondence relationship between the picked-up images and the synthesized image, and the correction value, and thereby generates the synthesized image.

Claims

1. An image processing apparatus that generates a synthesized image of a subject viewed from an arbitrary viewpoint from a plurality of picked-up images obtained by picking up images of surroundings of the subject in different directions, the image processing apparatus comprising

a computer which includes a processor that performs a process including:

a calculation process that calculates a correction value for a pixel value of a pixel that partially constitutes the picked-up images on the basis of a brightness of each of the plurality of picked-up images;

a storage process that stores to a storage of the computer the correction value calculated by the calculation process for each of the pixels; and

a synthesization process that corrects a pixel value of the pixel that partially constitutes the picked-up images on the basis of a positional relationship of the pixel in the picked-up images, a correspondence relationship between the picked-up images and the synthesized image, and the correction value stored in the storage to obtain a pixel value of a pixel in the synthesized image that corresponds to the pixel that partially constitutes the picked-up images, and thereby generates the synthesized image.

2. The image processing apparatus according to claim 1, wherein

the calculation process calculates a correction value for a pixel value of a pixel positioned at a boundary between two picked-up images that will be adjacent when the synthesization process generates the synthesized image.

3. The image processing apparatus according to claim 2, wherein

the calculation process calculates, as the correction value, a value by which brightness of the pixel positioned at the boundary between the two picked-up images is changed to an average value between brightnesses of the two picked-up images.

4. The image processing apparatus according to claim 2, wherein

the synthesized image is an image projected on a three-dimensional projection plane that is polygonal approximated by a plurality of polygons defined in a three-dimensional space, and

the calculation process further performs calculation of the correction value for a pixel in the picked-up images that will be positioned at a vertex of the polygon when the synthesized image is generated, by performing liner interpolation of the correction value for the pixel positioned at the boundary based on a positional relationship of the pixel with the pixel positioned at the boundary in the three-dimensional space.

5. The image processing apparatus according to claim 4, wherein

the polygon is quadrangular,

the image processing apparatus further includes a vertex data table in which coordinate values of four vertexes of the polygon in the three-dimensional space and coordinate values in the picked-up images of four points, in the picked-up images, that correspond to the four vertexes are stored in an associated manner, and

the synthesization process corrects a pixel value of a pixel positioned at an intersection point between a straight line that internally divides two sides of one of two pairs of opposite sides at a first ratio in a quadrangle including as vertexes the four corresponding points in the picked-up images and a straight line that internally divides two sides of the other of the two pairs at a second ratio by using a weighting average value in accordance with distances between the intersection point and the corresponding points for the correction value of each of the four corresponding points stored in the storage so as to obtain a pixel value of a pixel that will be positioned at an intersection point between a straight line that internally divides two sides of one of two pairs of opposite sides in the polygon at the first ratio and a straight line that internally divides two sides of the other of the two pairs at the second ratio, and thereby generates the synthesized image.

6. An image processing method by which a synthesized image of a subject viewed from an arbitrary viewpoint is generated from a plurality of picked-up images obtained by picking up images of surroundings of the subject in different directions, the image processing method comprising:

a processor, which included in a computer, that

calculates a correction value for a pixel value of a pixel that partially constitutes the picked-up images on the basis of a brightness of each of the plurality of picked-up images and storing the correction value in a storage of the computer for each of the pixels; and

corrects a pixel value of the pixel that partially constitutes the picked-up images on the basis of a positional relationship of the pixel in the picked-up images, a correspondence relationship between the picked-up images and the synthesized image and the correction value stored in the storage, obtaining a pixel value of a pixel in the synthesized image that corresponds to the pixel that partially constitutes the picked-up images, and thereby generates the synthesized image.

7. The image processing method according to claim 6, wherein

in calculation of the correction value, a correction value is calculated for a pixel value of a pixel positioned at a boundary between two picked-up images that will be adjacent when the synthesized image is generated.

8. The image processing method according to claim 7, wherein

in calculation of the correction value, a value is calculated, as the correction value, by which brightnesses the pixels positioned at the boundary between the two picked-up images are changed to an average value between brightnesses of the two picked-up images.

9. The image processing method according to claim 7, wherein

the synthesized image is an image projected on a three-dimensional projection plane that is polygonal approximated by a plurality of polygons defined in a three-dimensional space, and

in calculation of the correction value, calculation of the correction value is further performed for a pixel in the picked-up images that will be positioned at a vertex of the polygon when the synthesized image is generated, by performing liner interpolation of the correction value for the pixel positioned at the boundary based on a positional relationship of the pixel with the pixel positioned at the boundary in the three-dimensional space.

10. The image processing method according to claim 9, wherein

the polygon is quadrangular,

a correspondence relationship is obtained in advance between coordinate values of four vertexes of the polygon in the three-dimensional space and coordinate values in the picked-up images of four points, in the picked-up images, that correspond to the four vertexes, and

in the generating of the synthesized image, a pixel value of a pixel positioned at an intersection point between a straight line that internally divides two sides of one of two pairs of opposite sides at a first ratio in a quadrangle including as vertexes the four corresponding points in the picked-up images and a straight line that internally divides two sides of the other of the two pairs at a second ratio is corrected by using a weighting average value in accordance with distances between the intersection point and the corresponding points for the correction value of each of the four corresponding points stored in the storage so as to obtain a pixel value of a pixel that will be positioned at an intersection point between a straight line that internally divides two sides of one of two pairs of opposite sides in the polygon at the first ratio and a straight line that internally divides two sides of the other of the two pairs at the second ratio, and thereby the synthesized image is obtained.

11. A computer-readable, non-transitory storage medium storing a program for causing a computer to execute a process of generating a synthesized image of a subject viewed from an arbitrary viewpoint from a plurality of picked-up images obtained by picking up images of surroundings of the subject in different directions, the process comprising:

calculating a correction value for a pixel value of a pixel that partially constitutes the picked-up images on the basis of a brightness of each of the plurality of picked-up images and storing the correction value in a storage of the computer for each of the pixels; and

correcting a pixel value of the pixel that partially constitutes the picked-up images on the basis of a positional relationship of the pixel in the picked-up images, a correspondence relationship between the picked-up images and the synthesized image and the correction value stored in the storage, obtaining a pixel value of a pixel in the synthesized image that corresponds to the pixel that partially constitutes the picked-up images, and thereby generating the synthesized image.

12. The medium according to claim 11, wherein

in calculation of the correction value, a correction value is calculated for a pixel value of a pixel positioned at a boundary between two picked-up images that will be adjacent when the synthesized image is generated.

13. The medium according to claim 12, wherein

in calculation of the correction value, a value is calculated, as the correction value, by which brightness of the pixel positioned at the boundary between the two picked-up images is changed to an average value between brightnesses of the two picked-up images.

14. The medium according to claim 12, wherein

the synthesized image is an image projected on a three-dimensional projection plane that is polygonal approximated by a plurality of polygons defined in a three-dimensional space, and

in calculation of the correction value, calculation of the correction value is further performed for a pixel in the picked-up images that will be positioned at a vertex of the polygon when the synthesized image is generated, by performing liner interpolation of the correction value for the pixel positioned at the boundary based on a positional relationship of the pixel with the pixel positioned at the boundary in the three-dimensional space.

15. The medium according to claim 14, wherein

the polygon is quadrangular,

a vertex data table in which coordinate values of four vertexes of the polygon in the three-dimensional space and coordinate values in the picked-up images of four points, in the picked-up images, that correspond to the four vertexes are stored in an associated manner is stored in the computer in advance, and

in the generating of the synthesized image, a pixel value of a pixel positioned at an intersection point between a straight line that internally divides two sides of one of two pairs of opposite sides at a first ratio in a quadrangle including as vertexes the four corresponding points in the picked-up images and a straight line that internally divides two sides of the other of the two pairs at a second ratio is corrected by using a weighting average value in accordance with distances between the intersection point and the corresponding points for the correction value of each of the four corresponding points stored in the storage so as to obtain a pixel value of a pixel that will be positioned at an intersection point between a straight line that internally divides two sides of one of two pairs of opposite sides in the polygon at the first ratio and a straight line that internally divides two sides of the other of the two pairs at the second ratio, and thereby the synthesized image is obtained.