Image inpainting system and method for using the same

• US 10,037,597 B2
• Filed: 12/29/2015
• Issued: 07/31/2018
• Est. Priority Date: 11/06/2015
• Status: Active Grant

• Current Assignee: Industrial Technology Research Institute
• Original Assignee: Industrial Technology Research Institute
• Inventors: Chan, Sheng-Wei, Lin, Che-Tsung
• Primary Examiner(s): Beasley, Deirdre
• Application Number:
  US14/982,775
• Publication Number:
  US 20170132764A1
• Time in Patent Office: 945 Days
• Field of Search: 348148
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20224 Image subtraction
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: H04N 23/57 Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
• CPC Class Codes: H04N 23/81 for suppressing or minimising disturbance in the image signal generation
• CPC Class Codes: H04N 23/811 by dust removal, e.g. from surfaces of the image sensor or processing of the image signal output by the electronic image sensor
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

An image inpainting method includes the following steps: segmenting image, acquiring a plurality of images, and having the plurality of images segment into noise-contained pixel images and non-noise-contained pixel images, and confirming the positions of every noise pixel of the noise pixel image; and performing inpainting in light of the noise-contained pixel images, finding out the offset map and geometric relationship of the pixel corresponding relationship without being subjected to the affection of noise and having the pixel corresponding relationship with minimum parallax, making use of the offset map or the geometric relationship to extract corresponding pixel that is not subjected to the affection of noise, performing inpainting and substituting the noise pixel in the plurality of images to generate at least a synthetic image without containing noise.

Claims

1. An image inpainting system, comprising:

a camera module for acquiring a plurality of images; and

an inpainting module coupled to the camera module for receiving the plurality of images from the camera module and having the plurality of images perform a noise detection where the plurality of images are segmented into noise-contained pixel images and non-noise-contained pixel images, confirming the positions of every noise pixel of the noise pixel images, finding out the offset map and geometric relationship of the pixel corresponding relationship without being subjected to the affection of noise and having the pixel corresponding relationship with minimum parallax, making use of the offset map or the geometric relationship to extract corresponding pixel that is not subjected to the affection of noise, performing inpainting and substituting the noise pixel in the plurality of images to generate at least a synthetic image without containing noise;

wherein the image inpainting module is configured to make use of the offset map and inpainting image by;

selecting an inpainting image pair, selecting the image having no noise and having minimum parallax to perform pairing with the noise-contained image by choosing the already-known camera'"'"'s relative relationship, and this pairing is an inpainting image pair;

randomly generating an offset map, generating an initial random offset map of an inpainting image pair by the use of an Independent Uniform Sampling method or any random method capable of performing randomly sampling pixel point in the reference image to generate an initial random offset map of the inpainting image pair;

judging the n times iteration, judging if the inpainting images is performed first-time iteration, if not, proceed to the next Step, with n as a constant, the number of iteration is progressed until n times;

judging if the inpainting image pair from the next Step perform n times of iteration, if yes, proceed to Step of inpainting the noise-contained images by a new offset map;

judging if there is no not-updated patch in the inpainting image pair in the process of inpainting the current iteration, If yes, then the inpainting image pair from the previous Step has already performed the current iteration, and updating the patch and proceed back to the previous Step, if the inpainting image pair from the previous Step has no current iteration and no updating patch is performed, then proceed to the next Step;

in light of the current patch, selecting the minimum value from the offset amount of the adjacent patch and the current patch to overwrite the offset amount of the current patch;

in light of the current patch and in a preset radius, randomly searching for a better offset amount until the radius attenuate to 1 pixel, and proceed back to the step of judging if there is no not-updated patch in the inpainting image pair in the process of inpainting the current iteration is performed in the inpainting image pair; and

inpainting the noise-contained images by a new offset map.

2. The image inpainting system as claimed in claim 1, wherein the camera module possesses at least two camera units that are arranged in array mode.

3. The image inpainting system as claimed in claim 2, wherein the camera unit is an image sensor, an infrared image acquisition device, optical sensing element of either an optoelectronic coupling element or a complementary metal-oxide semiconductor, or at least a combination of both.

4. The image inpainting system as claimed in claim 1, wherein the inpainting module has a receiving unit, an operating unit and a storage unit wherein the receiving unit is signally connected to the camera module, the operating unit is signally connected to the receiving unit while the storage unit is signally connected to the operating unit.

5. The image inpainting system as claimed in claim 1, wherein the inpainting module has the plurality of images perform a noise pixel detection, and the noise pixel is one of the wiper pixel, raindrop pixel or dirty pixel or at least a combination of the arbitrary two of them.

6. An image inpainting method, comprising the following steps:

segmenting image, acquiring a plurality of images, and having the plurality of images segmented into noise-contained pixel images and non-noise-contained pixel images, and confirming the positions of every noise pixel of the noise pixel image;

performing inpainting in light of the noise-contained pixel images, finding out an offset map and geometric relationship of the pixel corresponding relationship without being subjected to the affection of noise and having the pixel corresponding relationship with minimum parallax, making use of the offset map or the geometric relationship to extract corresponding pixel that is not subjected to the affection of noise, performing inpainting and substituting the noise pixel in the plurality of images to generate at least a synthetic image without containing noise;

selecting an inpainting image pair, selecting the image having no noise and having minimum parallax to perform pairing with the noise-contained image by choosing the already-known camera'"'"'s relative relationship, and this pairing is an inpainting image pair;

randomly generating an offset map, generating an initial random offset map of an inpainting image pair by the use of an Independent Uniform Sampling method or any random method capable of performing randomly sampling pixel point in the reference image to generate an initial random offset map of the inpainting image pair;

judging the n times iteration, judging if the inpainting images is performed first-time iteration, if not, proceed to the next Step, with n as a constant, the number of iteration is progressed until n times;

judging if the inpainting image pair from the next Step perform n times of iteration, if yes, proceed to Step of inpainting the noise-contained images by a new offset map;

judging if there is no not-updated patch in the inpainting image pair in the process of inpainting the current iteration, If yes, then the inpainting image pair from the previous Step has already performed the current iteration, and updating the patch and proceed back to the previous Step, if the inpainting image pair from the previous Step has no current iteration and no updating patch is performed, then proceed to the next Step;

in light of the current patch, selecting the minimum value from the offset amount of the adjacent patch and the current patch to overwrite the offset amount of the current patch;

in light of the current patch and in a preset radius, randomly searching for a better offset amount until the radius attenuate to 1 pixel, and proceed back to the step of judging if there is no not-updated patch in the inpainting image pair in the process of inpainting the current iteration is performed in the inpainting image pair; and

inpainting the noise-contained images by a new offset map.

7. The image inpainting method as claimed in claim 6, wherein the noise pixel is one of the wiper pixel, raindrop pixel or dirty pixel or at least a combination of the arbitrary two of them.

8. The image inpainting method as claimed in claim 6, wherein the step of Segmenting image contains a dirty image segmentation method that further comprising the following steps:

extracting single color image, having the single color image of the plurality of images extract out;

smoothing the single color image, having the single color image perform smoothing by a Median Filter;

forming B-spline plot, having the smoothed single color image form a B-spline plot;

performing image subtraction, subtracting the single color image after being smoothed from the B-spline plot to obtain at least a pixel plot occupied by the dirty noise in the primary image;

performing Histogram Equalization, Employing Histogram Equalization to enhance the contrast of the occupied pixel plot of the primary image dirty noise;

inpainting the pixel group, having the pixel plot occupied by the primary image dirty noise perform correction by the way of erosion or dilation; and

differentiating the dirty area by the area threshold value, having the pixel plot occupied by the corrected primary image dirty noise perform binarization to segment images.

9. The image inpainting method as claimed in claim 8, wherein the single color image is a green color image, a red color image or a blue color image.

10. The image inpainting method as claimed in claim 8, wherein in the step of differentiating the dirty area by the area threshold value, if the binarization is 1 for all the bright pixel groups in the pixel plot occupied by the primary image dirty noise, it is the possible dirty pixel group while for the pixel group without containing dirty noise, the binarization is 0, by mean of this, the dirty area in the image is differentiated for segmenting images.

11. The image inpainting method as claimed in claim 8, wherein in the step of inpainting the pixel group, if the bright pixel in the pixel plot occupied by the primary image dirty noise is relatively integrated, ten have the appearance of the bright pixel group to be modified to become more integrated, but if the bright pixel group is relatively broken, then eliminate the broken pixel group.

12. The image inpainting method as claimed in claim 8, wherein in the step of Histogram Equalization, the bright pixel group occupied by the primary image dirty noise after being enhanced is considered to be a possible ingredient of the dirty noise.

13. The image inpainting method as claimed in claim 6, wherein the step of Segmenting image being containing a raindrop noise image segmentation method further comprising the following steps:

judging if the image acquired is in daytime or at night, if it is at night, performing the step of filtering out the interference light source, and if it is in the daytime, then execute step of searching for the edge;

searching for the edge, having the above-mentioned results of the step of judging if the acquired image is at night or in the day time and of the step of filtering out the interference light source, highlighting the edge of the raindrop noise by a Sobel Filter;

differentiating the dirty area by area threshold value, having the result of step of searching for the edge perform binarization; and

performing dilation process, having the raindrop center pixel of the raindrop pixel group change from black to white to integrate the raindrop pixel group.

14. The image inpainting method as claimed in claim 13, wherein in the above-mentioned inpainting method, the step of differentiating the dirty area by the area threshold value, the binarization is 1 in light of all possible raindrop pixel group while the binarization is 0 in light of the raindrop pixel group considered not to contain raindrop, by mean of this, the raindrop area is differentiated to segment images.

15. The image inpainting method as claimed in claim 13, wherein in the step of filtering the interference light source, the filter is a high flux filter, the interference light source is a headlight, a street light, an advertising light or the light source from the buildings.

16. The image inpainting method as claimed in claim 6, wherein the step of Segmenting image contains a raindrop noise image segmentation method which further comprises the following steps:

having the dark pixel group consider to be as one of the wiper objects;

inpainting the appearance of the dark pixel group, having the result of performing the step of considering the dark pixel group to be one of the wiper object eliminate the relatively broken dark pixel group in the current images by the use of the operation methods of erosion, dilation etc., and having the appearance of relatively integrated dark pixel group perform inpainting to make it become relatively integrated;

differentiating the dirty area by the area threshold value, having the result of performing the step of inpainting the appearance of the dark pixel group perform binarization to obtain a vector;

gaining an inner product value, the inner product (dot product) value is gained from the vector and Eigen Wiper Matrix obtained from the process of “

differentiating the dirty area by the area threshold value”

; and

finding out the training information and wiper images having maximum inner product (dot product) value.

17. The image inpainting method as claimed in claim 16, wherein in the step of having the dark pixel group consider to be one of the wiper object, if K is a natural number, having all the pixels of the current images divide into K groups and having the dark pixel group consider to be one of the possible ingredient of the wiper object, the dark pixel group is a pixel group that possesses minimum three-primary-color signals.

18. The image inpainting method as claimed in claim 16, wherein in the step of differentiating the dirty area by the area threshold value, the binarization is 1 in light of all possible wiper pixel group while the binarization is 0 in light of the pixel group considered not to contain wiper object, then perform a down-sampling process to obtain the vector.

19. The image inpainting method as claimed in claim 6, further comprising a step of judging if each of the current images contain noise pixel, if the amount of each of the noise pixel of all the current images is less than a threshold value, then a judgment is made that the current image does not contain noise.

20. The image inpainting method as claimed in claim 18, further comprising a step of judging if each piece of all the current noise-contained images are subjected to inpainting processing, if not, performing the inpainting in light of the noise-contained images;

if yes, no inpainting processing is performed;

also in the step of judging if each of current images contains noise pixel, if the amount of the noise images in a certain current image is greater than a preset threshold value, then proceed to the step of judgment that each piece of all the current noise-contained images are subjected to inpainting processing.

21. The image inpainting method as claimed in claim 6, wherein before performing the step of segmenting the image, as the wiper is turned on, an inpainting module receives a starting signal to perform the step of segmenting the image.