Method for in-image periodic noise pixel inpainting

• US 9,224,052 B2
• Filed: 10/28/2013
• Issued: 12/29/2015
• Est. Priority Date: 12/19/2012
• Status: Active Grant

• Current Assignee: Industrial Technology Research Institute
• Original Assignee: Industrial Technology Research Institute
• Inventors: Lin, Che-Tsung, Lin, Yu-Chen
• Primary Examiner(s): Yeh, Eueng-Nan
• Application Number:
  US14/064,706
• Publication Number:
  US 20140169695A1
• Time in Patent Office: 792 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20224 Image subtraction
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle

Abstract

A method for in-image periodic noise pixel inpainting is provided. It is determined whether a current frame includes periodic noise pixels, and locations of periodic noise pixels are identified. Non-periodic-noise pixels in a reference frame are utilized to inpaint the periodic noise pixels in the current frame.

Claims

1. A method for in-image periodic noise pixel inpainting, comprising:

determining whether a current frame comprises a periodic noise pixel group, and identifying locations of pixels of the periodic noise pixel group in the current frame if the current frame comprises the periodic noise pixel group;

matching a plurality of non-periodic-noise pixels in the current frame with a plurality of non-periodic-noise pixels in a reference frame to obtain a pixel relationship between the current frame and the reference frame; and

selecting corresponding pixels from the reference frame to inpaint the periodic noise pixel group in the current frame according to the locations of the pixels of the periodic noise pixel group in the current frame and the pixel relationship;

wherein the pixel relationship between the current frame and the reference frame is modeled by an affine transformation matrix, a projective transformation matrix, or a transformation matrix or a transformation formula which describes the pixel relationship between two frames;

a plurality of non-periodic-noise feature correspondences in the reference frame, a plurality of non-periodic-noise feature correspondences in the current frame, and a pixel relationship between the non-periodic-noise feature points in the reference frame and the non-periodic-noise feature points in the current frame are identified by a scale-invariant feature transform (SIFT) algorithm or a speeded up robust features (SURF) algorithm; and

a plurality of feature correspondences of the reference frame and the current frame is substituted by a random sample consensus (RANSAS) algorithm or a least-squares method to obtain the affine transformation matrix.

2. The method for in-image periodic noise pixel inpainting according to claim 1, wherein the current frame and the reference frame are captured by an image sensor, which is an infrared image capturing device, a photoelectric coupling element, and/or a complementary metal-oxide semiconductor (CMOS) photosensitive element.

3. The method for in-image periodic noise pixel inpainting according to claim 1, wherein the periodic noise pixel group includes a plurality of wiper pixels.

4. The method for in-image periodic noise pixel inpainting according to claim 1, wherein an image segmentation step determines whether the current frame comprises the periodic noise pixel group and identifies the locations of the pixels of the periodic noise pixel group.

5. The method for in-image periodic noise pixel inpainting according to claim 4, wherein the image segmentation step comprises:

dividing all pixels of the current frame into a plurality of groups, and regarding at least one dark pixel group as a possible periodic noise pixel group;

eliminating at least one incomplete dark pixel group in the current frame, and shaping the appearance of a complete dark pixel group;

binarizing and down-sampling the current frame to obtain a binarized average for obtaining a vector;

obtaining an inner product from the vector and a pre-trained eigen matrix; and

determining whether the current frame comprises the periodic noise pixel group according to the inner product.

6. The method for in-image periodic noise pixel inpainting according to claim 4, wherein the image segmentation step comprises:

subtracting the current frame from a background image, wherein the background image does not contain any periodic noise pixel group;

filtering out a pixel having an absolute value of a pixel difference smaller than a threshold from the current frame, wherein the pixel difference is a difference between a pixel in the current frame and the pixel at the same location in the background image;

filtering out any connected components having an area smaller than an area threshold, wherein each connected component is formed by connecting a plurality of unfiltered out pixels;

binarizing and down-sampling the current frame to obtain a binarized average for obtaining a vector;

obtaining an inner product from the vector and a pre-trained eigen matrix; and

determining whether the current frame comprises the periodic noise pixel group according to the inner product.

7. The method for in-image periodic noise pixel inpainting according to claim 6, further comprising:

subtracting the current frame from a previous frame; and

updating the current frame as the background image if the area of a largest connected component is smaller than the area threshold and an accumulation number is greater than a number threshold, wherein if the area of the largest connected component is smaller than the area threshold, the accumulation number is added by 1.

8. The method for in-image periodic noise pixel inpainting according to claim 1, wherein the periodic noise pixel group of the current frame is substituted into the affine transformation matrix to obtain locations of a plurality of corresponding pixels of the reference frame for inpainting.

9. The method for in-image periodic noise pixel inpainting according to claim 1, wherein the reference frame is earlier than the current frame in a real-time operation application, and is earlier or later than the current frame in a non-real-time operation application.

10. The method for in-image periodic noise pixel inpainting according to claim 1, which is applicable to be deployed to a computing device, an embedded system, a smart handset or a personal computer which executes one or more program instructions.

11. The method for in-image periodic noise pixel inpainting according to claim 1, further comprising:

receiving a wiper signal to determine whether to initiate the method for in-image periodic noise pixel inpainting, wherein the periodic noise pixel group is a wiper pixel group.

12. The method for in-image periodic noise pixel inpainting according to claim 11, wherein it is determined whether a wiper is operating according to the wiper signal;

and the wiper is activated according to a wiper activation event or according to whether it is raining as determined by a raindrop sensor.

13. The method for in-image periodic noise pixel inpainting according to claim 12, wherein the raindrop sensor is an optical reflective sensor or an audio sensor or a conductive sensor, or any combination thereof.