Method and apparatus for single-image-based rain streak removal

• US 8,774,546 B2
• Filed: 07/18/2012
• Issued: 07/08/2014
• Est. Priority Date: 03/08/2012
• Status: Active Grant

• Current Assignee: Industrial Technology Research Institute
• Original Assignee: Industrial Technology Research Institute
• Inventors: Lin, Chia-Wen, Lin, Yu-Chen, Kang, Li-Wei, Lin, Che-Tsung
• Primary Examiner(s): Do, Anh
• Application Number:
  US13/551,617
• Publication Number:
  US 20130236116A1
• Time in Patent Office: 720 Days
• Field of Search: 382/260, 382/263, 382/264, 382/254, 382/275, 382/233, 382/240, 382/131, 382/132, 375/240.25, 348/620, 378/38, 341/79, 702/189, 361/115
• US Class Current: 382/260
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 5/10 using non-spatial domain filtering
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal

Abstract

A method and an apparatus for single-image-based rain streak removal are introduced herein. In this method and apparatus, an original image is decomposed into a high frequency part and a low frequency part, and the high frequency image is then decomposed into a rain part and a non-rain part. The non-rain high frequency image and the original low frequency image are used to produce a non-rain image.

Claims

1. A single-image-based rain streak removal method executed by a processor of a single-image-based rain streak removal apparatus, comprising:

the processor decomposing an original image into an original high frequency image and an original low frequency image;

the processor obtaining a dictionary, wherein the dictionary is the best for representing a plurality of patches of a reference image and the dictionary comprises a rain sub-dictionary and a non-rain sub-dictionary;

the processor finding a coefficient of each patch of a plurality of patches of the original high frequency image corresponding to the dictionary according to a first cost function and a prerequisite;

the processor producing a non-rain high frequency image using the dictionary and the coefficient of each of the patches of the original high frequency image; and

the processor combining the original low frequency image and the non-rain high frequency image into a non-rain image.

2. The single-image-based rain streak removal method according to claim 1, further comprising:

using a bilateral filter to decompose the original image into the original high frequency image and the original low frequency image.

3. The single-image-based rain streak removal method according to claim 1, wherein the original image and the reference image belong to one same continuous video, and the reference image is prior to the original image in terms of time sequence or the reference image is the original image.

4. The single-image-based rain streak removal method according to claim 1, wherein the step of obtaining the dictionary comprises:

decomposing the reference image into a reference high frequency image and a reference low frequency image;

finding a plurality of patches from the reference high frequency image;

finding the dictionary that is the best for representing the plurality of patches of the reference high frequency image according to a second cost function;

acquiring a descriptor of a histogram of oriented gradient of each patch of the dictionary; and

separating the patches in the dictionary into the rain sub-dictionary and the non-rain sub-dictionary according to a variance of gradient direction of the descriptor of each of the patches of the dictionary.

5. The single-image-based rain streak removal method according to claim 4, further comprising:

finding the dictionary that makes the second cost function have a minimum value through online dictionary learning.

6. The single-image-based rain streak removal method according to claim 4, further comprising:

separating the patches in the dictionary into the rain sub-dictionary and the non-rain sub-dictionary using a K-means algorithm and the variance of each of the patches of the dictionary.

7. The single-image-based rain streak removal method according to claim 1, wherein the step of finding the coefficient of each patch of the original high frequency image comprises:

for each of the patches of the original high frequency image, finding the coefficient that makes the first cost function have a minimum value under the prerequisite using an orthogonal matching pursuit algorithm.

8. The single-image-based rain streak removal method according to claim 1, wherein the step of producing the non-rain high frequency image comprises:

taking a product of the dictionary and the coefficient of each of the patches of the original high frequency image approximately as the patch of the original high frequency image; and

combining non-rain portions of the plurality of patches of the original high frequency image into the non-rain high frequency image, wherein the non-rain portions are resulted from the non-rain sub-dictionary.

9. A single-image-based rain streak removal apparatus, comprising:

a processor, receiving an original image, decomposing the original image into an original high frequency image and an original low frequency image, and obtaining a dictionary, wherein the dictionary is the best for representing a plurality of patches of a reference image and the dictionary comprises a rain sub-dictionary and a non-rain sub-dictionary, the processor finds a coefficient of each of the patches of the original high frequency image corresponding to the dictionary according to a first cost function and a prerequisite, produces a non-rain high frequency image using the dictionary and the coefficient of each of the patches of the original high frequency image, and combines the original low frequency image and the non-rain high frequency image into a non-rain image.

10. The single-image-based rain streak removal apparatus according to claim 9, wherein the processor utilizes a bilateral filter to decompose the original image into the original high frequency image and the original low frequency image.

11. The single-image-based rain streak removal apparatus according to claim 9, wherein the original image and the reference image belong to one same continuous video, and the reference image is prior to the original image in terms of time sequence or the reference image is the original image.

12. The single-image-based rain streak removal apparatus according to claim 9, wherein the processor decomposes the reference image into a reference high frequency image and a reference low frequency image, finds a plurality of patches from the reference high frequency image, finds the dictionary that is the best for representing the plurality of patches of the reference high frequency image according to a second cost function, acquires a descriptor of a histogram of oriented gradient of each patch of the dictionary, and separates the patches in the dictionary into the rain sub-dictionary and the non-rain sub-dictionary according to a variance of gradient direction of the descriptor of each of the patches of the dictionary.

13. The single-image-based rain streak removal apparatus according to claim 12, wherein the processor finds the dictionary that makes the second cost function have a minimum value through online dictionary learning.

14. The single-image-based rain streak removal apparatus according to claim 12, wherein the processor separates the patches in the dictionary into the rain sub-dictionary and the non-rain sub-dictionary using a K-means algorithm and the variance of each of the patches of the dictionary.

15. The single-image-based rain streak removal apparatus according to claim 9, wherein, for each of the patches of the original high frequency image, the processor finds the coefficient that makes the first cost function have a minimum value under the prerequisite using an orthogonal matching pursuit algorithm.

16. The single-image-based rain streak removal apparatus according to claim 9, wherein the processor takes a product of the dictionary and the coefficient of each of the patches of the original high frequency image approximately as the patch of the original high frequency image, and combines non-rain portions of the plurality of patches of the original high frequency image into the non-rain high frequency image, wherein the non-rain portions are resulted from the non-rain sub-dictionary.

17. The single-image-based rain streak removal apparatus according to claim 9, further comprising an image sensor capturing the original image, wherein the processor is coupled to the image sensor.

18. The single-image-based rain streak removal apparatus according to claim 9, further comprising a rain sensor, wherein the processor is coupled to the rain sensor and determines whether to perform rain streak removal on the original image according to a sensing result of the rain sensor.