Method and system for removal of rain streak distortion from a video

• US 10,282,827 B2
• Filed: 09/22/2017
• Issued: 05/07/2019
• Est. Priority Date: 08/10/2017
• Status: Active Grant

• Current Assignee: Wipro Limited
• Original Assignee: Wipro Limited
• Inventors: Ulaganathan, Sethuraman, Ramachandra, Manjunath, Hegde, Prasanna
• Primary Examiner(s): Thomas, Mia M
• Application Number:
  US15/712,838
• Publication Number:
  US 20190050969A1
• Time in Patent Office: 592 Days
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30192 Weather; Meteorology
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/60 using machine learning, e.g. neural networks
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06T 7/001 using an image reference approach

Abstract

Systems and methods for removing rain streak distortion from a distorted video are described. The system receives sample non-distorted images and sample distorted images of a video. The sample non-distorted images are indicative of non-raining condition and the sample distorted images are indicative of raining condition in the video. The system further determines first temporal information from the sample distorted images and second temporal information from the sample non-distorted images. The first temporal information indicative of a change in the rain streak distortion pattern and the second temporal information indicative of a change in a non-rain streak distortion pattern. Further, the system correlates the first temporal information with the second temporal information to generate a training model comprising one or more trained weights. Further, the system removes the rain streak distortion from a real-time distorted video by applying the training model, which results in the generation of the non-distorted video.

Claims

1. A method of removing rain streak distortion from a distorted video, the method comprising:

receiving, by a model generator, a plurality of sample non-distorted images and a plurality of sample distorted images of a video, wherein the plurality of sample non-distorted images are indicative of a non-raining condition in the video, and wherein the plurality of sample distorted images are indicative of a raining condition in the video;

determining, by the model generator, a first temporal information from the plurality of sample distorted images and a second temporal information from the plurality of sample non-distorted images, wherein the first temporal information, indicative of a change in a rain streak distortion pattern, comprises a plurality of first set of pixel values corresponding to the plurality of sample distorted images, and wherein the second temporal information, indicative of a change in a non-rain streak distortion pattern, comprises a plurality of second set of pixel values corresponding to the plurality of sample non-distorted images;

correlating, by the model generator, the first temporal information with the second temporal information based on the plurality of first set of pixel values and the plurality of second set of pixel values;

generating, by the model generator, a training model comprising one or more trained weights based on the correlation; and

removing, by a video convertor, the rain streak distortion from a real-time distorted video by applying the training model, wherein removal of the rain streak distortion results in the generation of the non-distorted video.

2. The method as claimed in claim 1, wherein the removing of the rain streak distortion from the real-time distorted video comprises:

receiving, by a video capturing unit, the real-time distorted video comprising the rain streak distortion,

converting, by real-time image sequence generator, the real-time distorted video into a plurality of real-time distorted images;

determining, by an image generating unit, real-time rain streak distortion pattern in the plurality of real-time distorted images by applying the one or more trained weights on the plurality of real-time distorted images;

generating, by the image generating unit, a plurality of real-time non-distorted images corresponding to the plurality of real-time distorted images by removing the real-time rain streak distortion pattern; and

converting, by the video converter, the plurality of real-time non-distorted images into the non-distorted video.

3. The method as claimed in claim 1, wherein the correlating of the first temporal information and the second temporal information comprises:

comparing the plurality of first set of pixel values with the plurality of second set of pixel values to determine a difference between pixel values;

generating a plurality of reconstructed images based on the comparing; and

comparing the plurality of reconstructed images with the plurality of sample non-distorted images to determine correlation factor indicating a similarity error between the plurality of reconstructed images and the plurality of sample non-distorted images, wherein the correlation factor is minimized by applying the one or more trained weights such that the plurality of reconstructed images becomes similar to the plurality of sample non-distorted images.

4. The method as claimed in claim 1, wherein the plurality of first set of pixel values, of the first temporal information, indicates change in rain streak distortion pattern across the plurality of sample distorted images over a time interval.

5. The method as claimed in claim 1, wherein the plurality of second set of pixel values, of the second temporal information, indicates a change in non-rain streak distortion pattern across the plurality of sample non-distorted images over a time interval.

6. The method as claimed in claim 1, wherein the training model learns, from the plurality of sample distorted images and the plurality of sample non-distorted images, about the change in the rain streak distortion pattern and the change in the non-rain streak distortion pattern respectively.

7. A system for removing rain streak distortion from a distorted video, wherein the system comprises:

a processor; and

a memory communicatively coupled to the processor, wherein the memory stores processor instructions, which, on execution, causes the processor to;

receive a plurality of sample non-distorted images and a plurality of sample distorted images of a video, wherein the plurality of sample non-distorted images are indicative of a non-raining condition in the video, and wherein the plurality of sample distorted images are indicative of a raining condition in the video,

determine a first temporal information from the plurality of sample distorted images and a second temporal information from the plurality of sample non-distorted images, wherein the first temporal information, indicative of a change in a rain streak distortion pattern, comprises a plurality of first set of pixel values corresponding to the plurality of sample distorted images, and wherein the second temporal information, indicative of a change in a non-rain streak distortion pattern, comprises a plurality of second set of pixel values corresponding to the plurality of sample non-distorted images,

correlate the first temporal information with the second temporal information based on the plurality of first set of pixel values and the plurality of second set of pixel values, and

generate a training model comprising one or more trained weights based on the correlation; and

remove the rain streak distortion from a real-time distorted video by applying the training model, wherein the removal of the rain streak distortion results in the generation of the non-distorted video.

9. The system as claimed in claim 7, wherein the system performs correlating of the first temporal information and the second temporal information by:

comparing the plurality of first set of pixel values with the plurality of second set of pixel values to determine a difference between pixel values;

generating a plurality of reconstructed images based on the comparing, and

comparing the plurality of reconstructed images with the plurality of sample non-distorted images to determine a similarity error between the plurality of reconstructed images and the plurality of sample non-distorted images, wherein the correlation factor is minimized by applying the one or more trained weights such that the plurality of reconstructed images becomes similar to the plurality of sample non-distorted images.

10. The system as claimed in claim 7, wherein the plurality of first set of pixel values, of the first temporal information, indicates change in rain streak distortion pattern across the plurality of sample distorted images over a time interval.

11. The system as claimed in claim 7, wherein the wherein the plurality of second set of pixel values, of the second temporal information, indicates a change in non-rain streak distortion pattern across the plurality of sample non-distorted images over a time interval.

12. The system as claimed in claim 7, wherein the training model learns, from the plurality of sample distorted images and the plurality of sample non-distorted images, about the change in the rain streak distortion pattern and the change in the non-rain streak distortion pattern respectively.

8. The system as claimed in claimed 7, wherein the system removes the rain streak distortion from the real-time distorted video by:

receiving the real-time distorted video comprising the rain streak distortion;

converting the real-time distorted video into a plurality of real-time distorted images;

determining real-time rain streak distortion pattern in the plurality of real-time distorted images by applying the one or more trained weights on the plurality of real-time distorted images;

generating a plurality of real-time non-distorted images corresponding to the plurality of real-time distorted images by removing the real-time rain streak distortion pattern; and

converting the plurality of real-time non-distorted images into the non-distorted video.

13. A non-transitory computer-readable storage medium including instructions stored thereon that when processed by at least one processor cause a system to perform operations comprising:

receiving a plurality of sample non-distorted images and a plurality of sample distorted images of a video, wherein the plurality of sample non-distorted images are indicative of a non-raining condition in the video, and wherein the plurality of sample distorted images are indicative of a raining condition in the video;

determining a first temporal information from the plurality of sample distorted images and a second temporal information from the plurality of sample non-distorted images, wherein the first temporal information, indicative of a change in a rain streak distortion pattern, comprises a plurality of first set of pixel values corresponding to the plurality of sample distorted images, and wherein the second temporal information, indicative of a change in a non-rain streak distortion pattern, comprises a plurality of second set of pixel values corresponding to the plurality of sample non-distorted images;

correlating the first temporal information with the second temporal information based on the plurality of first set of pixel values and the plurality of second set of pixel values;

generating a training model comprising one or more trained weights based on the correlation; and

removing the rain streak distortion from a real-time distorted video by applying the training model, wherein removal of the rain streak distortion results in the generation of the non-distorted video.

14. The medium as claimed in claim 13, wherein the removing of the rain streak distortion from the real-time distorted video comprises:

receiving the real-time distorted video comprising the rain streak distortion;

converting the real-time distorted video into a plurality of real-time distorted images;

determining real-time rain streak distortion pattern in the plurality of real-time distorted images by applying the one or more trained weights on the plurality of real-time distorted images;

generating a plurality of real-time non-distorted images corresponding to the plurality of real-time distorted images by removing the real-time rain streak distortion pattern; and

converting the plurality of real-time non-distorted images into the non-distorted video.

15. The medium as claimed in claim 13, wherein the correlating of the first temporal information and the second temporal information comprises:

comparing the plurality of first set of pixel values with the plurality of second set of pixel values to determine a difference between pixel values;

generating a plurality of reconstructed images based on the comparing; and

comparing the plurality of reconstructed images with the plurality of sample non-distorted images to determine correlation factor indicating a similarity error between the plurality of reconstructed images and the plurality of sample non-distorted images, wherein the correlation factor is minimized by applying the one or more trained weights such that the plurality of reconstructed images becomes similar to the plurality of sample non-distorted images.

16. The medium as claimed in claim 13, wherein the plurality of first set of pixel values, of the first temporal information, indicates change in rain streak distortion pattern across the plurality of sample distorted images over a time interval.

17. The medium as claimed in claim 13, wherein the plurality of second set of pixel values, of the second temporal information, indicates a change in non-rain streak distortion pattern across the plurality of sample non-distorted images over a time interval.

18. The medium as claimed in claim 13, wherein the training model learns, from the plurality of sample distorted images and the plurality of sample non-distorted images, about the change in the rain streak distortion pattern and the change in the non-rain streak distortion pattern respectively.