Removal of shadows from images in a video signal

• US 8,761,497 B2
• Filed: 12/10/2009
• Issued: 06/24/2014
• Est. Priority Date: 12/24/2008
• Status: Active Grant

• Current Assignee: Rafael Advanced Defense Systems Limited (Government of Israel)
• Original Assignee: Rafael Advanced Defense Systems Limited (Government of Israel)
• Inventors: Berkovich, Erez, Peles, David
• Primary Examiner(s): Ahmed, Samir
• Assistant Examiner(s): LE, TOTAM HA
• Application Number:
  US13/141,751
• Publication Number:
  US 20110273620A1
• Time in Patent Office: 1,657 Days
• Field of Search: 382/155
• US Class Current: 382/155
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/94 based on local image properties, e.g. for local contrast enhancement

Abstract

Substantially removing shadows in video images obtained from a camera viewing a scene, at variable pointing angles and magnification, by digital processing of a sequence of frames in the video signal, the processing including—(a) creating and maintaining a model image of the scene, by accumulating image data from a plurality of video frames; (b) in the model image, detecting and defining model shadow zones; (c) calculating correction factors for the model shadow zones; (d) for each frame of the video signal—defining, in the image carried by the signal, shadow zones that correspond to respective model shadow zones; and (e) correcting video signal values in shadow zones accordingly.

Claims

1. A method for detecting and removing shadows, the method comprising the steps of:

(i) combining corresponding values from a plurality of frames that are carried by a digital video signal, obtained, directly or indirectly, from a video camera, any two frames of the video signal differing from each other owing to the field of view of the video camera changing by any camera action selected from tilting, panning, zooming or dollying or any combination thereof, said corresponding values being combined so as to create therefrom a background model image of the scene or of a portion thereof, said background model image covering a region greater than a field of view of at least one individual frame of the video signal;

(ii) processing the background model image to detect and define within the background model image model shadow zones;

(iii) calculating correction factors for some or all of said model shadow zones;

(iv) for each of a plurality of frames of the video signal, determining a region of said background model image corresponding to the field of view of the frame, and thereby deriving, in the frame carried by the signal, frame shadow zones that correspond to said model shadow zones; and

(v) correcting video signal values within said frame shadow zones.

2. The method of claim 1, wherein said processing includes segmentation of pixel values.

3. The method of claim 2, wherein said segmentation is in terms of a set of parameters, selected from among (a) brightness, (b) brightness and hue, (c) brightness, hue, and saturation, (d) R, G and B values, and (e) R, G, B values and texture.

4. The method of claim 2, wherein said segmentation includes supervised learning.

5. The method of claim 2, wherein said segmentation includes unsupervised learning.

6. The method of claim 1, wherein step i is repeated periodically;

upon each repetition a selected one from the latest frames in the video signal becomes part of said plurality of frames.

7. The method of claim 1, wherein step iv includes geometric transformation of at least one model shadow zone, such as to adjust for any effects due to said change in the field of view.

8. The method of claim 1, wherein step i includes geometric transformation of the image carried by at least one frame, such as to adjust for any effects due to said change in the field of view.

9. The method of claim 1, wherein if the image in any frame includes only part of the scene or of said portion thereof that corresponds to the background model image, said combining in step i is with respect to values corresponding to only the included part of the scene or of said portion.

10. The method of claim 1, wherein there are one or more additional digital video signals, similarly obtained from corresponding additional video cameras aimed at the scene, and in step i said values additionally include values from frames in said additional video signals.

11. The method of claim 1, further comprising—

(vi) in the background model image, detecting and defining boundaries of a region of interest;

and wherein step ii is carried out within said boundaries.

12. The method of claim 11, wherein said model shadow zones defined in step ii include only zones that border on a boundary of the region of interest.

13. The method of claim 1, wherein the direction of the illumination of the scene is known and wherein said model shadow zones defined in step ii include only zones whose outline shape is consistent with the direction of illumination.

14. The method of claim 1, further comprising—

(vii) automatically dividing the background model image into sectors, each having essentially uniform color or texture, any adjacent sectors having mutually different colors or textures,

and wherein steps ii and iii are carried out mutually independently among said sectors.

15. The method of claim 1, wherein in step iii the correction factors are such as to modify any image values within the respective one of said model shadow zones to essentially equal those in nearby areas outside any model shadow zone.

16. The method of claim 15, wherein said correction factors include mutually independent R, G and B values.

17. The method of claim 15, further comprising—

(viii) calculating brightness gradients in the background model image,

and wherein said equality is along lines normal to the direction of the nearest calculated gradient.

18. The method of claim 1, wherein step i includes applying image enhancing techniques to the background model image.

19. The method of claim 1, wherein said correcting in step v is according to the corresponding correction factors calculated in step iv.

20. The method of claim 19, wherein step v includes image improvements in the shadow zones after correction.

21. The method of claim 1, further comprising—

(ix) correcting values within model shadow zones in the background model image,

and wherein said correcting in step v includes applying to one or more pixels of the video signal corresponding ones of said corrected values from the background model image.

22. The method of claim 21, wherein said applying includes substituting the value of a pixel by the corresponding corrected value or blending the value of a pixel with the corresponding corrected value.

23. The method of claim 21, further including creating a corrected background model image which includes the corrected values from step ix.

24. The method of claim 1, further comprising:

(x) Detecting any moving objects in the image carried by the video signal;

(xi) for each detected moving object, or portion thereof, that is within any of said shadow zones, modifying corresponding correction factors and applying the modified correction factors to pixels that correspond to said object.

25. The method of claim 24, wherein said detecting includes mutually comparing at least two video frames.

26. The method of claim 24, wherein said detecting includes comparing any video frame with the background model image.

27. The method of claim 24, further comprising—

(xii) for each detected moving object, or portion thereof, that is outside all the shadow zones, detecting the shadow cast by the object and removing or reducing it in the video signal.

28. The method of claim 1, wherein in step iii said calculating includes applying a gradient function.

29. The method of claim 20, wherein said improvements include noise reduction or deblocking or both noise reduction and deblocking.

30. A shadows removal system, adapted to receive, directly or indirectly, a digital video signal from a video camera moving so as to view regions within a scene and to implement a method for detecting and removing shadows in images that are carried by the video signal, the method comprising:

combining corresponding values from a plurality of frames that are carried by a digital video signal, obtained, directly or indirectly, from a video camera, any two frames of the video signal differing from each other owing to the field of view of the video camera changing by any camera action selected from tilting, panning, zooming or dollying or any combination thereof, said corresponding values being combined so as to create therefrom a background model image of the scene or of a portion thereof, said background model image covering a region greater than a field of view of at least one individual frame of the video signal;

processing the background model image to detect and define within the background model image model shadow zones;

calculating correction factors for some or all of said model shadow zones;

for each of a plurality of frames of the video signal, determining a region of said background model image corresponding to the field of view of the frame, and thereby deriving, in the frame carried by the signal, frame shadow zones that correspond to said model shadow zones; and

correcting video signal values within said frame shadow zones.

31. The system of claim 30, including a computing device having a program code embodied therein and adapted to execute said program code;

execution of said program code implements said method.

32. A computer program product, comprising a non-transitory computer usable medium having a computer readable program code embodied therein, said computer readable program code adapted to be executed to implement a method for detecting and removing shadows in images that are carried by a digital video signal, obtained, directly or indirectly, from a video camera moving so as to view regions within a scene said method comprising:

combining corresponding values from a plurality of frames that are carried by a digital video signal, obtained, directly or indirectly, from a video camera, any two frames of the video signal differing from each other owing to the field of view of the video camera changing by any camera action selected from tilting, panning, zooming or dollying or any combination thereof, said corresponding values being combined so as to create therefrom a background model image of the scene or of a portion thereof, said background model image covering a region greater than a field of view of at least one individual frame of the video signal;

processing the background model image to detect and define within the background model image model shadow zones;

calculating correction factors for some or all of said model shadow zones;

for each of a plurality of frames of the video signal, determining a region of said background model image corresponding to the field of view of the frame, and thereby deriving, in the frame carried by the signal, frame shadow zones that correspond to said model shadow zones; and

correcting video signal values within said frame shadow zones.