REMOVAL OF SHADOWS FROM IMAGES IN A VIDEO SIGNAL

• US 20110273620A1
• Filed: 12/10/2009
• Published: 11/10/2011
• Est. Priority Date: 12/24/2008
• Status: Active Grant

• Current Assignee: Rafael Advanced Defense Systems Limited (Government of Israel)
• Original Assignee: Rafael Advanced Defense Systems Limited (Government of Israel)
• Inventors: Peles, David, Berkovich, Erez
• Granted Patent: US 8,761,497 B2
• Time in Patent Office: Days
• US Class Current: 348/584
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 5/002 Denoising; Smoothing noise processing or correction adapted to be used in an image pickup device containing and electronic image sensor H04N23/81, H04N25/60 - H04N25/67
• CPC Class Codes: G06T 5/008 Local, e.g. shadow enhancement
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/94 based on local image properties, e.g. for local contrast enhancement

Abstract

Substantially removing shadows in video images obtained from a camera viewing a scene, at variable pointing angles and magnification, by digital processing of a sequence of frames in the video signal, the processing including—(a) creating and maintaining a model image of the scene, by accumulating image data from a plurality of video frames; (b) in the model image, detecting and defining model shadow zones; (c) calculating correction factors for the model shadow zones; (d) for each frame of the video signal—defining, in the image carried by the signal, shadow zones that correspond to respective model shadow zones; and (e) correcting video signal values in shadow zones accordingly.

Claims

1. A method for detecting and removing shadows in images that are carried by a digital video signal, obtained, directly or indirectly, from a video camera aimed at a scene, the method comprising:

(i) Combining corresponding values from a plurality of frames in the video signal and creating therefrom a model image of the scene or of a portion thereof;

(ii) in the model image, detecting and defining model shadow zones;

(iii) calculating correction factors for some or all of said model shadow zones;

(iv) for each frame of the video signal—

defining, in the image carried by the signal, shadow zones that correspond to respective ones of said model shadow zones;

(v) correcting video signal values within shadow zones defined in step iv.

2. The method of claim 1, wherein in step ii said detecting and defining includes segmentation of pixel values.

3. The method of claim 2, wherein said segmentation is in terms of a set of parameters, selected from among (a) brightness, (b) brightness and hue (c) (c) brightness, hue, and saturation, (d) R, G and B values and (e) R, G, B values and texture.

4. The method of claim 2, wherein said segmentation includes supervised learning.

5. The method of claim 2, wherein said segmentation includes unsupervised learning.

6. The method of claim 1, wherein step i is repeated periodically;

upon each repetition a selected one from the latest frames in the video signal becomes part of said plurality of frames.

7. The method of claim 1, wherein any two frames of the video signal differ from each other owing to the field of view of the video camera changing by any camera action selected from tilting, panning, zooming or dollying or any combination thereof.

8. The method of claim 7, wherein step iv includes geometric transformation of at least one model shadow zone, such as to adjust for any effects due to said change in the field of view.

9. The method of claim 7, wherein step i includes geometric transformation of the image carried by at least one frame, such as to adjust for any effects due to said change in the field of view.

10. The method of claim 1, wherein if the image in any frame includes only part of the scene or of said portion thereof that corresponds to the model image, said combining in step i is with respect to values corresponding to only the included part of the scene or of said portion.

11. The method of claim 1, wherein there are one or more additional digital video signals, similarly obtained from corresponding additional video cameras aimed at the scene, and in step i said values additionally include values from frames in said additional video signals.

12. The method of claim 1, further comprising—

(vi) in the model image, detecting and defining boundaries of a region of interest

and wherein step ii is carried out within said boundaries,

13. The method of claim 12, wherein shadow zones defined in step ii include only zones that border on a boundary of the region of interest.

14. The method of claim 1, wherein the direction of the illumination of the scene is known and wherein shadow zones defined in step ii include only zones whose outline shape is consistent with the direction of illumination.

15. The method of claim 1, further comprising—

(vii) automatically dividing the model image into sectors, each having essentially uniform color or texture, any adjacent sectors having mutually different colors or textures,

and wherein steps ii and iii are carried out mutually independently among said sectors.

16. The method of claim 1, wherein in step iii the correction factors are such as to modify any image values within the respective one of said zones to essentially equal those in nearby areas outside any model shadow zone.

17. The method of claim 16, wherein said correction factors include mutually independent R, G and B values.

18. The method of claim 16, further comprising—

(viii) calculating brightness gradients in the model image

and wherein said equality is along lines normal to the direction of the nearest calculated gradient.

19. The method of claim 1, wherein step i includes applying image enhancing techniques to the model image.

20. The method of claim 1, wherein said correcting in step v is according to the corresponding correction factors calculated in step iv.

21. The method of claim 20, wherein step v includes image improvements in the shadow zones after correction.

22. The method of claim 1, further comprising—

(ix) correcting values within model shadow zones in the model image and wherein said correcting in step v includes applying to one or more pixels of the video signal corresponding ones of said corrected values from the model image.

23. The method of claim 22, wherein said applying includes substituting the value of a pixel by the corresponding corrected value or blending the value of a pixel with the corresponding corrected value.

24. The method of claim 22, further including creating a corrected model age which includes the corrected values from step ix.

25. The method of claim 1, further comprising:

(x) Detecting any moving objects in the image carried by the video signal;

(xi) for each detected moving object, or portion thereof, that is within any of said shadow zones, modifying corresponding correction factors and applying the modified correction factors to pixels that correspond to said object.

26. The method of claim 25, wherein said detecting includes mutually comparing at least two video frames.

27. The method of claim 25, wherein said detecting includes comparing any video frame with the model image.

28. The method of claim 25, further comprising—

(xii) for each detected moving object, or portion thereof, that is outside all the shadow zones, detecting the shadow cast by the object and removing or reducing it in the video signal.

29. The method of claim 1, wherein in step iii said calculating includes applying a gradient function.

30. The method of claim 21, wherein said improvements include noise reduction or deblocking or both noise reduction and deblocking.

31. A shadows removal system, adapted to receive, directly or indirectly, a digital video signal from a video camera aimed at a scene and to implement a method for detecting and removing shadows in images that are carried by the video signal, the method comprising:

Combining corresponding values from a plurality of frames in the video signal and creating therefrom a model image of the scene or of a portion thereof;

in the model image, detecting and defining model shadow zones;

calculating correction factors for some or all of said model shadow zones;

for each frame of the video signal—

defining, in the image carried by the signal, shadow zones that correspond to respective ones of said model shadow zones;

correcting video signal values within said shadow zones.

32. The system of claim 31, including a computing device having a program code embodied therein and adapted to execute said program code;

execution of said program code implements said method.

33. A computer program product, comprising a computer usable medium having a computer readable program code embodied therein, said computer readable program code adapted to be executed to implement a method for detecting and removing shadows in images that are carried by a digital video signal, obtained, directly or indirectly, from a video camera aimed at a scene, said method comprising:

Combining corresponding values from a plurality of frames in the video signal and creating therefrom a model image of the scene or of a portion thereof;

in the model image, detecting and defining model shadow zones;

calculating correction factors for some or all of said model shadow zones;

for each frame of the video signal—

defining, in the image carried by the signal, shadow zones that correspond to respective ones of said model shadow zones;

correcting video signal values within said shadow zones.