Shadow and highlight detection system and method of the same in surveillance camera and recording medium thereof

• US 8,280,106 B2
• Filed: 06/10/2008
• Issued: 10/02/2012
• Est. Priority Date: 09/29/2007
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Ma, Gengyu
• Primary Examiner(s): Repko, Jason M
• Assistant Examiner(s): ROWLAND, JHERE M
• Application Number:
  US12/155,839
• Publication Number:
  US 20090087025A1
• Time in Patent Office: 1,575 Days
• Field of Search: None
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

A method and system for detecting a shadow region and a highlight region from a foreground region in a surveillance system, and a recording medium thereof, are provided. The system includes an image capturing unit to capture a new image, a background model unit to receive the new image and update a stored background model with the new image, a difference image obtaining unit to compare the new image with the background model and to obtain a difference image between the new image and the background model, a penumbra region extraction unit to extract a partial shadow region or a partial highlight region by measuring a sharpness of an edge of the difference image and expanding a background region, and an umbra region extraction unit to extract a complete shadow region or a complete highlight region based on the result of the extraction by the penumbra region extraction unit.

Claims

1. A system for detecting a shadow region and a highlight region, the system comprising:

a processor to control one or more processor-executable units;

an image capturing unit to capture a new image;

a background model unit to receive the new image and update a stored background model with the new image;

a difference image obtaining unit to compare the new image with the background model and to obtain a difference image between the new image and the background model;

a penumbra region extraction unit to extract a partial shadow region or a partial highlight region by measuring a sharpness of an edge of the difference image and expanding a background region by way of a horizontal and vertical scan of the background region, wherein a foreground pixel is determined as one of the partial shadow region or the partial highlight region when the foreground pixel is reachable from the background region without crossing a sharp edge; and

an umbra region extraction unit to extract a complete shadow region or a complete highlight region based on the result of the extraction by the penumbra region extraction unit.

2. The system of claim 1, wherein the background model is a collection of pixel distributions, and a color distribution of each pixel is described in a Gaussian Mixture model given by,

Σ

w_i·

N(u_i,σ

_i),

where w_i is a weight of each unit Gaussian, and N(u_i, σ

_i) is a unit Gaussian distribution with a center u_i and a variance aσ

_i.

3. The system of claim 2, wherein the background model is formed to include a pixel in the background region when the pixel belongs to a unit Gaussian with a relatively great weight.

4. The system of claim 2, wherein the background model unit compares a pixel of the new image with the Gaussian Mixture model, and when the pixel of the new image belongs to a predetermined unit Gaussian, the background model unit updates the center, the variance, and the weight of the unit Gaussian, and when the pixel is excluded from all unit Gaussians, the background model unit deletes a unit Gaussian with a smallest weight and adds a new unit Gaussian centered at the pixel.

5. The system of claim 4, wherein when the pixel of the new image belongs to the predetermined unit Gaussian, the background model unit updates the center of the unit Gaussian by assigning a weight to an original center of the unit Gaussian and a color of the pixel and obtaining the average thereof, and updates the variance by obtaining the average of an original variance of the unit Gaussian and a distance from the pixel to the original center of the unit Gaussian, and updates the weight by increasing the weight of the unit Gaussian and decreasing weights of the other unit Gaussians.

6. The system of claim 2, wherein the difference image obtaining unit finds a unit Gaussian of the background model closest to the pixel, and when the distance between the pixel and a center of the closest unit Gaussian is less than a variance of the unit Gaussian, labels the pixel as a background and sets a difference of the difference image as zero, and when the distance is greater than or equal to the variance, sets the difference as the distance and stores the difference image.

7. The system of claim 1, wherein the penumbra region extraction unit comprises:

a sharpness measurement unit to measure the sharpness of the edge of the difference image; and

a background expansion unit to horizontally and vertically scan the background region of the background model and expand the background region of the background model by labeling a predetermined pixel of the background region as the partial shadow region when a sharpness of the pixel of the background region of the background model is less than a sharpness threshold of a partial shadow and when a difference between the pixel and the background model is less than a difference threshold of the partial shadow, and labeling the pixel as the partial highlight region when the sharpness of the pixel is less than a sharpness threshold of a highlight and when the difference between the pixel and the background model is greater than a difference threshold of the highlight.

8. The system of claim 7, wherein the sharpness measurement unit computes gradients of the pixel of the new image in a small scale and a large scale respectively, and computes, as the sharpness of the pixel, a value that is obtained by dividing the small scale by the large scale or by dividing the small scale by a sum of the large scale and a constant.

9. The system of claim 8, wherein the sharpness measurement unit sets the sharpness as zero when the sharpness of the pixel is less than a predetermined threshold.

10. The system of claim 7, wherein the background expansion unit labels, as the partial shadow region, a nearest left-hand side pixel in the background region, when a sharpness of the nearest left-hand side pixel is less than a sharpness threshold of the shadow and the difference is less than a difference threshold of the shadow, and

the background expansion unit labels the pixel as the partial highlight region when the sharpness is less than a sharpness threshold of the highlight and the difference is greater than a difference threshold of the highlight, and

the background expansion unit repeats the operation with respect to a subsequent left-hand side pixel until the above conditions are not satisfied, and performs the expanding operation to a right-hand side direction in the same way and thereby extracts the partial shadow region or the partial highlight region.

11. The system of claim 7, wherein the background expansion unit labels, as the partial shadow region, a nearest upper pixel in the background region when a sharpness of the nearest upper pixel is less than a sharpness threshold of the shadow and the difference is less than a difference threshold of the shadow, and

the background expansion unit labels the pixel as the partial highlight region when the sharpness is less than a sharpness threshold of the highlight and the difference is greater than a difference threshold of the highlight, and

the background expansion unit repeats the operation with respect to a subsequent upper pixel until the above conditions are not satisfied and performs the expanding operation in a lower direction in the same way and thereby extracts the partial shadow region or the partial highlight region.

12. The system of claim 1, wherein the umbra region extraction unit labels a predetermined region as the complete shadow region when the region is surrounded by the partial shadow region, and labels the region as the complete highlight region when the region is surrounded by the partial highlight region.

13. The system of claim 12, wherein the umbra region extraction unit labels a part of a foreground region as the complete shadow region by applying a horizontal and vertical scan when the part is surrounded by the partial shadow region or the partial highlight region,

the umbra region extraction unit labels an object region as the complete shadow region when the scanned line region has a partial shadow region-object region-partial shadow region pattern and the length of the object region is less than a length threshold of the shadow, and

the umbra region extraction unit labels the object region as the complete highlight region when the scanned line region has a partial highlight region-object region-partial highlight region pattern and the length of the object region is less than a length threshold of the highlight.

14. A method of detecting a shadow region and a highlight region, the method comprising:

capturing a new image;

comparing the new image with a background model and updating the background model with the new image;

obtaining a difference image between the new image and the background image;

extracting a partial shadow region or a partial highlight region by measuring a sharpness of a edge of the difference image and expanding a background region by way of a horizontal and vertical scan of the background region;

determining a foreground pixel as one of the partial shadow region or the partial highlight region when the foreground pixel is reachable from the background region without crossing a sharp edge; and

extracting a complete shadow region or a complete highlight region based on the result of the extraction of the partial shadow region or the partial highlight region.

15. The method of claim 14, wherein the background model is a collection of pixel distributions and a color distribution of each pixel is described in a Gaussian Mixture model, given by,

Σ

w_i·

N(u_i,σ

_i),

where w_i is a weight of each unit Gaussian, and N(u_i, σ

_i) is a unit Gaussian with a center u_i and a variance σ

_i.

16. The method of claim 15, wherein the updating comprises:

determining whether a pixel of the new image belongs to a predetermined unit Gaussian in the Gaussian Mixture model;

when the pixel belongs to the unit Gaussian, updating the center and the variance of the unit Gaussian, increasing a weight of the Gaussian, and decreasing weights of the other unit Gaussians, and thereby updating the weight of the unit Gaussian; and

when the pixel is excluded from all unit Gaussians, deleting a unit Gaussian with a smallest weight and adding a new unit Gaussian centered at the pixel.

17. The method of claim 15, wherein the obtaining comprises:

comparing a pixel of the new image with a center of each unit Gaussian in the background model, and locating a unit Gaussian closest to the center;

determining whether a distance between the pixel and the center of the closest unit Gaussian is less than a variance of the closest unit Gaussian;

when the distance is less than the variance, labeling the pixel as a background and setting a difference of the difference image as zero; and

when the distance is greater than or equal to the variance, setting the difference as the distance and thereby storing the difference image.

18. The method of claim 14, wherein the extracting of the partial shadow region or the partial highlight region comprises:

computing gradients of a pixel of the new image in a small scale and a large scale respectively; and

measuring a sharpness of the pixel by dividing the small scale by the large scale, or by dividing the small scale by a sum of the large scale and a constant.

19. The method of claim 18, wherein the measuring sets the sharpness of the pixel as zero when the small scale is less than a predetermined threshold.

20. The method of claim 14, wherein the operation of expanding the background region in the extracting of the partial shadow region or the partial highlight region comprises:

locating every background region of each horizontal scan line and horizontally expanding the background region for each pixel with respect to each horizontal scan line; and

locating every background region of each vertical scan line and vertically expanding the background region for each pixel with respect to each vertical scan line.

21. The method of claim 20, wherein the operation of horizontally expanding the background region labels, as the partial shadow region, a nearest left-hand side pixel in the background region, when a sharpness of the nearest left-hand side pixel is less than a sharpness threshold of the shadow and the difference is less than a difference threshold of the shadow, and

labels the pixel as the partial highlight region when the sharpness is less than a sharpness threshold of the highlight and the difference is greater than a difference threshold of the highlight, and

the operation is repeated with respect to a subsequent left-hand side pixel until the above conditions are not satisfied, and the expanding operation is performed in a right-hand side direction in the same way.

22. The method of claim 20, wherein the operation of vertically expanding the background region labels, as the partial shadow region, a nearest upper pixel in the background region when a sharpness of the nearest upper pixel is less than a sharpness threshold of the shadow and the difference is less than a difference threshold of the shadow, and

labels the pixel as the partial highlight region when the sharpness is less than a sharpness threshold of the highlight and the difference is greater than a difference threshold of the highlight, and

the operation is repeated with respect to a subsequent upper pixel until the above conditions are not satisfied and the expanding operation is performed to a lower direction in the same way.

23. The method of claim 14, wherein the extracting of the complete shadow region or the complete highlight region comprises:

labeling a predetermined region as the complete shadow region when the region is surrounded by the partial shadow region, and labeling the region as the complete highlight region when the region is surrounded by the partial highlight region; and

labeling a part of a foreground region as the complete shadow region by applying a horizontal and vertical scan when the part is surrounded by the partial shadow region, and labeling the part as the complete highlight region by applying the horizontal and vertical scan when the part of the foreground region is surrounded by the partial highlight region.

24. The method of claim 23, wherein the labeling of the part as the complete shadow region or the complete highlight region labels an object region as the complete shadow region when the scanned line region has a partial shadow region-object region-partial shadow region pattern and the length of the object region is less than a length threshold of the shadow, and

labels the object region as the complete highlight region when the scanned line region has a partial highlight region-object region-partial highlight region pattern and the length of the object region is less than a length threshold of the highlight.

25. At least one non-transitory medium comprising computer readable instructions to control at least one processing element in a computer to implement the method of claim 14.