Apparatus and method for detecting foreground in image

• US 10,311,579 B2
• Filed: 01/13/2017
• Issued: 06/04/2019
• Est. Priority Date: 01/22/2016
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd., Seoul National University R&DB Foundation (Seoul National University)
• Original Assignee: Samsung Electronics Co. Ltd., Seoul National University R&DB Foundation (Seoul National University)
• Inventors: Yun, Ki Min, Choi, Jin Young, Lim, Jong In
• Primary Examiner(s): Hung, Yubin
• Application Number:
  US15/405,735
• Publication Number:
  US 20170213100A1
• Time in Patent Office: 872 Days
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06V 10/26 Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
• CPC Class Codes: G06V 10/28 Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns

Abstract

An apparatus and method for detecting a foreground in an image is provided, and the foreground detecting apparatus includes a context information estimator configured to estimate context information on a scene from an image frame of the image, a background model constructor configured to construct a background model of the image frame using the estimated context information, and a foreground detector configured to detect a foreground from the image frame based on the constructed background model.

Claims

1. An apparatus for detecting a foreground in an image, the apparatus comprising:

a memory configured to store instructions; and

a processor configured to execute the instructions to;

estimate context information, including a foreground speed, on a scene from an image frame of the image;

construct a background model of the image frame using the estimated context information; and

detect a foreground from the image frame based on the constructed background model,

wherein the foreground speed is estimated by calculating a speed using an optical flow of the image frame and a previous image frame, and correcting the speed based on a projective transform matrix that represents background motion.

2. The apparatus of claim 1, wherein the context information further includes an illumination change.

3. The apparatus of claim 2, wherein the processor is further configured to execute the instructions to:

generate the projective transform matrix using the speed; and

estimate the illumination change based on a difference between a mean of brightness intensities of the image frame and a mean of brightness intensities of the background model.

4. The apparatus of claim 3, wherein the previous image frame comprises an immediately previous image frame.

5. The apparatus of claim 2, wherein for the construction of the background model, the processor is further configured to execute the instructions to generate the background model of the image frame using a background model of the previous image frame and the background motion.

6. The apparatus of claim 5, wherein for the construction of the background model, the processor is further configured to execute the instructions to determine locations on the previous image frame based on a location of the previous image frame which corresponds to a first location of the image frame, and to generate a background model of the image frame for the first location by performing weighted summation on background models of the previous image frame for the determined locations.

7. The apparatus of claim 6, wherein for the construction of the background model, the processor is further configured to execute the instructions to correct a variance of the generated background model, in response to a value of a context variable based on the foreground speed being greater than or equal to a block size.

8. The apparatus of claim 7, wherein for the construction of the background model, the processor is further configured to execute the instructions to correct a variance of the background model of the first location by reflecting differences between a mean of the background model of the first location and means of the background models at the determined locations.

9. The apparatus of claim 5, wherein for the construction of the background model, the processor is further configured to execute the instructions to update the generated background model based on the illumination change.

10. The apparatus of claim 9, wherein for the updating of the generated background model, the processor is further configured to execute the instructions to update a mean of the generated background model by reflecting the illumination change, in response to value of a context variable being smaller than a block size, and to update the mean and a variance of the generated background model by reflecting the illumination change and additional information, in response to the value of the context variable being greater than or equal to the block size.

11. The apparatus of claim 10, wherein the additional information comprises either one or both of a mean of block-specific brightness intensities and information on a difference between a mean of the generated background model and the image frame.

12. The apparatus of claim 10, wherein for the updating of the generated background model, the processor is further configured to execute the instructions to calculate a value of a time-variant variable in units of blocks starting from an initial image frame input from the camera to the image frame, and to adjust an update intensity of the generated background model using the value of the time-variant variable.

13. The apparatus of claim 1, wherein for the updating of the generated background model, the processor is further configured to execute the instructions to generate a foreground probability map by calculating a foreground probability for each pixel of the image frame based on the constructed background model, and extract the foreground from the image frame using the generated foreground probability map.

14. The apparatus of claim 13, wherein for the extracting of the foreground, the processor is further configured to execute the instructions to classify each pixel of the image frame into any one or any combination of any two or more of the foreground, a candidate, and a background based on a comparison of the foreground probability of each pixel of the image frame with a threshold value, and to categorize pixels classified into the candidate into the foreground or the background based on applying watershed segmentation to the pixels classified into the candidate.

15. A method of detecting a foreground in an image, the method comprising:

estimating context information, including a foreground speed, on a scene from an image frame of the image;

constructing a background model of the image frame using the estimated context information;

correcting a variance of the constructed background model, dependent on whether a value of a context variable based on the foreground speed is greater than or equal to a block size; and

detecting a foreground from the image frame based on the variance-corrected constructed background model.

16. The method of claim 15, wherein the context information further comprises an illumination change.

17. The method of claim 16, wherein the constructing of the background model comprises generating the background model of the image frame using a background model of a previous image frame based on information on a background motion.

18. The method of claim 17, wherein the constructing of the background model further comprises updating the constructed background model based on the illumination change.

19. The method of claim 18, wherein the updating of the background model further comprises, updating a mean of the constructed background model by reflecting the illumination change, in response to a value of the context variable being smaller than a block size, and updating a mean and the variance of the constructed background model by reflecting the illumination change and additional information, in response to the value of the context variable being greater than or equal to the block size.

20. The method of claim 15, wherein the detecting of the foreground comprises

generating a foreground probability map by calculating a foreground probability for each pixel of the image frame based on the constructed background model, and

extracting the foreground from the image frame using the generated foreground probability map.

21. A non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method of claim 15.

22. A digital device, comprising:

an antenna;

a cellular radio configured to transmit and receive data via the antenna according to a cellular communications standard;

a touch-sensitive display;

a camera configured to capture an image;

a memory configured to store instructions; and

a processor configured to execute the instructions to

receive the image from the camera,

estimate context information, including an illumination change, on a scene from an image frame of the image,

construct a background model of the image frame using the estimated context information,

update a mean of the constructed background model by reflecting the illumination change, dependent on whether a value of a context variable is smaller than a block size, and

detect a foreground from the image frame based on the constructed background model.