Background model for complex and dynamic scenes

• US 8,358,834 B2
• Filed: 08/18/2009
• Issued: 01/22/2013
• Est. Priority Date: 08/18/2009
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: Behavioral Recognition Systems Incorporated
• Inventors: Cobb, Wesley Kenneth, Seow, Ming-Jung, Yang, Tao
• Primary Examiner(s): Bali, Vikkram
• Application Number:
  US12/543,336
• Publication Number:
  US 20110044537A1
• Time in Patent Office: 1,253 Days
• Field of Search: 382/155
• US Class Current: 382/155
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06F 18/23211 with adaptive number of clusters
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06V 10/763 Non-hierarchical techniques, e.g. based on statistics of modelling distributions
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

Techniques are disclosed for learning and modeling a background for a complex and/or dynamic scene over a period of observations without supervision. A background/foreground component of a computer vision engine may be configured to model a scene using an array of ART networks. The ART networks learn the regularity and periodicity of the scene by observing the scene over a period of time. Thus, the ART networks allow the computer vision engine to model complex and dynamic scene backgrounds in video.

Claims

1. A computer-implemented method for generating a background model of a scene depicted in a sequence of video frames captured by a video camera, the method comprising:

receiving a video frame, wherein the video frame includes one or more appearance values for each of a plurality of pixels;

for one or more of the pixels;

passing the appearance values for the pixel to an input layer of an adaptive resonance theory (ART) network corresponding to the pixel;

mapping, by the ART network, the appearance values to one of one or more clusters of the ART network;

classifying the pixel as depicting one of scene background and scene foreground, based on the mapping of the appearance values to the cluster of the ART network, wherein the pixel is classified as depicting scene background in response to determining that the cluster to which the appearance values are mapped to is a matured cluster in the ART network.

2. The method of claim 1, wherein the appearance values for the pixel comprise a set of RGB color values.

3. The method of claim 1, wherein mapping the appearance values to the cluster of the ART network comprises generating a new cluster in the ART network.

4. The method of claim 1, further comprising, decaying a first cluster out of the ART network in response to determining the first cluster has not been reinforced by the mapping of the appearance values to the first cluster.

5. The method of claim 1, further comprising merging two or more clusters in the ART network.

6. The method of claim 1, wherein mapping the appearance values to the cluster of the ART network comprises evaluating the appearance values according to a vigilance test and a choice test.

7. The method of claim 6, wherein the appearance values for the pixel comprise a set of RGB color values input to the ART network and the choice test comprises determining a Euclidean distance between the RGB color values input to the ART network and a prototype set of RGB values associated with the cluster.

8. The method of claim 6, wherein the appearance values for the pixel comprise a set of RGB color values input to the ART network and the vigilance test comprises determining a cosine angle and a prototype set of RGB values associated with the cluster, relative to an origin of the RGB space.

9. A computer-readable storage medium containing a program, which when executed on a processor, performs an operation for generating a background model of a scene depicted in a sequence of video frames captured by a video camera, the operation comprising:

receiving a video frame, wherein the video frame includes one or more appearance values for each of a plurality of pixels;

for one or more of the pixels;

passing the appearance values for the pixel to an input layer of an adaptive resonance theory (ART) network corresponding to the pixel;

mapping, by the ART network, the appearance values to one of one or more clusters of the ART network;

classifying the pixel as depicting one of scene background and scene foreground, based on the mapping of the appearance values to the cluster of the ART network, wherein the pixel is classified as depicting scene background in response to determining that the cluster to which the appearance values are mapped to is a matured cluster in the ART network.

10. The computer-readable storage medium of claim 9, wherein mapping the appearance values to the cluster of the ART network comprises generating a new cluster in the ART network.

11. The computer-readable storage medium of claim 9, wherein the operation further comprises, decaying a first cluster out of the ART network in response to determining the first cluster has not been reinforced by the mapping of the appearance values to the first cluster.

12. The computer-readable storage medium of claim 9, wherein the operation further comprises merging two or more clusters in the ART network.

13. The computer-readable storage medium of claim 9, wherein mapping the appearance values to the cluster of the ART network comprises evaluating the appearance values according to a vigilance test and a choice test.

14. The computer-readable storage medium of claim 13, wherein the appearance values for the pixel comprise a set of RGB color values input to the ART network and the choice test comprises determining a Euclidean distance between the RGB color values input to the ART network and a prototype set of RGB values associated with the cluster.

15. The computer-readable storage medium of claim 13, wherein the appearance values for the pixel comprise a set of RGB color values input to the ART network and the vigilance test comprises determining a cosine angle and a prototype set of RGB values associated with the cluster, relative to an origin of the RGB space.

16. A system, comprising:

a video input source configured to provide a sequence of video frames, each depicting a scene;

a processor; and

a memory containing a program, which, when executed on the processor is configured to perform an operation for generating a background model of a scene depicted in a sequence of video frames captured by a video camera, the operation comprising;

receiving a video frame, wherein the video frame includes one or more appearance values for each of a plurality of pixels;

for one or more of the pixels;

passing the appearance values for the pixel to an input layer of an adaptive resonance theory (ART) network corresponding to the pixel;

mapping, by the ART network, the appearance values to one of one or more clusters of the ART network;

classifying the pixel as depicting one of scene background and scene foreground, based on the mapping of the appearance values to the cluster of the ART network, wherein the pixel is classified as depicting scene background in response to determining that the cluster to which the appearance values are mapped to is a matured cluster in the ART network.

17. The system of claim 16, wherein mapping the appearance values to the cluster of the ART network comprises generating a new cluster in the ART network.

18. The system of claim 16, wherein the operation further comprises, decaying a first cluster out of the ART network in response to determining the first cluster has not been reinforced by the mapping of the appearance values to the first cluster.

19. The system of claim 16, wherein the operation further comprises merging two or more clusters in the ART network.

20. The system of claim 16, wherein mapping the appearance values to the cluster of the ART network comprises evaluating the appearance values according to a vigilance test and a choice test.

21. The system of claim 20, wherein the appearance values for the pixel comprise a set of RGB color values input to the ART network and the choice test comprises determining a Euclidean distance between the RGB color values input to the ART network and a prototype set of RGB values associated with the cluster.

22. The system of claim 20, wherein the appearance values for the pixel comprise a set of RGB color values input to the ART network and the vigilance test comprises determining a cosine angle and a prototype set of RGB values associated with the cluster, relative to an origin of the RGB space.