Spatio-activity based mode matching

• US 8,553,086 B2
• Filed: 02/25/2009
• Issued: 10/08/2013
• Est. Priority Date: 02/28/2008
• Status: Active Grant

• Current Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Original Assignee: Canon Kabushiki Kaisha (Canon Inc.)
• Inventors: Springett, Jarrad Michael, Pakulski, Peter Jan, Vendrig, Jeroen
• Primary Examiner(s): Harold, Jefferey
• Assistant Examiner(s): DESIR, JEAN WICEL
• Application Number:
  US12/919,958
• Publication Number:
  US 20110043699A1
• Time in Patent Office: 1,686 Days
• Field of Search: 340/903, 382/104, 382/107, 382/219, 348/142, 348/135
• US Class Current: 348/143
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/143 involving probabilistic approaches, e.g. Markov random field [MRF] modelling
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69

Abstract

Disclosed is a method (101), in relation to a current video frame (300) comprising a visual element (320) associated with a location in a scene captured in the frame (300), said visual element (320) being associated with a plurality of mode models (350), said method matching (140) one of said plurality of mode models (350) to the visual element (320), said method comprising, for each said mode model (350), the steps of determining (420) a visual support value depending upon visual similarity between the visual element (320) and the mode model (350), determining (440) a spatial support value depending upon similarity of temporal characteristics of the mode models (350) associated with the visual element and mode models (385) of one or more other visual elements (331); and identifying (450) a matching one of said plurality of mode models (350) depending upon the visual support value and the spatial support value.

Claims

1. A method, in relation to a current video frame comprising a visual element having visual characteristics and corresponding to a location in the current video frame, of matching at least one of a plurality of mode models to the visual element, comprising the steps of:

associating the visual element of the current video frame with the plurality of mode models, each corresponding to the same location as the visual element and having visual and temporal characteristics; and

matching at least one of the plurality of mode models to the visual element of the current video frame, said matching step further comprising the steps of;

determining a visual support value for each mode model depending upon a similarity between the visual characteristics of the visual element and of the mode model;

determining a spatial support value for each mode model depending upon a similarity between the temporal characteristics of the mode model and of mode models associated with one or more other visual elements in the vicinity of the visual element of the current video frame; and

identifying a matching one of the plurality of mode models depending upon the visual support values and the spatial support values.

2. A method according to claim 1, wherein said associating step is based on a time history of values for the visual element over a number of frames.

3. A method according to claim 1, wherein said first determining step comprises mapping at least one image value from the visual element to a probability value indicating the probability that the mode model matches the visual element.

4. A method according to claim 1, wherein said second determining step comprises determining if a mode model associated with the visual element and a mode model associated with one of the other visual elements have similar temporal characteristics.

5. A method according to claim 4, wherein the temporal characteristic is one or more members selected from the group consisting of:

the number of times the mode model has matched the visual element in previous frames;

the number of times the mode model has been updated;

a time stamp corresponding to the creation of the mode model;

the last time the mode model was found not to match the input visual element; and

a weighted sum of all the times the mode model matched the visual element.

6. A method according to claim 1, wherein the other visual elements are neighbouring visual elements of the visual element.

7. A method according to claim 1, further comprising the steps of:

performing the method iteratively over the current frame a succession of times; and

selecting, in each iteration, the mode models associated with the one or more other visual elements used in the said second determining step based upon the previous iteration.

8. A method according to claim 1, further comprising the steps of:

determining a temporal support value for a mode model depending upon the number of times the mode model was identified as the matching mode model in previous frames within a predefined search window; and

identifying the matching mode model further depending upon the temporal support value.

9. A method of determining whether a first visual element and a neighbouring second visual element are parts of a connected component, comprising the steps of:

identifying matching mode models for the first and second visual elements according to the method of claim 1; and

determining if the first visual element and the second visual element are parts of the connected component depending upon whether a temporal characteristic of the matching mode model for the first visual element is similar to a corresponding temporal characteristic of the matching mode model for the second visual element.

10. A method according to claim 1, wherein prior to said identifying step, the method further comprises a step of creating, depending upon the visual support values and the spatial support values, a new mode model associated with the visual element.

11. A method according to claim 1, wherein prior to said identifying step, the method further comprises the steps of:

determining a confidence level that the mode model to be identified matches the visual element depending upon the visual support value and the spatial support value for the mode model to be identified; and

when the confidence level is below a predetermined threshold, creating a new mode model.

12. An apparatus, in relation to a video frame comprising a visual element having visual characteristics and corresponding to a location in the current video frame, for matching at least one of a plurality of mode models to the visual element, comprising:

a memory and a processor cooperating to function as;

an associating unit configured to associate the visual element of the current video frame with the plurality of mode models, each corresponding to the same location as the visual element and having visual and temporal characteristics; and

a matching unit configured to match at least one of the plurality of mode models to the visual element of the current video frame, said matching unit further comprising;

a first determining unit configured to determine a visual support value for each mode model depending upon a similarity between the visual characteristics of the visual element and of the mode model;

a second determining unit configured to determine a spatial support value for each mode model depending upon a similarity between the temporal characteristics of the mode model and of mode models associated with one or more other visual elements in the vicinity of the visual element of the current video frame; and

an identifying unit configured to identify a matching one of the plurality of mode models depending upon the visual support values and the spatial support values.

13. A non-transitory computer-readable storage medium having recorded thereon a computer program for directing a processor to execute a method, in relation to a video frame comprising a visual element having visual characteristics and corresponding to a location in the current video frame, of matching at least one of a plurality of mode models to the visual element, comprising:

associating the visual element of the current video frame with the plurality of mode models, each corresponding to the same location as the visual element and having visual and temporal characteristics; and

matching at least one of the plurality of mode models to the visual element of the current video frame, said matching step further comprising;

determining a visual support value for each mode model depending upon a similarity between the visual characteristics of the visual element and of the mode model;

determining a spatial support value for each mode model depending upon a similarity between the temporal characteristics of the mode model and of mode models associated with one or more other visual elements in the vicinity of the visual element of the current video frame; and

identifying a matching one of the plurality of mode models depending upon the visual support values and the spatial support values.

14. A method, in relation to a current video frame comprising a visual element having visual characteristics and corresponding to a location in the current video frame, of matching the visual element to at least one of a plurality of mode models in a visual element model, comprising the steps of:

associating said visual element of the current video frame with the plurality of mode models, each corresponding to the same location as the visual element and having visual and temporal characteristics; and

matching at least one of the plurality of mode models to the visual element of the current video frame, said matching step further comprising the steps of;

determining a visual support value for each mode model based on a correspondence between visual characteristics of the visual element and of the mode model in the visual element model;

determining a spatial support value for each mode model based on a correspondence between temporal characteristics of the mode model and of one or more mode models associated with visual elements at locations other than the location of the visual element in the vicinity of the visual element of the current video frame; and

selecting a matching mode model from the plurality of mode models based at least on the visual support values and the spatial support values.

15. A method according to claim 14, wherein the one or more mode models associated with visual elements at locations other than the location of the visual element are selected based on the output of a previous application of said method.

16. A method according to claim 14, further comprising the steps of:

determining a confidence level in the selection based on the visual support value and the spatial support value of the selected matching mode model; and

updating the visual characteristics of the selected matching mode model based on the confidence level.

17. A method according to claim 14, further comprising the steps of:

determining a confidence level in the selection based on the visual support value and the spatial support value of the selected matching mode model;

determining a shadow visual support value based on a correspondence between the visual characteristics of the visual element and of a mode model in the visual element model corresponding to the location of the visual element; and

creating a new mode model and selecting the new mode model as the matching mode based on the confidence level and the shadow visual support value.