Context processor for video analysis system

• US 8,200,011 B2
• Filed: 04/30/2008
• Issued: 06/12/2012
• Est. Priority Date: 09/27/2007
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: Behavioral Recognition Systems Incorporated
• Inventors: Eaton, John Eric, Cobb, Wesley Kenneth, Blythe, Bobby Ernest, Gottumukkal, Rajkiran Kumar, Saitwal, Kishor Adinath
• Primary Examiner(s): Liu, Li
• Application Number:
  US12/112,864
• Publication Number:
  US 20090087024A1
• Time in Patent Office: 1,504 Days
• Field of Search: None
• US Class Current: 382/173
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

Embodiments of the present invention provide a method and a system for mapping a scene depicted in an acquired stream of video frames that may be used by a machine-learning behavior-recognition system. A background image of the scene is segmented into plurality of regions representing various objects of the background image. Statistically similar regions may be merged and associated. The regions are analyzed to determine their z-depth order in relation to a video capturing device providing the stream of the video frames and other regions, using occlusions between the regions and data about foreground objects in the scene. An annotated map describing the identified regions and their properties is created and updated.

Claims

1. A method for mapping a scene depicted in a sequence of video frames, the method comprising:

generating, from the sequence of video frames, a background image representing static elements in the scene;

receiving a current video frame;

identifying a plurality of regions of the current frame depicting the background image;

analyzing the plurality of regions of the current frame to identify at least a first region of the background image of the scene that occludes at least a portion of a second region of the background image of the scene; and

updating an annotated map specifying geometry for one or more of the plurality of regions in the background image to indicate that the first region is in front of the second region, relative to a position of a camera capturing the sequence of video frames.

2. The method of claim 1, further comprising:

identifying at least one foreground object depicted in the current frame, wherein the foreground object is defined by a collection of pixels of the current frame having one or more features different from a corresponding set of pixels in the background image.

3. The method of claim 2, further comprising:

determining a conversion factor based on a classification of the foreground object and a number of pixels used to depict the foreground object in the current frame; and

updating the annotated map of the scene to reflect the conversion factor.

4. The method of claim 2, wherein analyzing the plurality of regions of the current frame to identify the first region that occludes at least a portion of the second region comprises determining that at least a portion of the first region occludes the foreground object and that at least a portion of the second region is not occluded by the foreground object.

5. The method of claim 1, further comprising:

prior to analyzing the plurality of regions of the current frame, identifying two or more regions of the plurality sharing a set of features; and

merging the two or more identified regions.

6. The method of claim 1, further comprising:

determining at least one physical characteristic of at least one of the regions; and

updating the annotated map of the scene based on the determined at least one physical characteristic of the at least one region, wherein the at least one physical characteristic is one of position, orientation, velocity, direction, trajectory, size, shape, class, and distance scale.

7. The method of claim 1, further comprising:

classifying at least one of the plurality of regions using a trained classifier, wherein the at least one region is classified as one of natural-made, man-made, structure, and non-structure; and

updating the annotated map of the scene to reflect the classification of the at least one region the re-defined segmented image.

8. A computer-readable storage medium containing a program, which, when executed on a processor is configured to perform an operation for mapping a scene depicted in a sequence of video frames, the operation comprising:

generating, from the sequence of video frames, a background image representing static elements in the scene;

receiving a current video frame;

identifying a plurality of regions of the current frame depicting the background image;

analyzing the plurality of regions of the current frame to identify at least a first region of the background image of the scene that occludes at least a portion of a second region of the background image of the scene; and

updating an annotated map specifying geometry for one or more plurality of regions in the background image to indicate that the first region is in front of the second region, relative to a position of a camera capturing the sequence of video frames.

9. The computer-readable storage medium of claim 8, wherein the operation further comprises:

identifying at least one foreground object depicted in the current frame, wherein the foreground object is defined by a collection of pixels of the current frame having one or more features different from a corresponding set of pixels in the background image.

10. The computer-readable storage medium of claim 9, wherein the operation further comprises:

determining a conversion factor based on a classification of the foreground object and a number of pixels used to depict the foreground object in the current frame; and

updating the annotated map of the scene to reflect the conversion factor.

11. The computer-readable storage medium of claim 9, wherein analyzing the plurality of regions of the current frame to identify the first region that occludes at least a portion of the second region comprises determining that at least a portion of the first region occludes the foreground object and that at least a portion of the second region is not occluded by the foreground object.

12. The computer-readable storage medium of claim 8, wherein the operation further comprises:

prior to analyzing the plurality of regions of the current frame, identifying two or more regions of the plurality sharing a set of features; and

merging the two or more identified regions.

13. The computer-readable storage medium of claim 8, wherein the operation further comprises:

determining at least one physical characteristic of at least one of the regions; and

updating the annotated map of the scene based on the determined at least one physical characteristic of the at least one region, wherein the at least one physical characteristic is one of position, orientation, velocity, direction, trajectory, size, shape, class, and distance scale.

14. The computer-readable storage medium of claim 8, wherein the operation further comprises:

classifying at least one of the plurality of regions using a trained classifier, wherein the at least one region is classified as one of natural-made, man-made, structure, and non-structure; and

updating the annotated map of the scene to reflect the classification of the at least one region the re-defined segmented image.

15. The computer-readable storage medium of claim 8, wherein each of the sequence of video frames provides a two-dimensional array of pixel values.

16. The computer-readable storage medium of claim 15, wherein each region is defined by a collection of pixels of the current frame.

17. The computer-readable storage medium of claim 8, wherein the background image comprises a two-dimensional array of pixel values.

18. A system comprising,

a processor;

and

a memory containing a context processor application, which when executed by the processor is configured to;

generate, from the sequence of video frames, a background image representing static elements in the scene,

receive a current video frame,

identify a plurality of regions of the current frame depicting the background image,

analyze the plurality of regions of the current frame to identify at least a first region of the background image of the scene that occludes at least a portion of a second region of the background image of the scene, and

update an annotated map specifying geometry for one or more of the plurality of regions in the background image to indicate that the first region is in front of the second region, relative to a position of a camera capturing the sequence of video frames.

19. The system of claim 18, wherein the context processor is further configured to:

identify at least one foreground object depicted in the current frame, wherein the foreground object is defined by a collection of pixels of the current frame having one or more features different from a corresponding set of pixels in the background image.

20. The system of claim 19, wherein the context processor is further configured to:

determine a conversion factor based on a classification of the foreground object and a number of pixels used to depict the foreground object in the current frame; and

update the annotated map of the scene to reflect the conversion factor.

21. The system of claim 19, wherein the context processor is configured to analyze the plurality of regions of the current frame to identify first region that occludes at least a portion of the second region by determining that at least a portion of the first region occludes the foreground object and that at least a portion of the second region is not occluded by the foreground object.

22. The system of claim 18, wherein the context processor is further configured to:

prior to analyzing the plurality of regions of the current frame, identify two or more regions of the plurality sharing a set of features; and

merge the two or more identified regions.

23. The system of claim 18, wherein the context processor is further configured to:

determine at least one physical characteristic of at least one of the regions; and

update the annotated map of the scene based on the determined at least one physical characteristic of the at least one region, wherein the at least one physical characteristic is one of position, orientation, velocity, direction, trajectory, size, shape, class, and distance scale.

24. The system of claim 18, wherein the context processor is further configured to:

classify at least one of the plurality of regions using a trained classifier, wherein the at least one region is classified as one of natural-made, man-made, structure, and non-structure; and

update the annotated map of the scene to reflect the classification of the at least one region the re-defined segmented image.