Method and system for tracking and behavioral monitoring of multiple objects moving through multiple fields-of-view

• US 8,547,437 B2
• Filed: 11/12/2003
• Issued: 10/01/2013
• Est. Priority Date: 11/12/2002
• Status: Active Grant

• Current Assignee: Tyco Fire & Security GmbH (Johnson Controls International plc)
• Original Assignee: Sensormatic Electronics LLC (Johnson Controls International plc)
• Inventors: Buehler, Christopher J., Brock, Neil, Edulbehram, Jehanbux, Sobalvarro, Patrick
• Primary Examiner(s): Perungavoor, Sath V
• Assistant Examiner(s): Pe, Geepy
• Application Number:
  US10/706,850
• Publication Number:
  US 20040130620A1
• Time in Patent Office: 3,611 Days
• Field of Search: 348/143, 348150-156, 348169-172
• US Class Current: 348/169
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06V 10/24 Aligning, centring, orientation detection or correction of the image
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G08B 13/19602 Image analysis to detect motion of the intruder, e.g. by frame subtraction
• CPC Class Codes: G08B 13/19608 Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position
• CPC Class Codes: G08B 13/19641 Multiple cameras having overlapping views on a single scene
• CPC Class Codes: G08B 13/19671 Addition of non-video data, i.e. metadata, to video stream
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A computerized method of video analysis that includes receiving several series of video frames generated by a number of image sensors. Each image sensor has its own field-of-view which can, but does not have to, overlap with another image sensor'"'"'s field-of-view. The image sensors monitor a portion of a monitored environment. The computerized method also includes concurrently tracking, independent of calibration, multiple objects within the monitored environment as the objects move between fields-of-view, and multiple objects within one field-of-view. The tracking is based on the plurality of received series of video frames.

Claims

1. A computerized method of video analysis comprising:

receiving, at a computerized receiving device, a plurality of series of video frames generated by a plurality of image sensors, each image sensor having a field-of-view that monitors a portion of a monitored environment;

concurrently tracking, using a tracking module, a plurality of objects with respect to the monitored environment as the objects move among fields-of-view, wherein the tracking is based at least in part on a transition probability table, in which a first axis of the table represents a first set of image regions within video frames generated by the image sensors at a first point in time, a second axis of the table represents a second set of image regions within video frames generated by the image sensors at a second point in time, and each entry in the table represents a likelihood that one of the plurality of objects included in the first-axis image region corresponding to the entry will transition by the second time period to the second-axis image region corresponding to the entry;

storing a plurality of blob states over time, each state including a number of objects included in the blob and a blob signature, wherein the transition probability table comprises probabilities that objects within one blob at one instant in time correspond to objects within other blobs at other instants in time; and

predicting, concurrently with the tracking, an image sensor in the plurality of image sensors having a field-of-view in which at least one of the plurality of objects will be located at the second point in time, wherein the prediction is based on the presence of the at least one of the plurality of objects in a region in the first set of image regions at the first point in time and on the transition probability table, independent of calibration among the image sensors and the monitored environment.

2. The method of claim 1 wherein the image sensors are cameras.

3. The method of claim 1 further comprising altering the entries in the transition probability table upon analysis of additional video frames.

4. The method of claim 1 further comprising storing object data indicating correspondences between objects and blob states.

5. The method of claim 1 further comprising generating a tracking solution based on the blob states and the entries in the transition probability table.

6. The method of claim 1 generating tracking metadata including at least one of object track data, tracking solutions, object feature data and field-of-view data.

7. The method of claim 6 further comprising:

selecting a rule set to analyze generated tracking metadata; and

evaluating, using a rules engine, the tracking metadata based on the rule set.

8. The method of claim 7 further comprising selecting the rule set to monitor parking lot security.

9. The method of claim 7 further comprising selecting the rule set to detect property theft.

10. The method of claim 7 further comprising selecting the rule set to detect hazards to children.

11. The method of claim 7 further comprising selecting the rule set to monitor public safety.

12. The method of claim 7 further comprising selecting the rule set to determine merchandizing and operations statistics.

20. The method of claim 1 wherein the fields-of-view are non-overlapping.

27. The method of claim 1, further comprising:

concurrently tracking, using the tracking module and based on at least some of the received series of video frames, the plurality of objects with respect to the monitored environment based on an analysis of the monitored environment over time; and

storing in at least one entry of the transition probability table the likelihood that the object included in the corresponding image region within the first video frame corresponds to the object included in the corresponding image region within the second video frame.

29. The method of claim 1, wherein the tracking is further based at least in part on a second transition probability table, in which a first axis of the second table represents the first set of image regions at the first point in time, a second axis of the second table represents the second set of image regions at the first point in time, and each entry in the second table represents a likelihood that at least one of the plurality of objects included in the first-axis image region corresponding to the entry and at least one of the plurality of objects included in the second-axis image region corresponding to the entry correspond to the same object;

and

determining, based on the second transition probability table, a likelihood that one of the first set of image regions at least partially overlaps with one of the second set of image regions.

13. A computerized system for video analysis comprising:

a receiving module configured to receive a plurality of series of video frames, the series of video frames generated over time by a plurality of image sensors which monitor portions of a monitored environment and have a field-of-view; and

a calibration-independent tracking module in communication with the receiving module and configured to i) concurrently track a plurality of objects with respect to within the monitored environment as the objects move among fields-of-view, wherein the tracking is based at least in part on a transition probability table, in which a first axis of the table represents a first set of image regions within frames generated by the image sensors at a first point in time, a second axis of the table represents a second set of image regions within video frames generated by the image sensors at a second point in time, and each entry in the table represents a likelihood that one of the plurality of objects included in the first-axis image region corresponding to the entry will transition by the second time period to the second-axis image region corresponding to the entry, ii) store a plurality of blob states over time, each state including a number of objects included in the blob and a blob signature, wherein the transition probability table comprises probabilities that objects within one blob at one instant in time correspond to objects within other blobs at other instants in time, and iii) predict, concurrently with the tracking, an image sensor in the plurality of image sensors having a field-of-view in which at least one of the plurality of objects will be located at the second point in time, wherein the prediction is based on the presence of the at least one of the plurality of objects in a region in the first set of image regions at the first point in time and on the transition probability table, independent of calibration among the image sensors and the monitored environment.

14. The system of claim 13 wherein the tracking module is further configured to concurrently track the plurality of objects within one field-of-view based on at least some of the received series of video frames and independent of calibration among the image sensors and the monitored environment, the tracking module outputting tracking metadata.

21. The system of claim 13 wherein the fields-of-view are non-overlapping.

28. The system of claim 14 further comprising a rules engine in communication with the tracking module and receiving the tracking metadata.

15. A system for monitoring parking lot security comprising:

a receiving module configured to receive a plurality of series of video frames, the series of video frames generated over time by a plurality of image sensors which monitor portions of a monitored environment and have a field-of-view;

a calibration-independent tracking module in communication with the receiving module and configured to i) concurrently track a plurality of objects with respect to within the monitored environment as the objects move among fields-of-view, wherein the tracking is based at least in part on a transition probability table, in which a first axis of the table represents a first set of image regions within video frames generated by the image sensors at a first point in time, a second axis of the table represents a second set of image regions within video frames generated by the image sensors at a second point in time, and each entry in the table represents a likelihood that one of the plurality of objects included in the first-axis image region corresponding to the entry will transition by the second time period to the second-axis image region corresponding to the entry, ii) store a plurality of blob states over time, each state including a number of objects included in the blob and a blob signature, wherein the transition probability table comprises probabilities that objects within one blob at one instant in time correspond to objects within other blobs at other instants in time, and iii) predict, concurrently with tracking the plurality of objects, an image sensor in the plurality of image sensors having a field-of-view in which at least one of the plurality of objects will be located at the second point in time, wherein the prediction is based on the presence of the at least one of the plurality of objects in a region in the first set of image regions at the first point in time and on the transition probability table, independent of calibration among the image sensors and the monitored environment, and iii) concurrently track the plurality of objects within one field-of-view based on at least some of the received series of video frames and independent of calibration among the image sensors and the monitored environment, the tracking module outputting tracking metadata; and

a rules engine utilizing a parking lot security rule set configured to receive and evaluate the tracking metadata.

22. The system of claim 15 wherein the fields-of-view are non-overlapping.

16. A system for property theft detection comprising:

a receiving module configured to receive a plurality of series of video frames, the series of video frames generated over time by a plurality of image sensors which monitor portions of a monitored environment and have a field-of-view;

a calibration-independent tracking module in communication with the receiving module and configured to i) concurrently track a plurality of objects with respect to within the monitored environment as the objects move among fields-of-view, wherein the tracking is based at least in part on a transition probability table, in which a first axis of the table represents a first set of image regions within video frames generated by the image sensors at a first point in time, a second axis of the table represents a second set of image regions within video frames generated by the image sensors at a second point in time, and each entry in the table represents a likelihood that one of the plurality of objects included in the first-axis image region corresponding to the entry will transition by the second time period to the second-axis image region corresponding to the entry, ii) store a plurality of blob states over time, each state including a number of objects included in the blob and a blob signature, wherein the transition probability table comprises probabilities that objects within one blob at one instant in time correspond to objects within other blobs at other instants in time, and iii) predict, concurrently with tracking the plurality of objects, an image sensor in the plurality of image sensors having a field-of-view in which at least one of the plurality of objects will be located at the second point in time, wherein the prediction is based on the presence of the at least one of the plurality of objects in a region in the first set of image regions at the first point in time and on the transition probability table, independent of calibration among the image sensors and the monitored environment and iii) concurrently track the plurality of objects within one field-of-view based on at least some of the received series of video frames and independent of calibration among the image sensors and the monitored environment, the tracking module outputting tracking metadata; and

a rules engine utilizing a theft detection rule set configured to receive and evaluate the tracking metadata.

23. The system of claim 16 wherein the fields-of-view are non-overlapping.

17. A system for child hazard detection comprising:

a receiving module configured to receive a plurality of series of video frames, the series of video frames generated over time by a plurality of image sensors which monitor portions of a monitored environment and have a field-of-view;

a calibration-independent tracking module in communication with the receiving module and configured to i) concurrently track a plurality of objects with respect to within the monitored environment as the objects move among fields-of-view, wherein the tracking is based at least in part on a transition probability table, in which a first axis of the table represents a first set of image regions within video frames generated by the image sensors at a first point in time, a second axis of the table represents a second set of image regions within video frames generated by the image sensors at a second point in time, and each entry in the table represents a likelihood that one of the plurality of objects included in the first-axis image region corresponding to the entry will transition by the second time period to the second-axis image region corresponding to the entry, ii) store a plurality of blob states over time, each state including a number of objects included in the blob and a blob signature, wherein the transition probability table comprises probabilities that objects within one blob at one instant in time correspond to objects within other blobs at other instants in time, and iii) predict, concurrently with tracking the plurality of objects, an image sensor in the plurality of image sensors having a field-of-view in which at least one of the plurality of objects will be located at the second point in time, wherein the prediction is based on the presence of the at least one of the plurality of objects in a region in the first set of image regions at the first point in time and on the transition probability table, independent of calibration among the image sensors and the monitored environment and iii) concurrently track the plurality of objects within one field-of-view based on at least some of the received series of video frames and independent of calibration among the image sensors and the monitored environment, the tracking module outputting tracking metadata; and

a rules engine utilizing a child safety rule set configured to receive and evaluate the tracking metadata.

24. The system of claim 17 wherein the fields-of-view are non-overlapping.

18. A system for public safety monitoring comprising:

a receiving module configured to receive a plurality of series of video frames, the series of video frames generated over time by a plurality of image sensors which monitor portions of a monitored environment and have a field-of-view;

a calibration-independent tracking module in communication with the receiving module and configured to i) concurrently track a plurality of objects with respect to within the monitored environment as the objects move among fields-of-view, wherein the tracking is based at least in part on a transition probability table, in which a first axis of the table represents a first set of image regions within video frames generated by the image sensors at a first point in time, a second axis of the table represents a second set of image regions within video frames generated by the image sensors at a second point in time, and each entry in the table represents a likelihood that one of the plurality of objects included in the first-axis image region corresponding to the entry will transition by the second time period to the second-axis image region corresponding to the entry, ii) store a plurality of blob states over time, each state including a number of objects included in the blob and a blob signature, wherein the transition probability table comprises probabilities that objects within one blob at one instant in time correspond to objects within other blobs at other instants in time, and iii) predict, concurrently with tracking the plurality of objects, an image sensor in the plurality of image sensors having a field-of-view in which at least one of the plurality of objects will be located at the second point in time, wherein the prediction is based on the presence of the at least one of the plurality of objects in a region in the first set of image regions at the first point in time and on the transition probability table, independent of calibration among the image sensors and the monitored environment and iii) concurrently track the plurality of objects within one field-of-view based on at least some of the received series of video frames and independent of calibration among the image sensors and the monitored environment, the tracking module outputting tracking metadata; and

a rules engine utilizing a public safety monitoring rule set configured to receive and evaluate the tracking metadata.

25. The system of claim 18 wherein the fields-of-view are non-overlapping.

19. A system for merchandizing and operations statistical analysis comprising:

a receiving module configured to receive a plurality of series of video frames, the series of video frames generated over time by a plurality of image sensors which monitor portions of a monitored environment and have a field-of-view;

a calibration-independent tracking module in communication with the receiving module and configured to i) concurrently track a plurality of objects with respect to within the monitored environment as the objects move among fields-of-view, wherein the tracking is based at least in part on a transition probability table, in which a first axis of the table represents a first set of image regions within video frames generated by the image sensors at a first point in time, a second axis of the table represents a second set of image regions within video frames generated by the image sensors at a second point in time, and each entry in the table represents a likelihood that one of the plurality of objects included in the first-axis image region corresponding to the entry will transition by the second time period to the second-axis image region corresponding to the entry, ii) store a plurality of blob states over time, each state including a number of objects included in the blob and a blob signature, wherein the transition probability table comprises probabilities that objects within one blob at one instant in time correspond to objects within other blobs at other instants in time, and iii) predict, concurrently with tracking the plurality of objects, an image sensor in the plurality of image sensors having a field-of-view in which at least one of the plurality of objects will be located at the second point in time, wherein the prediction is based on the presence of the at least one of the plurality of objects in a region in the first set of image regions at the first point in time and on the transition probability table, independent of calibration among the image sensors and the monitored environment and iii) concurrently track the plurality of objects within one field-of-view based on at least some of the received series of video frames and independent of calibration among the image sensors and the monitored environment, the tracking module outputting tracking metadata; and

a rules engine utilizing a merchandizing and operations statistical rule set configured to receive and evaluate the tracking metadata.

26. The system of claim 19 wherein the fields-of-view are non-overlapping.