Computerized method and apparatus for determining field-of-view relationships among multiple image sensors

• US 7,286,157 B2
• Filed: 09/11/2003
• Issued: 10/23/2007
• Est. Priority Date: 09/11/2003
• Status: Active Grant

• Current Assignee: Johnson Controls Tyco IP Holdings LLP (Johnson Controls International plc)
• Original Assignee: Intellivid Corporation (Johnson Controls International plc)
• Inventors: Buehler, Christopher J.
• Primary Examiner(s): PHILIPPE, GIMS S
• Application Number:
  US10/660,955
• Publication Number:
  US 20050058321A1
• Time in Patent Office: 1,503 Days
• Field of Search: 348/143, 348/411, 348/153, 348/161, 348/135, 382/100, 382/103, 382/246, 382/278
• US Class Current: 348/143
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A computerized method of image analysis that includes receiving first image data for a plurality of first video frames representing a first scene. Each first video frame is composed of a plurality of image regions, and a first object is present in an image region of at least one first video frame. Second image data is received for a plurality of second video frames representing a second scene. Each second video frame is composed of a plurality of image regions, and a second object is present in an image region of at least one second video frame. The method also includes determining a relationship between a first image region and a second image region based on a probabilistic correlation between occurrences of the first object being present in the first image region and occurrences of the second object being present in the second image region.

Claims

1. A computerized method of image analysis comprising:

a) receiving first image data for a plurality of first video frames representing a first scene, wherein (i) each first video frame comprises a plurality of image regions and (ii) a first object is present in an image region of at least one first video frame;

b) receiving second image data for a plurality of second video frames representing a second scene, wherein (i) each second video frame comprises a plurality of image regions and (ii) a second object is present in an image region of at least one second video frame; and

c) determining (i) a relationship between first and second image regions of the plurality of image regions of the first and second video frames based on a probabilistic correlation between occurrences of the first object being present in the first image region and occurrences of the second object being present in the second image region and (ii) a lift associated with the first object being present in the first image region and the second object being present in the second image region.

2. The method of claim 1 wherein step c) further comprises c-1) determining a conditional probability that the first object is present in the first image region given that the second object is present in the second image region.

3. The method of claim 1 wherein step c) further comprises c-1) determining a joint probability that the first object is present in the first object region and that the second object is present in the second image region.

4. The method of claim 1 wherein step c) further comprises c-1) determining a correlation coefficient associated with the first object being present in the first image region and the second object being present in the second image region.

5. The method of claim 1 wherein the first image data originates from a first camera having a first field-of-view, and the second image data originates from a second camera having a second field-of-view.

6. The method of claim 5 wherein the first field-of-view and the second field-of-view at least in part overlap.

7. The method of claim 5 wherein the first field-of-view and the second field-of-view do not overlap.

8. The method of claim 1 wherein the first image region is one of the image regions that comprises the first video frames and the second image region is one of the image regions that comprises the second video frames.

9. The method of claim 1 wherein the first image region and the second image region are both image regions that comprise one of the first video frames and the second video frames.

10. The method of claim 1 wherein the first object and the second object are the same object.

11. The method of claim 1 wherein the first object and the second object are different objects.

12. The method of claim 1 wherein step c) comprises determining a temporal relationship between occurrences of the presence of the first object in the first image region and occurrences of the presence of the second object in the second image region.

13. The method of claim 1 wherein step c) comprises determining a spatial relationship between the first image region and the second image region.

14. The method of claim 1 further comprising d) counting the number of occurrences of any object being present in the first and second image regions.

15. The method of claim 1 further comprising d) counting the number of occurrences of any object being concurrently present in both the first and second image regions.

16. The method of claim 1 further comprising d) counting, for each of a predetermined number of earlier time instants, the number of occurrences of both any object being currently present in the first image region and any object being present in the second image region at the earlier time instants.

17. The method of claim 1 further comprising d) correlating the first object in the first video frames with a second object present in the second video frames based on the determined relationship.

18. A surveillance system comprising:

a) a first memory module for receiving first image data for a plurality of first video frames representing a first scene, wherein (i) each first video frame comprises a plurality of image regions and (ii) a first object is present in an image region of at least one first video frame;

b) a second memory module for receiving second image data for a plurality of second video frames representing a second scene, wherein (i) each second video frame comprises a plurality of image regions and (ii) a second object is present in an image region of at least one second video frame; and

c) a processing module for determining (i) a relationship between first and second image regions of the plurality of image regions of the first and second video frames based on a probabilistic correlation between occurrences of the first object being present in the first image region and occurrences of the second object being present in the second image region and (ii) a lift associated with the first object being present in the first image region and the second object being present in the second image region.

19. The surveillance system of claim 18 wherein the processing module further determines a conditional probability that the first object is present in the first image region given that the second object is present in the second image region.

20. The surveillance system of claim 18 wherein the processing module further determines a joint probability that the first object is present in the first object region and that the second object is present in the second image region.

21. The surveillance system of claim 18 wherein the processing module further determines a correlation coefficient associated with the first object being present in the first image region and the second object being present in the second image region.

22. The surveillance system of claim 18 wherein the first image data originates from a first camera having a first field-of-view, and the second image data originates from a second camera having a second field-of-view.

23. The surveillance system of claim 22 wherein the first field-of-view and the second field-of-view at least in part overlap.

24. The surveillance system of claim 22 wherein the first field-of-view and the second field-of-view do not overlap.

25. The surveillance system of claim 18 wherein the first image region is one of the image regions that comprises the first video frames and the second image region is one of the image regions that comprises the second video frames.

26. The surveillance system of claim 18 wherein the first image region and the second image region are both image regions that comprise one of the first video frames and the second video frames.

27. The surveillance system of claim 18 wherein the first object and the second object are the same object.

28. The surveillance system of claim 18 wherein the first object and the second object are different objects.

29. The surveillance system of claim 18 wherein the processing module determines a temporal relationship between occurrences of the presence of the first object in the first image region and occurrences of the presence of the second object in the second image region.

30. The surveillance system of claim 18 wherein the processing module determines a spatial relationship between the first image region and the second image region.

31. The surveillance system of claim 18 wherein the processing module further counts the number of occurrences of any object being present in the first and second image regions.

32. The surveillance system of claim 18 wherein the processing module further counts the number of occurrences of any object being concurrently present in both the first and second image regions.

33. The surveillance system of claim 18 wherein the processing module further counts, for each of a predetermined number of earlier time instants, the number of occurrences of both any object being currently present in the first image region and any object being present in the second image region at the earlier time instants.

34. The surveillance system of claim 18 wherein the processing module further correlates the first object in the first video frames with a second object present in the second video frames based on the determined relationship.

35. A computerized method of image analysis comprising:

a) receiving first image data for a plurality of first video frames representing a first scene, wherein (i) each first video frame comprises a plurality of image regions and (ii) a first object is present in an image region of at least one first video frame;

b) receiving second image data for a plurality of second video frames representing a second scene, wherein (i) each second video frame comprises a plurality of image regions and (ii) a second object is present in an image region of at least one second video frame;

c) providing a plurality of attributes for the first and second objects;

d) counting for each of the plurality of attributes, the number of occurrences of at least one of the first and second objects having such attribute being present in the first and second image regions; and

e) determining a relationship between first and second image regions of the plurality of image regions of the first and second video frames based on a probabilistic correlation between occurrences of the first object being present in the first image region and occurrences of the second object being present in the second image region.

36. The method of claim 35 wherein step d) further comprises counting, for each of the plurality of attributes, the number of occurrences of one of the first and second objects with each attribute being present in the first image region concurrently with one of the first and second objects having each attribute being present in second image regions.

37. The method of claim 35 wherein step d) further comprises counting, for each of a predetermined number of earlier time instants, and for each of the plurality of attributes, the number of occurrences of both any one of the first and second objects having such attribute being currently present in the first image region and any one of the first and second objects having each attribute being present in the second image region at the earlier time instants.

38. A computerized method of image analysis comprising:

a) receiving first image data for a plurality of first video frames representing a first scene, wherein (i) each first video frame comprises a plurality of image regions and (ii) a first object is present in an image region of at least one first video frame;

b) receiving second image data for a plurality of second video frames representing a second scene, wherein (i) each second video frame comprises a plurality of image regions and (ii) a second object is present in an image region of at least one second video frame;

c) determining a relationship between first and second image regions of the plurality of image regions of the first and second video frames based on a probabilistic correlation between occurrences of the first object being present in the first image region and occurrences of the second object being present in the second image region.

d) determining that the first object present in one first video frame was not present in a preceding first video frame; and

e) correlating the first object with the second object present in the second video frames based on the determined relationship.

39. A computerized method of image analysis comprising:

a) receiving first image data for a plurality of first video frames representing a first scene, wherein (i) each first video frame comprises a plurality of image regions and (ii) a first object is present in an image region of at least one first video frame;

b) receiving second image data for a plurality of second video frames representing a second scene, wherein (i) each second video frame comprises a plurality of image regions wherein each image region has a location in one of the plurality of first and second video frames that is substantially the same across the plurality of first and second video frames and (ii) a second object is present in an image region of at least one second video frame; and

c) determining a relationship between first and second image regions of the plurality of image regions of the first and second video frames based on a probabilistic correlation between occurrences of the first object being present in the first image region and occurrences of the second object being present in the second image region.

40. The method of claim 39 further comprising d) maintaining an object list identifying at least the locations of the first object and the second object.

41. The method of claim 40 further comprising:

e) selecting an object present in one of the first and second video frames;

f) determining whether the selected object is one of the first object, the second object, and a third object based at least in part on the object list and the determined relationship.

42. A surveillance system comprising:

a) a first memory module for receiving first image data for a plurality of first video frames representing a first scene, wherein (i) each first video frame comprises a plurality of image regions and (ii) a first object is present in an image region of at least one first video frame;

b) a second memory module for receiving second image data for a plurality of second video frames representing a second scene, wherein (i) each second video frame comprises a plurality of image regions and (ii) a second object is present in an image region of at least one second video frame; and

c) a processing module for (i) providing a plurality of attributes for objects, (ii) counting for each of the plurality of attributes, the number of occurrences of an object having such attribute being present in the first and second image regions, and (iii) determining a relationship between first and second image regions of the plurality of image regions of the first and second video frames based on a probabilistic correlation between occurrences of the first object being present in the first image region and occurrences of the second object being present in the second image region.

43. The surveillance system of claim 42 wherein the processing module further counts, for each of the plurality of attributes, the number of occurrences of one of the first and second objects with each attribute being present in the first image region concurrently with one of the first and second objects having each attribute being present in second image regions.

44. The surveillance system of claim 42 wherein the processing module further counts, for each of a predetermined number of earlier time instants, and for each of the plurality of attributes, the number of occurrences of both any one of the first and second objects having such attribute being currently present in the first image region and any one of the first and second objects having each attribute being present in the second image region at the earlier time instants.

45. A surveillance system comprising:

a) a first memory module for receiving first image data for a plurality of first video frames representing a first scene, wherein (i) each first video frame comprises a plurality of image regions and (ii) a first object is present in an image region of at least one first video frame;

b) a second memory module for receiving second image data for a plurality of second video frames representing a second scene, wherein (i) each second video frame comprises a plurality of image regions and (ii) a second object is present in an image region of at least one second video frame; and

c) a processing module for (i) determining a relationship between first and second image regions of the plurality of image regions of the first and second video frames based on a probabilistic correlation between occurrences of the first object being present in the first image region and occurrences of the second object being present in the second image region, (ii) determining that the first object present in one first video frame was not present in a preceding first video frame and (iii) correlating the first object with the second object present in the second video frames based on the determined relationship.

46. A surveillance system comprising:

a) a first memory module for receiving first image data for a plurality of first video frames representing a first scene, wherein (i) each first video frame comprises a plurality of image regions and (ii) a first object is present in an image region of at least one first video frame;

b) a second memory module for receiving second image data for a plurality of second video frames representing a second scene, wherein (i) each second video frame comprises a plurality of image regions, wherein each image region has a location in one of the plurality of first and second video frames that is substantially the same across the plurality of first and second video frames, and (ii) a second object is present in an image region of at least one second video frame; and

c) a processing module for determining a relationship between first and second image regions of the plurality of image regions of the first and second video frames based on a probabilistic correlation between occurrences of the first object being present in the first image region and occurrences of the second object being present in the second image region.

47. The surveillance system of claim 46 wherein the processing module further maintains an object list identifying at least the locations of the first object and the second object.

48. The surveillance system of claim 47 wherein the processing module further selects an object present in one of the first and second video frames and determines whether the selected object is one of the first object, the second object, and a third object based at least in part on the object list and the determined relationship.