Object matching for tracking, indexing, and search

• US 8,224,029 B2
• Filed: 03/03/2009
• Issued: 07/17/2012
• Est. Priority Date: 03/03/2008
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: VideoIQ, Inc. (Motorola Solutions, Inc.)
• Inventors: Saptharishi, Mahesh, Lisin, Dimitri A.
• Primary Examiner(s): PERUNGAVOOR, SATHYANARAYA V
• Application Number:
  US12/397,276
• Publication Number:
  US 20090245573A1
• Time in Patent Office: 1,232 Days
• Field of Search: 382/103
• US Class Current: 382/103
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/631 Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
• CPC Class Codes: H04N 23/66 Remote control of cameras or camera parts, e.g. by remote control devices
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A camera system comprises an image capturing device, object detection module, object tracking module, and match classifier. The object detection module receives image data and detects objects appearing in one or more of the images. The object tracking module temporally associates instances of a first object detected in a first group of the images. The first object has a first signature representing features of the first object. The match classifier matches object instances by analyzing data derived from the first signature of the first object and a second signature of a second object detected in a second image. The second signature represents features of the second object derived from the second image. The match classifier determine whether the second signature matches the first signature. A training process automatically configures the match classifier using a set of possible object features.

Claims

1. A camera system comprising:

an object detection module receiving image data produced by an image capturing device, the image data representing multiple images of a field of view of the image capturing device, the object detection module being operable to detect objects appearing in one or more of the multiple images;

an object tracking module connected to the object detection module and operable to temporally associate instances of a first object detected in a first group of the multiple images, the first object having a first signature representing features of the first object derived from the images of the first group; and

a match classifier connected to the object tracking module for matching object instances and including a set of matching and non-matching parameters populating a feature classification space, the match classifier being operable to match object instances by analyzing data derived from the first signature of the first object and a second signature of a second object, the second object being detected in a second image distinct from the images of the first group, and the second signature representing features of the second object derived from the second image, the match classifier analyzing with reference to the set of match and non-match parameters data derived from combinations of the features represented by the first and second signatures to determine whether the second signature matches the first signature and thereby indicate whether the first and second objects correspond to the same physical object.

2. The camera system of claim 1, wherein the second object is an instance of the first object.

3. The camera system of claim 1, wherein the second signature is generated in real-time.

4. The camera system of claim 1, wherein the match classifier includes N decision steps configured in a cascade configuration, the N decisions steps operable to determine whether the first object and the second object match, wherein at least one of the N decision steps is operable to either (a) accept the first object and the second object as a match, (b) reject the first object and the second object as a match, or (c) call on a next step to determine whether the second object matches the first object.

5. The camera system of claim 4, wherein said at least one of the N decision steps includes a stage for mapping a portion of the data to a scalar value that correlates to a match confidence.

6. The camera system of claim 5, wherein the first signature represents appearance features of the first object and includes a first edge orientation histogram of the first object, and the second signature represents appearance features of the second object and includes a second edge orientation histogram of the second object.

7. The camera system of claim 6, wherein the first and second edge orientation histograms are produced using a steerable filter.

8. The camera system of claim 6, wherein the first and second edge orientation histograms are combined to produce the portion of the data.

9. The camera system of claim 8, wherein the first and second edge orientation histograms are combined by concatenation.

10. The camera system of claim 6, wherein the first signature includes a first group of multiple edge orientation histograms corresponding to a first set of different spatial regions of the first object, and the second signature includes a second group of multiple edge orientation histograms corresponding to a second set of different spatial regions of the second object, the different spatial regions of the second set corresponding to the different spatial regions of the first set.

11. The camera system of claim 6, wherein the first signature includes a first hue histogram and a first intensity histogram of the first object, and the second signature includes a second hue histogram and a second intensity histogram of the second object.

12. The camera system of claim 6, wherein the first signature includes a first group of multiple hue histograms corresponding to a first set of different spatial regions of the first object, and the second signature includes a second group of multiple hue histograms corresponding to a second set of different spatial regions of the second object, the different spatial regions of the second set corresponding to the different spatial regions of the first set.

13. The camera system of claim 6, wherein the first signature includes a first group of multiple intensity histograms corresponding to a first set of different spatial regions of the first object, and the second signature includes a second group of multiple intensity histograms corresponding to a second set of different spatial regions of the second object, the different spatial regions of the second set corresponding to the different spatial regions of the first set.

14. The camera system of claim 1, wherein the match classifier includes N decision steps configured in a cascade configuration, each of the N decision steps including one or more stages for mapping portions of the data to scalar values, wherein a first one of the stages includes a first discriminant function for determining a first scalar value, a second one of the stages includes a second discriminant function for determining a second scalar value, and the first and second discriminant functions are selected from a group consisting of plural discriminant functions of different types.

15. The camera system of claim 1, wherein the first signature is a cumulative signature of the first object derived from the images of the first group, and the second signature is single-frame signature of the second object derived from the second image.

16. The camera system of claim 15, wherein the cumulative signature of the first object is a weighted average of single-frame signatures of the first object derived from the images of the first group.

17. The camera system of claim 1, wherein the object tracking module includes a motion modeling module operable to estimate changes of location of objects between sequential images of the field of view, wherein the motion modeling module is operable to be used in conjunction with the match classifier to determine whether the second object is an instance of the first object.

18. The camera system of claim 1, further comprising multiple image capturing devices connected to one another in a network, the multiple image capturing devices including first and second image capturing devices, the first group of images being captured by the first image capturing device, and the second image being captured by the second image capturing device.

19. The camera system of claim 18, wherein the first image capturing device includes a housing, the match classifier is contained within the housing, and the second image capturing device is operable to transmit to the match classifier of the first image capturing device the second signature of the second object.

20. The camera system of claim 19, wherein the match classifier is operable to determine that the second signature transmitted from the second image capturing device and the first signature of the first object match, and the first image capturing device is operable to notify the second image capturing device of the match.

21. The camera system of claim 18, wherein the match classifier is part of a processor that is remote from the first and second image capturing devices, and the first and second image capturing devices are operable to transmit, respectively, the first and second signatures to the processor.

22. The camera system of claim 21, wherein the processor is operable to transmit a notification to one or both of the first and second image capturing devices indicating that the first and second signatures match.

23. The camera system of claim 1, wherein the second object is one of multiple objects detected in the second image, and the first object is one of multiple tracked objects being tracked by the object tracking module, the camera system further comprising:

an object classification module, the object classification module being operable to classify objects as members or non-members of an object class; and

an object categorization module connected to the match classifier and the object classification module, the object categorization module being operable to determine an order in which the match classifier attempts to match the tracked objects with the multiple objects detected in the second image, the order being based on classifications of the tracked objects determined by the object classification module.

24. The camera system of claim 1, wherein the first signature is a cumulative signature representing a weighted average of single-frame signatures of the first object derived from the images of the first group, the second signature represents a single-frame signature of the second object derived from the second image, and the match classifier determines that the second object is an instance of the first object.

25. The camera system of claim 24, further comprising:

a database for storing average signatures of detected objects, the average signatures being index elements of the detected objects; and

an object indexing module connected to the match classifier and the database, the object indexing module being operable to generate a first average signature for the first object, the first average signature representing an average of the single-frame signatures of the first object, wherein the object indexing module is operable to supply the first average signature to the database for storage.

26. The camera system of claim 25, wherein said data derived from the first signature of the first object and the second signature of the second object is a first set of data, the object indexing module is operable to call on the match classifier to determine whether the second signature of the second object is to contribute to the first average signature, and the match classifier is operable to analyze a second set of data representing the first signature and the first average signature to determine whether the first signature and the first average signature match.

27. The camera system of claim 26, wherein the object indexing module is operable to update the first average signature with contributions from the second signature when the match classifier determines that the first signature matches the first average signature.

28. The camera system of claim 26, wherein the object indexing module is operable to generate a second average signature for the first object when the match classifier determines that the first signature does not match the first average signature, the second average signature includes contributions from the second signature, and the object indexing module is operable to supply the first and second average signatures to the database for storage.

29. The camera system of claim 28, wherein the first average signature represents a first view of the first object, and the second average signature represents a second view of the first object.

30. The camera system of claim 1, further comprising:

a database for storing signatures of detected objects, the database providing an index to images of the detected objects;

a user interface including a display for presenting the multiple images of the field of view to a user, the user interface including an input device enabling user selection of an object appearing in one or more of the multiple images, wherein an image of the first group including a representation of the first object is presented on the display, and the first object is selected by the user; and

a search module connected to the user interface, the database, and the match classifier, the search module being operable to search the signatures stored in the database to identify whether the first object appears in a second group of images of the detected objects, the search module being operable to call on the match classifier to determine whether the first signature of the first object matches one or more of the signatures stored in the database.

31. The camera system of claim 30, wherein the match classifier is operable to generate confidence levels of matches made between the first signature and the signatures stored in the database, and the search module is operable to generate a list of matches organized according to confidence levels.

32. The camera system of claim 31, wherein the signatures stored in the database include average signatures of the detected objects representing averages of single-frame signatures derived from images of the second group in which the detected objects appear, and the signatures stored in the database include fast signatures of the detected objects representing simplified versions of the average signatures.

33. The camera system of claim 32, wherein the first object includes a fast signature representing a simplified version of the first signature, and the search module is operable to compare the fast signature of the first object with the fast signatures stored in the database to identify a group of the detected objects that do not correspond to the first object.

34. The camera system of claim 33, wherein said data derived from the first signature of the first object and the second signature of the second object is a first set of data, and the match classifier is operable to analyze a second set of data representing the first signature of the first object and the average signatures of the detected objects that are not members of the group that do not correspond to the first object.

35. The camera system of claim 31, wherein when the match classifier determines that the first signature matches a signature of a first one of the detected objects, the search module is operable to retrieve image data representing an image of said first one of the detected objects for transmission to the user interface.

36. The camera system of claim 31, further comprising multiple image capturing devices connected to one another in a network, the multiple image capturing devices including first and second image capturing devices.

37. The camera system of claim 36, wherein the images of the first group are captured by the first image capturing device, and the images of the second group are captured by the second image capturing device.

38. The camera system of claim 37, wherein the second image capturing device includes a housing, and the match classifier is contained within the housing.

39. The camera system of claim 37, wherein the match classifier is part of a processor remote from the first and second image capturing devices.

40. The camera system of claim 1, further comprising:

a user interface including a display for displaying objects detected by the object detection module and an input device operable to select objects detected by the object detection module in response to user commands; and

a search module connected to the match classifier and the user interface, the search module being operable to retrieve stored images of objects selected by the input device, and the search module being operable to call on the match classifier to locate a stored image of a selected object.

41. The camera system of claim 40, further comprising an object indexing module connected to the match classifier and the object detection module, the object indexing module being operable to generate one or more index elements for objects detected by the object detection module, the object indexing module being operable to call on the match classifier to determine whether to generate a new index element for a detected object.

42. The camera system of claim 1, wherein the manner in which the match classifier operates to match object instances has been determined at least in part by a training process that automatically configures the match classifier using a set of possible object features.