Dynamic object classification

• US 8,934,709 B2
• Filed: 03/03/2009
• Issued: 01/13/2015
• Est. Priority Date: 03/03/2008
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: VideoIQ, Inc. (Motorola Solutions, Inc.)
• Inventors: Saptharishi, Mahesh, Lisin, Dimitri A., Lipchin, Aleksey, Reyzin, Igor
• Primary Examiner(s): Haskins, Twyler
• Assistant Examiner(s): Chon, Peter
• Application Number:
  US12/397,287
• Publication Number:
  US 20090244291A1
• Time in Patent Office: 2,142 Days
• Field of Search: 382/103, 382/190, 382/224, 382/159, 382/225, 382/226, 382/227, 382/228, 706/20
• US Class Current: 382/159
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/631 Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
• CPC Class Codes: H04N 23/66 Remote control of cameras or camera parts, e.g. by remote control devices
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A camera system comprises an image capturing device and an object classification module connected to the image capturing device. The image capturing device has a field of view and produces image data representing an image of the field of view. The object classification module is operable to determine whether an object in an image is a member of an object class. The object classification module includes N decision steps configured in a cascade configuration, wherein at least one of the N decision steps is operable to (a) accept an object as a member of the object class, (b) reject an object as a member of the object class, and (c) call on a next step to determine whether an object is a member of the object class.

Claims

1. A method of improving performance of video analytics for a camera system in response to a detection preference of a system user, comprising:

while the camera system is already in in-field use, receiving image data representing multiple images of a scene of a field of view of the camera system, the multiple images including representations of multiple objects, a first set of the multiple objects having members of an object class, and a second set of the multiple objects not having members of the object class;

while the camera system is already in in-field use, using video analytics implemented with a general classifier that performs general classifier steps in analyzing the received image data to produce a general classification determination classifying the multiple objects as either members or non-members of the object class;

while the camera system is already in in-field use, generating mistake metadata in response to acknowledgement by the system user that the general classification determination resulted in a mistaken classification determination based on the detection preference of the system user; and

while the camera system is already in in-field use, improving video analytics performance based on the mistake metadata by performing a specialization step in addition to the general classification steps performed, the specialization step producing for presentation to the system user a specialized classification determination of the multiple objects classified by the general classifier so as to reduce a number of future mistaken classification determinations.

2. A method according to claim 1, wherein the general classifier includes N decision steps configured in a cascade configuration, and wherein the specialization step comprises an additional step constructed from the mistake metadata to add to the N decision steps of the general classifier, thereby resulting in an object classifier with N+1 steps.

3. A method according to claim 1, wherein the mistaken classification determination is a false alarm representing an instance when the video analytics determine that an object of the second set is a member of the object class.

4. A method according to claim 1, further comprising:

displaying on a display a representation of the mistaken classification determination for acknowledgement by the system user.

5. A method according to claim 1, wherein the general classifier before the improving of video analytics performance represents a deployed classifier, and the general classifier after the improving of video analytics performance represents a specialized classifier, the method further comprising validating the specialized classifier.

6. A method according to claim 5, wherein the validating the specialized classifier comprises passive validation of the specialized classifier, the passive validation including:

classifying a group of objects with the specialized classifier;

classifying the group of objects with the deployed classifier, the deployed classifier generating false alarms and true alarms;

presenting the false alarms and true alarms to the system user for acknowledgment;

generating feedback information based on system user acknowledgment of the false alarms, the true alarms, or both the false alarms and the true alarms; and

comparing the classifications of the specialized classifier with the feedback information.

7. A method according to claim 5, wherein the validating the specialized classifier comprises active validation of the specialized classifier, the active validation including:

classifying a group of objects with the deployed classifier;

classifying the group of objects with the specialized classifier, wherein some of the classifications of the specialized classifier are inconsistent with the classifications of the deployed classifier; and

presenting representations of the inconsistent classifications of the specialized classifier to the system user for validation.

8. A method according to claim 1, wherein the camera system includes multiple image capturing devices deployed at a site in which the image capturing devices have different fields of view, the field of view of the camera system corresponding to the different fields of view of the image capturing devices, wherein the improving video analytics performance includes modifying the general classifier to form a site-specific classifier.

9. A method according to claim 1, wherein the camera system includes an image capturing device that produces the image data, wherein the improving video analytics performance includes modifying the general classifier to form a camera-specific classifier corresponding to the image capturing device.

10. A method according to claim 1, wherein the camera system is one of multiple camera systems connected to a central feedback server over a network, further comprising:

communicating the mistake metadata to the central feedback server, wherein the mistake metadata is used for improving video analytics performance of the multiple camera systems.

11. A method according to claim 10, wherein the general classifier includes a stage representing a feature/transformation combination that includes a feature and a discriminant function, and the improving video analytics performance includes using the mistake metadata to select the feature from a set of possible features and to select the discriminant function from a set of possible discriminant functions, the feature and the discriminant function of the feature/transformation combination maximizing object classification performance of the general classifier.

12. A method according to claim 11, wherein the mistake metadata is communicated to a central feedback server that selects the feature and the discriminant function.

13. A method according to claim 11, wherein the mistake metadata is used to modify a feature/transformation combination of an existing stage of the object classifier.

14. A method according to claim 1, wherein the camera system is connected to a central feedback server over a network, further comprising:

communicating the mistake metadata to the central feedback server that uses the mistake metadata to improve video analytics performance of the camera system.

15. A method according to claim 1, wherein the object class corresponds to a new object class of a new object classifier that is to be included in an object classification module of the video analytics, and the improving step includes constructing the new object classifier based on a deployed object classifier of the object classification module, the deployed object classifier determining whether the multiple objects are members or non-members of a deployed object class, the constructing step including:

using the deployed object classifier to identify the objects of the first set as members of the deployed object class;

presenting on a display representations of the identified objects, wherein the system user labels the identified objects as either members or non-members of the new object class;

generating the mistake metadata based on the labels given by the user; and

forming the new object classifier in which the new object classifier is characterized as a version of the deployed object classifier that is modified based on the mistake metadata.

16. A method according to claim 15, wherein the new object class is a subclass of the object class.

17. A method according to claim 1, wherein the object class represents a general class of objects of interest.

18. A field-deployed camera system, comprising:

an object classification module operable, while the camera system is already in in-field use, to receive image data including representations of objects captured in images of a field of view of an image capturing device, the object classification module including an object classifier that is operable to determine whether the objects in the images are members of an object class, wherein the object classifier generates mistaken classifications;

a user station having a display for presenting the images of the objects to a user, the user station being operable to present on the display representations of the mistaken classifications generated by the object classifier, wherein the user station is operable to produce user feedback information in response to user acknowledgement of the mistaken classifications, the user feedback generating mistake metadata; and

a classifier evolution module receiving the mistake metadata and being operable, while the camera system is already in in-field use, to modify the object classifier using the mistake metadata so as to reduce the number of future mistaken classifications, the classifier evolution module thereby generating a specialized classifier.

19. A field-deployed camera system according to claim 18, further comprising:

an object detection module connected to the object classification module, the object detection module receiving the image data and being operable, while the camera system is already in in-field use, to detect whether objects are in the images of the field of view, wherein the object detection module is operable to improve object detection performance based on the mistake metadata.

20. A field-deployed camera system according to claim 18, wherein the object classifier includes N decision steps configured in a cascade configuration, and wherein the specialized classifier includes the N decision steps and an additional step constructed based on the mistake metadata, thereby resulting in a specialized classifier having N+1 steps.

21. A field-deployed camera system according to claim 18, wherein the specialized classifier includes a stage corresponding to a feature/transformation combination, the feature/transformation combination corresponding to a feature extracted from an image and a discriminant function for mapping the feature to a scalar value, the feature being selected from a set of possible features and the discriminant function being selected from a set of possible discriminant functions, the feature and the discriminant function being selected so as to maximize object classification performance of the specialized classifier.

22. A field-deployed camera system according to claim 18, wherein the camera system includes the image capturing device, and wherein the classifier evolution module is located remotely from the image capturing device, the object classification module, and the user station.

23. A field-deployed camera system according to claim 18, wherein the camera system includes the image capturing device, and wherein the image capturing device, the object classification module, the user station, and the classifier evolution module are colocated at a same site.

24. A method of improving performance of video analytics for a camera system in response to a detection preference of a system user, comprising:

while the camera system is already in in-field use, receiving image data representing multiple images of a scene of a field of view of the camera system, the multiple images including representations of multiple objects, a first set of the multiple objects having members of an object class, and a second set of the multiple objects not having members of the object class;

while the camera system is already in in-field use, using video analytics implemented with a general classifier that performs general classifier steps in analyzing the received image data to produce a general classification determination classifying the multiple objects as either members or non-members of the object class;

while the camera system is already in in-field use, generating a user feedback indication in response to acknowledgement by the system user that the general classification determination resulted in a mistaken classification determination based on detection preference of the system user; and

while the camera system is already in in-field use, using the user feedback indication and video analytics implemented with the general classifier to perform the general classification steps in analyzing the received image data to construct a feedback-based classifier that performs classification based on the system user feedback;

while the camera system is already in in-field use, testing operational performance of the feedback-based classifier in analyzing the received image data to produce a feedback-based classification determination classifying the multiple objects as either members or non-members of the object class; and

while the camera system is already in in-field use, evaluating whether the feedback-based classification determination in comparison with the general classification determination is more consistent with the detection preference of the system user and thereby validates an improved performance of the video analytics.

25. A method according to claim 24, further comprising, upon validation of improved performance of the video analytics, deploying the feedback based classifier in place of the general classifier.

26. A method according to claim 24, wherein the feedback-based classifier includes the general classifier with an additional specialization step that modifies the output of the general classifier to improve video analytics performance based on the user feedback indication.

27. A method of improving performance of video analytics for a camera system in response to a detection preference of a system user, comprising:

while the camera system is already in in-field use, receiving image data representing multiple images of a scene of a field of view of the camera system, the multiple images including representations of multiple objects, a first set of the multiple objects having members of an object class, and a second set of the multiple objects not having members of the object class;

while the camera system is already in in-field use, using video analytics implemented with a general classifier that performs general classifier steps in analyzing the received image data to produce a general classification determination classifying the multiple objects as either members or non-members of the object class;

while the camera system is already in in-field use, generating mistake metadata in response to acknowledgement by the system user that the general classification determination resulted in a mistaken classification determination based on the detection preference of the system user; and

while the camera system is already in in-field use, improving video analytics performance based on the mistake metadata by performing a specialization step in addition to the general classification steps performed, the specialization step including

labeling the received image data that does not include the mistaken classification as correctly classified image data,

labeling the received image data that does the mistaken classification as incorrectly classified image data,

generating a new general classifier that is only trained with the correctly classified image data, and

replacing the general classifier with the new general classifier.

28. The method according to claim 27, where the mistaken classification is a false positive.

29. The method according to claim 27, where the mistaken classification is a false negative.

30. The method according to claim 27, where the camera system includes a plurality of cameras each trained with the general classifier, and wherein the specialization step includes distributing the new general classifier to the plurality of cameras to replace the general classifier of each camera among the plurality of cameras.