DYNAMIC OBJECT CLASSIFICATION

• US 20090244291A1
• Filed: 03/03/2009
• Published: 10/01/2009
• Est. Priority Date: 03/03/2008
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: VideoIQ, Inc. (Motorola Solutions, Inc.)
• Inventors: Reyzin, Igor, Lisin, Dimitri A., Lipchin, Aleksey, Saptharishi, Mahesh
• Granted Patent: US 8,934,709 B2
• Time in Patent Office: Days
• US Class Current: 348/187
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/631 Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
• CPC Class Codes: H04N 23/66 Remote control of cameras or camera parts, e.g. by remote control devices
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A camera system comprises an image capturing device and an object classification module connected to the image capturing device. The image capturing device has a field of view and produces image data representing an image of the field of view. The object classification module is operable to determine whether or not an object in an image is a member of an object class. The object classification module includes N decision steps configured in a cascade configuration, wherein at least one of the N decision steps is operable to (a) accept an object as a member of the object class, (b) reject an object as a member of the object class, and (c) call on a next step to determine whether or not an object is a member of the object class.

Claims

1. A camera system comprising:

an image capturing device having a field of view and producing image data representing an image of the field of view; and

an object classification module connected to the image capturing device, the object classification module being operable to determine whether or not an object in an image is a member of an object class, the object classification module including N decision steps configured in a cascade configuration, wherein at least one of the N decision steps is operable to (a) accept an object as a member of the object class, (b) reject an object as a member of the object class, and (c) call on a next step to determine whether or not an object is a member of the object class.

2. A camera system according to claim 1, further comprising,

an object detection module connected to the image capturing device and the object classification module, the object detection module receiving the image data and being operable to detect whether or not an object is in the field of view of the image capturing device, the object detection module passing a detected object to the object classification module.

3. A camera system according to claim 1, wherein said at least one of the N decision steps includes a stage for mapping an object feature to a scalar value.

4. A camera system according to claim 3, wherein the stage includes a discriminant function for determining the scalar value.

5. A camera system according to claim 4, wherein the scalar value is correlated with a classification confidence.

6. A camera system according to claim 3, wherein the object feature is an aspect ratio of an object.

7. A camera system according to claim 3, wherein the object feature is an edge orientation histogram of an object.

8. A camera system according to claim 7, wherein the edge orientation histogram is produced by using a steerable filter.

9. A camera system according to claim 1, wherein multiple features of the object are identified, wherein said at least one of the N decision steps includes a stage for mapping multiple object features to a scalar value.

10. A camera system according to claim 1, wherein the object classification module is operable to classify an object as a human or non-human.

11. A camera system according to claim 1, wherein the object classification module is operable to classify an object as a vehicle or non-vehicle.

12. A method of classifying an object captured by a camera system, the camera system including an object classification module having N decision steps configured in a cascade configuration, the method comprising:

capturing an image of an object;

transmitting image data representing the object to a first one of the N decision steps;

identifying a feature of the object represented in the image data to determine whether the object is a member of an object class, wherein a decision step value is derived from the feature of the object; and

making a decision to either accept the object as a member of the object class, reject the object as a member of the object class, or forward the image data to a second one of the N decision steps for further analysis, the decision being based on a comparison of the decision step value to one or more of an acceptance threshold and a rejection threshold, the acceptance threshold being a higher value than the rejection threshold, wherein the object is accepted as a member of the object class when the decision step value is above the acceptance threshold, the object is rejected as a member of the object class when the decision step value is below the rejection threshold, and the image data is forwarded to the second decision step when the decision step value is between the acceptance and rejection thresholds.

13. A method according to claim 12, further comprising:

detecting the object.

14. A method according to claim 13, wherein the object is rejected as a member of the object class when the decision step value is at or below the rejection threshold.

15. A method according to claim 13, wherein the image data is accepted as a member of the object class when the decision step value is at or above the acceptance threshold.

16. A method according to claim 13, wherein multiple features of the object are identified and the decision step value is derived from the multiple features.

17. A camera system comprising:

an image capturing device having a field of view and producing image data representing an image of the field of view; and

an object classification module connected to the image capturing device, the object classification module being operable to determine whether or not an object in the image is a member of an object class, the object classification module including N decision steps configured in a cascade configuration, each of the decision steps including one or more stages for mapping object features to scalar values, wherein a first one of the stages includes a first discriminant function for determining a first scalar value, a second one of the stages includes a second discriminant function for determining a second scalar value, and the first and second discriminant functions are of different types.

18. A camera system according to claim 17, further comprising:

an object detection module connected to the image capturing device and the object classification module, the object detection module receiving the image data and being operable to detect whether or not an object is in the field of view of the image capturing device.

19. A camera system according to claim 17, wherein the first and second stages correspond to the same decision step.

20. A camera system according to claim 19, wherein the object classification module is operable to apply first and second weighting coefficients to the first and second scalar values, respectively, and the object classification module is operable to add the weighted first and second scalar values to produce a decision step value.

21. A camera system according to claim 17, wherein the first and second stages correspond to different ones of the N decision steps.

22. A camera system according to claim 17, wherein the first stage is operable to map a first object feature to the first scalar value, and the second stage is operable map a second object feature to the second scalar value.

23. A camera system according to claim 22, wherein the first object feature corresponds to an aspect ratio of an object, and the second object feature corresponds to an edge orientation histogram of the object.

24. A camera system according to claim 23, wherein the edge orientation histogram is produced by using a steerable filter.

25. A camera system according to claim 17, wherein the first discriminant function includes a radial basis function, and the second discriminant function includes a sigmoid function.

26. A method of classifying an object captured by a camera system, the method comprising:

producing image data representing an image of an object captured by the camera system;

identifying first and second features of the object represented in the image data;

mapping the first and second features of the object to respective first and second scalar values, wherein a first discriminant function is used to produce the first scalar value, a second discriminant function is used to produce the second scalar value, and the first and second discriminant functions are selected during a training operation from a group consisting of plural different discriminant functions; and

determining whether the object is a member of an object class based on a decision step value derived from the first and second scalar values.

27. A camera system comprising:

an image capturing device having a field of view and an image plane, the image capturing device producing image data representing an image of the field of view projected on the image plane;

an object classification module connected to the image capturing device and operable to detect and classify objects captured in the field of view based on the image data, wherein the object classification module is operable to classify objects as members or non-members of an object class; and

a calibration module connected to the object classification module for estimating representative sizes of members of the object class, the representative sizes corresponding to different regions of the image plane, wherein the calibration module is operable to automatically update the representative sizes in response to classifications performed by the object classification module during live operation, and the calibration module is operable to supply information representing the updated representative sizes to the object classification module to improve its object classification performance.

28. A camera system according to claim 27, wherein when the object classification module classifies an object as a member of an object class, a confidence parameter is associated with that classification, and wherein the calibration module updates the representative sizes only if the confidence parameter indicates a high confidence that the classification is correct.

29. A camera system according to claim 27, further comprising:

a user station having a display and an input device for providing user feedback information in response to classifications made by the object classification module, wherein the calibration module is operable to use the user feedback information to update the representative sizes.

30. A camera system according to claim 27, wherein the video analytics module is operable to use calibration information from the calibration module to improve object detection and classification accuracy.

31. A camera system according to claim 27, further comprising:

a speed estimation module connected to the calibration module and operable to estimate speeds of classified objects located in different regions of the field of view.

32. A camera system according to claim 27, in which the object classification module is operable to detect and classify objects without requiring initial manual calibration.

33. A camera system according to claim 27, wherein the object classification module constructs an edge orientation histogram of an object.

34. A camera system according to claim 33, wherein the object classification module comprise a steerable filter that produces the edge orientation histogram.

35. A method of automatically calibrating a field-deployed camera system, the method comprising:

capturing multiple images of a field of view of the camera system, the multiple images corresponding to an image plane of the camera system on which the field of view is projected;

detecting a first object in the multiple images, the first object being detected at different locations of the image plane in the multiple images, and images of the first object having different sizes corresponding to the different locations;

classifying the first object as a first member of an object class;

calculating a parameter of a size function for the image plane based on the different sizes of the first object, wherein the size function is used to estimate representative sizes of the object class for the image plane; and

updating the parameter of the size function in response to detection and classification of a second member of the object class, the second member being detected and classified during live operation of the camera system.

36. A method according to claim 35, wherein the first member of the object class and the second member of the object class both correspond to the first object.

37. A method according to claim 35, wherein the second member of the object class corresponds to a second object different from the first object.

38. A method according to claim 35, wherein classification of the second member of the object class generates a confidence parameter that represents a measure of confidence that the second object is a member of the object class, and wherein the updating step is performed only if the confidence parameter indicates a high confidence that the classification is correct.

39. A method of modifying an object classification module used by a user in a field-deployed camera system, the method comprising:

capturing multiple images of a field of view of the camera system, the multiple images including representations of multiple objects, wherein a first set of the multiple objects are members of an object class and a second set of the multiple objects are not members of the object class;

classifying the multiple objects as either members or non-members of the object class, wherein the object classification module produces one or more mistaken classifications;

generating mistake metadata based on acknowledgement by the user of at least some of the one or more mistaken classifications; and

modifying the object classification module based on the mistake metadata so as to reduce a number of mistaken classifications, the modifying being performed automatically during use in the field of the camera system.

40. A method according to claim 39, wherein the object classification module initially includes N decision steps configured in a cascade configuration, and wherein the modifying step comprises constructing an additional step from the mistake metadata to add to the N decision steps of the object classification module, thereby resulting in an object classification module with N+1 steps.

41. A method according to claim 39 wherein the mistaken classifications are false alarms representing instances when the object classification module classifies objects of the second set as members of the object class.

42. A method according to claim 39 further comprising:

displaying on a display representations of the mistaken classifications for acknowledgement by the user.

43. A method according to claim 39, wherein the object classification module before performance of the modifying step represents a deployed classifier, and the object classification module after performance of the modifying step represents a specialized classifier, the method further comprising validating the specialized classifier.

44. A method according to claim 43, wherein the validating step comprises passive validation of the specialized classifier, the passive validation including:

classifying objects with the specialized classifier;

classifying the objects with the deployed classifier, the deployed classifier generating false alarms and true alarms;

presenting the false alarms and true alarms to a user for acknowledgment;

generating feedback information based on user acknowledgment of the false alarms and true alarms; and

comparing the classifications of the specialized classifier with the feedback information.

45. A method according to claim 43, wherein the validating step comprises active validation of the specialized classifier, the active validation including:

classifying the objects with the deployed classifier;

classifying objects with the specialized classifier, wherein some of the classifications of the specialized classifier are inconsistent with the classifications of the deployed classifier; and

presenting representations of the inconsistent classifications of the specialized classifier to a user for validation.

46. A method according to claim 39, wherein the camera system includes multiple image capturing devices having different fields of view, the field of view of the camera system corresponding to the different fields of view of the image capturing devices, and the modifying step represents a site-specialization step.

47. A method according to claim 46 wherein the modifying step represents a camera specialization step for one of the multiple image capturing devices.

48. A method according to claim 39, wherein the camera system is connected to a central feedback server over a network, the method further comprising:

communicating the mistake metadata to the central feedback server, where the mistake metadata is used for training other object classifiers.

49. A method according to claim 48, wherein the camera system is deployed at a site, the method further comprising:

communicating performance statistics of the object classification module to the central feedback server; and

communicating to the central feedback server information representing the site at which the camera system is deployed.

50. A camera system comprising:

an image capturing device having a field of view and producing image data representing images of the field of view;

an object classification module connected to the image capturing device, the object classification module being operable to determine whether or not objects in the images are members of an object class, wherein the object classification module generates mistaken classifications;

a user station connected to the image capturing device and having a display for presenting images of the field of view to a user, the user station being operable to present on the display representations of the mistaken classifications generated by the object classification module, wherein the user station is operable to produce user feedback information in response to user acknowledgement of the mistaken classifications, the user feedback generating mistake metadata; and

a classifier evolution module receiving the mistake metadata and being operable to modify the object classification module using the mistake metadata so as to reduce the number of mistaken classifications, the classifier evolution module thereby generating a specialized classifier.

51. A camera system according to claim 50, further comprising:

an object detection module connected to the image capturing device and to the object classification module, the object detection module receiving the image data, the object detection module being operable to detect whether or not objects are in the field of view of the image capturing device.

52. A camera system according to claim 50, wherein the object classification module includes N decision steps configured in a cascade configuration, and wherein the specialized classifier includes the N decision steps and an additional step constructed based on the mistake metadata, thereby resulting in a specialized classifier having N+1 steps.

53. A camera system according to claim 50, wherein the mistaken classifications comprise false alarms representing instances when the object classification module incorrectly classifies objects as members of the object class.

54. A camera system according to claim 50, wherein the classifier evolution module is located remotely from the image capturing device, the object classification module, and the user station.

55. A camera system according to claim 50, wherein the image capturing device, the object classification module, the user station, and the classifier evolution module are colocated at a same site.

56. A method of constructing a new object classification module for use in a field-deployed camera system, the new object classification module for classifying objects as members or non-members of a new object class selected by a user, the method comprising:

capturing multiple images of a field of view of the camera system, the multiple images including representations of multiple objects, wherein a first set of the multiple objects are members of the new object class and a second set of the multiple objects are not members of the new object class;

using a deployed object classification module to classify the multiple objects as members of a deployed object class;

presenting on a display representations of the multiple objects classified by the deployed object classification module, wherein a user labels the multiple objects as either members or non-members of the new object class;

generating metadata based on the labels; and

constructing the new object classification by modifying the deployed object classification module based on the metadata.

57. A method according to claim 56, further comprising:

validating the new object classification module.

58. A method according to claim 56, wherein the new object class is a subclass of the deployed object class.

59. A method according to claim 56, wherein the new object class represents objects whose classification is unknown.

60. A method according to claim 56, wherein the deployed object classification module includes N decision steps configured in a cascade configuration, and wherein the new object classification module includes the N decision steps of the deployed object classification module and an additional step that is trained by the metadata to classify members of the deployed object class as either members or non-member of the new object class, thereby resulting in the new object classification module have N+1 steps.