Video object classification with object size calibration

• US 10,127,445 B2
• Filed: 05/25/2017
• Issued: 11/13/2018
• Est. Priority Date: 03/03/2008
• Status: Active Grant

• Current Assignee: Motorola Solutions, Inc.
• Original Assignee: Avigilon Analytics Corporation
• Inventors: Saptharishi, Mahesh, Lisin, Dimitri A., Lipchin, Aleksey, Reyzin, Igor
• Primary Examiner(s): Tran, Nhan T
• Application Number:
  US15/605,818
• Publication Number:
  US 20170262702A1
• Time in Patent Office: 537 Days
• Field of Search: 348143, 348169, 348226
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 2218/12 Classification; Matching
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/631 Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
• CPC Class Codes: H04N 23/66 Remote control of cameras or camera parts, e.g. by remote control devices
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

A camera system comprises an image capturing device, and connected to it are an object classification module and a calibration module. The object classification module is operable to determine whether or not an object in an image is a member of an object class, and the calibration module is operable to estimate representative sizes of the object. The object classification module may determine a confidence parameter that is used by the calibration module, or conversely, the calibration module may produce a size that is used by the classification module.

Claims

1. A camera system comprising:

video analytics for processing image data representing an image of a field of view projected on an image plane of an image capturing device, the video analytics comprising;

an object classification module including an object classifier that is operable to classify objects captured in the field of view based on the image data, wherein the object classifier is operable to classify objects as members of one object class among a plurality of object classes or a member of an unknown object class;

a calibration module connected to the object classification module for estimating representative sizes of objects, the representative sizes corresponding to different regions of the image plane, wherein the calibration module is operable to automatically update the representative sizes of objects classified as members of the object class in response to classifications performed by the object classifier during live operation, and wherein the calibration module is operable to supply information representing the updated representative sizes to the object classifier to improve its object classification performance; and

a temporal object classification module connected to the calibration module that is operable to change the classification of an object classified to the unknown object class to the object class based on a threshold.

2. The camera system according to claim 1, wherein when the object classifier classifies an object as a member of an object class, a confidence parameter is associated with that classification, and wherein the calibration module updates the representative sizes when the confidence parameter indicates a high confidence that the classification is correct.

3. The camera system according to claim 1, further comprising:

a user station having a display and an input device for providing user feedback information in response to classifications made by the object classifier, wherein the user feedback information is used to improve estimation of the representative sizes.

4. The camera system according to claim 1, wherein the calibration information from the calibration module is used to improve object detection and classification accuracy of the camera system.

5. The camera system according to claim 1, further comprising:

a speed estimation module connected to the calibration module and operable to estimate speeds of classified objects located in different regions of the field of view.

6. The camera system according to claim 1, in which the object classifier is operable to classify objects without requiring initial manual calibration.

7. The camera system according to claim 1, wherein the object classifier includes a stage corresponding to a feature/transformation combination among a group of multiple feature/transformation combinations, the feature/transformation combination corresponding to a feature extracted from the image data and a discriminant function for mapping the feature to a scalar value, the feature being selected from a set of possible features and the discriminant function being selected from a set of possible discriminant functions, the feature and the discriminant function being selected so as to maximize object classification performance of the object classifier by selecting the feature/transformation combination based on a comparison how the feature and the discriminant function perform on other feature/transformation combinations among the group of multiple feature/transformation combinations.

8. The camera system according to claim 7, wherein the set of possible features includes one or more of an aspect ratio, an edge orientation histogram, color information, and normalized saturation.

9. The camera system according to claim 8, wherein the edge orientation histogram is produced by using a steerable filter.

10. The camera system according to claim 7, wherein the set of possible discriminant functions includes one or both of a radial basis function and a sigmoid function.

11. The camera system according to claim 7, wherein the object classifier is operable to classify an object as a human or non-human.

12. The camera system according to claim 7, wherein the object classifier is operable to classify an object as a vehicle or non-vehicle.

13. A method of automatically calibrating a camera system, the method comprising:

receiving image data representing an image of a scene, the image corresponding to an image plane on which the scene is projected;

detecting a first object in the image, an image of the first object being detected at a location of the image plane, the image of the first object having a size corresponding to the location;

classifying the first object as a first member of an object class among a plurality of object classes or as a first member of an unknown object class;

when the first object is classified as a first member of the object class, calculating a parameter of a size function for the image plane based on the size of the first object;

when the first object is classified as a first member of the unknown object class, changing the classification of the first object from the unknown object class to the object class based on a temporal threshold; and

updating the parameter of the size function in response to detection and classification of a second member of the object class.

14. The method according to claim 13, wherein the first member of the object class and the second member of the object class both correspond to the first object.

15. The method according to claim 13, wherein the second member of the object class corresponds to a second object different from the first object.

16. The method according to claim 13, wherein classification of the second member of the object class generates a confidence parameter that represents a measure of confidence that the second object is a member of the object class, and wherein the updating step is performed when the confidence parameter indicates a high confidence that the classification is correct.

17. The method according to claim 13, wherein the representative sizes are determined by a parameterized function of position.

18. The method according to claim 17, wherein the parameterized function of position is a second-order polynomial in X and Y coordinates.

19. The method according to claim 13, wherein the second member is detected and classified during live operation of the camera system.

20. The method according to claim 13, wherein:

the image data represents multiple images of the scene, the first object is detected in the multiple images, multiple images of the first object are detected in different locations of the image plane, the multiple images of the first object having different sizes corresponding to the different locations;

the first object is classified in the multiple images as the first member of the object class; and

the parameter of the size function is calculated based on the different sizes of the first object, the size function being used to estimate representative sizes of the object class for the image plane.

21. A camera system comprising:

video analytics for processing image data representing an image of a field of view projected on an image plane of an image capturing device, the video analytics comprising;

an object classification module including an object classifier that is operable to classify objects captured in the field of view based on the image data, wherein the object classifier is operable to classify objects as members of one object class among a plurality of object classes or a member of an unknown object class;

a calibration module connected to the object classification module for estimating representative sizes of objects, the representative sizes corresponding to different regions of the image plane, wherein the calibration module is operable to automatically update the representative sizes of objects classified as members of the object class in response to classifications performed by the object classifier during in-field operation, a confidence parameter is associated with classifications of objects by the classification module, and the confidence parameter is determined at least in part based on the representative size of the associated object; and

a temporal object classification module connected to the calibration module that is operable to change the classification of an object classified to the unknown object class to the object class based on a threshold.