Object detection and classification

• US 9,934,447 B2
• Filed: 09/07/2017
• Issued: 04/03/2018
• Est. Priority Date: 03/20/2015
• Status: Active Grant

• Current Assignee: Netra, Inc.
• Original Assignee: Netra, Inc.
• Inventors: Kant, Shashi
• Primary Examiner(s): Alavi, Amir
• Application Number:
  US15/698,220
• Publication Number:
  US 20170372164A1
• Time in Patent Office: 208 Days
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 10/449 Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 10/762 using clustering, e.g. of similar faces in social networks
• CPC Class Codes: G06V 20/38 Outdoor scenes
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/586 of parking space
• CPC Class Codes: G06V 30/194 References adjustable by an adaptive method, e.g. learning

Abstract

Object detection and classification across disparate fields of view are provided. A first image generated by a first recording device with a first field of view, and a second image generated by a second recording device with a second field of view can be obtained. An object detection component can detect a first object within the first field of view, and a second object within the second field of view. An object classification component can determine first and second level classification categories of the first object. A data processing system can create a data structure indicating a probability identifier for a descriptor of the first object. An object matching component can correlate the first object with the second object based on the descriptor of the first object, the probability identifier for the descriptor of the first object, or a descriptor of the second object.

Claims

1. A system of object detection across disparate fields of view, comprising:

a data processing hardware system having an object detection component, an object classification component, and an object matching component, the data processing hardware system obtains a first set of images generated by a recording device, the recording device having a first field of view at a first time;

the object detection component of the data processing hardware system detects, from the first set of images, movement of a first object within the first field of view;

the object classification component of the data processing hardware system determines a first level classification category of the first object and determines, from the first level classification category of the first object, a second level classification category of the first object;

the data processing hardware system generates a descriptor of the first object based on at least one of the first level classification category of the first object and the second level classification category of the first object;

the data processing hardware system creates, for the first object, a data structure indicating a probability identifier for the descriptor of the first object;

the data processing hardware system obtains a second set of images generated by the recording device of the first field of view at a second time after the first time;

the object detection component of the data processing hardware system detects, from the second set of images, movement of a second object within the first field of view;

the data processing hardware system generates a descriptor of the second object based on at least one of a first level classification category of the second object and a second level classification category of the second object; and

the object matching component of the data processing hardware system determines the second object and the first object are a same object in the first field of view based on the descriptor of the first object, the probability identifier for the descriptor of the first object, and the descriptor of the second object.

2. The system of claim 1, comprising:

the object classification component of the data processing hardware system operational to determine the first level classification category of the second object and to determine, from the first level classification category of the second object, the second level classification category of the second object.

3. The system of claim 2, comprising:

the object matching component of the data processing hardware system configured to correlate the first object with the second object based on the second level classification category of the first object and the second level classification category of the second object.

4. The system of claim 2, comprising:

the object matching component of the data processing hardware system configured to correlate the first object with the second object based the first level classification category of the first object and the first level classification category of the second object.

5. The system of claim 1, wherein the recording device is a first recording device, comprising:

the data processing hardware system obtains a third set of images generated by a second recording device with a second field of view;

the object detection component of the data processing hardware system detects, from the third set of images, a third object moving within the second field of view;

the data processing hardware system generates a descriptor of the third object based on at least one of a first level classification category of the third object and a second level classification category of the third object; and

the object matching component of the data processing hardware system determines the third object is the first object based on the descriptor of the first object, the probability identifier for the descriptor of the first object, and the descriptor of the third object.

6. The system of claim 1, comprising:

the data processing hardware system obtains a plurality of sets of images generated by the recording device;

the object detection component detects, in each of the plurality of sets of images, an object moving within the first field of view;

the data processing hardware system generates a descriptor of the object moving within the first field of view in each of the plurality of sets of images;

the object matching component of the data processing hardware system determines a count of times the first object entered the first field of view based on a number of times the descriptor of the object moving within the first field of view in each of the plurality of sets of images matches the descriptor of the first object.

7. The system of claim 1, comprising:

the object matching component of the data processing hardware system operational to determine a probability that the first object and the second object are the same object based on the descriptor of the first object and the descriptor of the second object.

8. The system of claim 1, comprising the data processing hardware system configured to:

receive, from an end user computing device via a computer network, a request to generate an electronic document associated with at least one of the first set of images and the second set of images;

responsive to the request, generate the electronic document, the electronic document indicating at least one of a track of the first object, a track of the second object, a utilization rate associated with the first field of view, and a traffic indicator associated with the first field of view; and

provide the electronic document to the end user computing device via the computer network.

9. The system of claim 1, comprising the data processing hardware system configured to:

determine that the first object is more closely associated with the second object than with a third object in a third set of images.

10. The system of claim 1, comprising:

the object detection component of the data processing hardware system configured to detect, from the first set of images, the first object present within the first field of view based on a location of the first object in a first frame of the first set of images and a location of the first object in a second frame of the first set of images.

11. The system of claim 1, comprising:

the object detection component of the data processing hardware system configured to detect, from the first set of images, the first object present within the first field of view based on edge detection of an edge of the first object.

12. The system of claim 1, wherein the second level classification category of the first object is a characteristic of the first level classification category of the first object, wherein the characteristic is at least one of a physical appearance, a weight, a height, or a type of clothing.

13. The system of claim 1, comprising:

the data processing hardware system operational to provide the first set of images and the second set of images to an end user computing device via a network, for simultaneous display by the end user computing device.

14. The system of claim 1, wherein the recording device is one of a mobile device, a smartphone, a security camera, or a dash camera.

15. The system of claim 1, wherein the first object is one of a person, a shopping cart, an animal, or a vehicle.

16. The system of claim 1, comprising:

the data processing hardware system configured to store the data structure indicating the probability identifier for the descriptor of the first object in an index data structure.

17. The system of claim 1, comprising:

the data processing hardware system determines that a difference between the second time and the first time is less than a predetermined threshold.

18. A method of digital image object detection across disparate fields of view, comprising:

obtaining, by a data processing hardware system having at least one of an object detection component, an object classification component, and an object matching component, a first set of images generated by a recording device, the recording device having a first field of view at a first time;

detecting, by the object detection component of the data processing hardware system, from the first set of images, a first object moving within the first field of view;

determining, by the object classification component of the data processing hardware system, a first level classification category of the first object and determining, from the first level classification category of the first object, a second level classification category of the first object;

generating, by the data processing hardware system, a descriptor of the first object based on at least one of the first level classification category of the first object and the second level classification category of the first object;

creating, by the data processing hardware system, for the first object, a data structure indicating a probability identifier for the descriptor of the first object;

obtaining, by the data processing hardware system, a second set of images generated by the recording device of the first field of view at a second time after the first time;

detecting, by the object detection component of the data processing hardware system, from the second set of images, a second object moving within the first field of view;

generating, by the data processing hardware system, a descriptor of the second object based on at least one of a first level classification category of the second object and a second level classification category of the second object; and

determining, by the object matching component of the data processing hardware system, the first object and the second object are a same object based on the descriptor of the first object, the probability identifier for the descriptor of the first object, and the descriptor of the second object.

19. The method of claim 18, comprising:

correlating, by the object matching component of the data processing hardware system, the first object with the second object based on a probability identifier for the descriptor of the second object.

20. The method of claim 18, comprising:

providing, by the data processing hardware system via a computer network, for display by an end user computing device, a first electronic document that includes at least one of the first set of images and the second set of images, the first electronic document including an actuation mechanism;

receiving, by the data processing hardware system via the computer network, from the end user computing device, an indication of actuation of the actuation mechanism; and

generating, by the data processing hardware system, responsive to the indication of actuation of the actuation mechanism, a second electronic document that includes at least one of a track of the first object within the first set of images, a track of the second object within the second set of images, a utilization rate associated with the first field of view, and a traffic indicator associated with the first field of view.