Object detection and classification

• US 9,760,792 B2
• Filed: 03/18/2016
• Issued: 09/12/2017
• Est. Priority Date: 03/20/2015
• Status: Active Grant

• Current Assignee: Netra, Inc.
• Original Assignee: Netra, Inc.
• Inventors: Kant, Shashi
• Primary Examiner(s): Alavi, Amir
• Application Number:
  US15/074,104
• Publication Number:
  US 20160275375A1
• Time in Patent Office: 543 Days
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 10/449 Biologically inspired filters, e.g. difference of Gaussians [DoG] or Gabor filters
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 10/762 using clustering, e.g. of similar faces in social networks
• CPC Class Codes: G06V 20/38 Outdoor scenes
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/586 of parking space
• CPC Class Codes: G06V 30/194 References adjustable by an adaptive method, e.g. learning

Abstract

Object detection and classification across disparate fields of view are provided. A first image generated by a first recording device with a first field of view, and a second image generated by a second recording device with a second field of view can be obtained. An object detection component can detect a first object within the first field of view, and a second object within the second field of view. An object classification component can determine first and second level classification categories of the first object. A data processing system can create a data structure indicating a probability identifier for a descriptor of the first object. An object matching component can correlate the first object with the second object based on the descriptor of the first object, the probability identifier for the descriptor of the first object, or a descriptor of the second object.

Claims

1. A system of object detection across disparate fields of view, comprising:

a data processing hardware system having an object detection component, an object classification component, and an object matching component, the data processing system obtains a first image generated by a first recording device, the first recording device having a first field of view;

the object detection component of the data processing hardware system detects, from the first image, a first object present within the first field of view;

the object classification component of the data processing hardware system determines a first level classification category of the first object and determines, from the first level classification category of the first object, a second level classification category of the first object;

the data processing hardware system generates a descriptor of the first object based on at least one of the first level classification category of the first object and the second level classification category of the first object;

the data processing hardware system creates, for the first object, a data structure indicating a probability identifier for the descriptor of the first object;

the data processing system obtains a second image generated by a second recording device, the second recording device having a second field of view different than the first field of view;

the object detection component of the data processing hardware system detects, from the second image, a second object present within the second field of view;

the data processing hardware system generates a descriptor of the second object based on at least one of a first level classification category of the second object and a second level classification category of the second object; and

the object matching component of the data processing hardware system correlates the first object with the second object based on the descriptor of the first object, the probability identifier for the descriptor of the first object, and the descriptor of the second object.

2. The system of claim 1, wherein the first image includes a background object, comprising:

the data processing hardware system configured to detect the first object based on an obstruction, by the first object, of the background object within the first image.

3. The system of claim 1, comprising:

the object classification component of the data processing hardware system operational to determine the first level classification category of the second object and to determine, from the first level classification category of the second object, the second level classification category of the second object.

4. The system of claim 3, comprising:

the object matching component of the data processing hardware system configured to correlate the first object with the second object based on the second level classification category of the first object and the second level classification category of the second object.

5. The system of claim 3, comprising:

the object matching component of the data processing hardware system configured to correlate the first object with the second object based the first level classification category of the first object and the first level classification category of the second object.

6. The system of claim 1, comprising:

the data processing hardware system operational to generate a track of the first object within the first field of view, and to generate a track of the second object within the second field of view.

7. The system of claim 1, comprising the data processing hardware system operational to:

receive, from an end user computing device via a computer network, a query that identifies at least one of the first object and the second object;

generate, responsive to the query, a track of at least one first object and the second object; and

provide, via the network, to the end user computing device, the track for display by the end user computing device.

8. The system of claim 1, comprising:

the object matching component of the data processing hardware system operational to determine a probability that the first object and the second object are a same object based on the descriptor of the first object and the descriptor of the second object.

9. The system of claim 1, comprising the data processing hardware system configured to:

receive, from an end user computing device via a computer network, a request to generate an electronic document associated with at least one of the first image and the second image;

responsive to the request, generate the electronic document, the electronic document indicating at least one of a track of the first object, a track of the second object, a utilization rate associated with the first field of view, a utilization rate associate with the second field of view, a traffic indicator associated with the first field of view, and a traffic indicator associated with the second field of view; and

provide the electronic document to the end user computing device via the computer network.

10. The system of claim 1, comprising the data processing hardware system configured to:

determine that the first object present within the first field of view of the first recording device is more closely associated with the second object present within the second field of view of the second recording device than with a third image in a third field of view of a third recording device.

11. The system of claim 1, comprising:

the object detection component of the data processing hardware system configured to detect, from the first image, the first object present within the first field of view based on a location of the first object in a first frame of the first image and a location of the first object in a second frame of the first image.

12. The system of claim 1, comprising:

the object detection component of the data processing hardware system configured to detect, from the first image, the first object present within the first field of view based on edge detection of an edge of the first object.

13. The system of claim 1, wherein the second level classification category of the first object includes a sub-category of the first level classification category of the first object.

14. The system of claim 1, comprising:

the data processing hardware system operational to provide the first image and the second image to an end user computing device via a network, for simultaneous display by the end user computing device.

15. The system of claim 1, wherein the first object and the second object are a same object.

16. The system of claim 1, comprising:

the data processing hardware system configured to store the data structure indicating the probability identifier for the descriptor of the first object in an index data structure.

17. The system of claim 1, wherein the first object is present within the first field of view at a first time, and the second object is present within the second field of view at a second time subsequent to the first time.

18. A method of digital image object detection across disparate fields of view, comprising:

obtaining, by a data processing hardware system having at least one of an object detection component, an object classification component, and an object matching component, a first image generated by a first recording device, the first recording device having a first field of view;

detecting, by the object detection component of the data processing hardware system, from the first image, a first object present within the first field of view;

determining, by the object classification component of the data processing hardware system, a first level classification category of the first object and determining, from the first level classification category of the first object, a second level classification category of the first object;

generating, by the data processing hardware system, a descriptor of the first object based on at least one of the first level classification category of the first object and the second level classification category of the first object;

creating, by the data processing hardware system, for the first object, a data structure indicating a probability identifier for the descriptor of the first object;

obtaining, by the data processing system, a second image generated by a second recording device, the second recording device having a second field of view different than the first field of view;

detecting, by the object detection component of the data processing hardware system, from the second image, a second object present within the second field of view;

generating, by the data processing hardware system, a descriptor of the second object based on at least one of a first level classification category of the second object and a second level classification category of the second object; and

correlating, by the object matching component of the data processing hardware system, the first object with the second object based on the descriptor of the first object, the probability identifier for the descriptor of the first object, and the descriptor of the second object.

19. The method of claim 18, comprising:

correlating, by the object matching component of the data processing hardware system, the first object with the second object based on a probability identifier for the descriptor of the second object.

20. The method of claim 18, comprising:

providing, by the data processing hardware system via a computer network, for display by an end user computing device, a first electronic document that includes at least one of the first image and the second image, the first electronic document including an actuation mechanism;

receiving, by the data processing hardware system via the computer network, from the end user computing device, an indication of actuation of the actuation mechanism; and

generating, by the data processing hardware system, responsive to the indication of actuation of the actuation mechanism, a second electronic document that includes at least one of a track of the first object within the first image, a track of the second object within the second image, a utilization rate associated with the first field of view, a utilization rate associate with the second field of view, a traffic indicator associated with the first field of view, and a traffic indicator associated with the second field of view.