REAL-TIME OBJECT ANALYSIS WITH OCCLUSION HANDLING

• US 20170061239A1
• Filed: 11/10/2016
• Published: 03/02/2017
• Est. Priority Date: 05/22/2015
• Status: Active Grant

• Current Assignee: International Business Machines Corporation, The University of Queensland
• Original Assignee: International Business Machines Corporation, The University of Queensland
• Inventors: Brown, Lisa M., Emami, Sayed Ali, Harandi, Mehrtash, Pankanti, Sharathchandra U.
• Granted Patent: US 10,002,309 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 20/53 Recognition of crowd images, e.g. recognition of crowd congestion
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands

Abstract

A method includes the following steps. A video sequence including detection results from one or more detectors is received, the detection results identifying one or more objects. A clustering framework is applied to the detection results to identify one or more clusters associated with the one or more objects. The clustering framework is applied to the video sequence on a frame-by-frame basis. Spatial and temporal information for each of the one or more clusters are determined. The one or more clusters are associated to the detection results based on the spatial and temporal information in consecutive frames of the video sequence to generate tracking information. One or more target tracks are generated based on the tracking information for the one or more clusters. The one or more target tracks are consolidated to generate refined tracks for the one or more objects.

Claims

1. A method, comprising the steps of:

receiving a video sequence comprising detection results from one or more detectors, the detection results identifying one or more objects;

applying a clustering framework to the detection results to identify one or more clusters associated with the one or more objects, wherein the clustering framework is applied to the video sequence on a frame-by-frame basis;

determining spatial and temporal information for each of the one or more clusters;

associating the one or more clusters to the detection results based on the spatial and temporal information in consecutive frames of the video sequence to generate tracking information;

generating one or more target tracks based on the tracking information for the one or more clusters; and

consolidating the one or more target tracks to generate refined tracks for the one or more obj ects;

wherein the steps are performed by at least one processor device coupled to a memory.

2. The method of claim 1, wherein applying the cluster framework further comprises:

detecting one or more non-associated detections, wherein the one or more non-associated detections have not been assigned to an existing cluster;

applying a Non-Max Suppression (NMS) method to the one or more non-associated detections;

evaluating the NMS results with a confidence score for the one or more non-associated detections; and

deciding on at least one of;

instantiating a new cluster based on the confidence score; and

eliminating one or more existing clusters.

3. The method of claim 2, wherein the confidence score comprises a depth-height map and a detection frequency.

4. The method of claim 3, wherein the depth-height map represents a probability of a correct detection according to a relevancy of detection size and position in the frame.

5. The method of claim 2, wherein the new cluster is instantiated when the confidence score exceeds a threshold.

6. The method of claim 2, wherein one or more clusters are eliminated when the one or more clusters are void of objects for a period of time.

7. The method of claim 1, wherein the detection results comprise one or more bounding boxes identifying one or more of the objects.

8. The method of claim 1, wherein at least one of the one or more objects is at least partially occluded by another object.

9. The method of claim 7, wherein the at least one partially occluded object is associated to at least one cluster based on the spatial and temporal information.

10. The method of claim 1, wherein consolidating the one or more target tracks comprises calculating a distinct confidence score for each of the one or more target tracks.

11. The method of claim 10, wherein consolidating the one or more target tracks comprises removing one or more target tracks based on the calculated distinct confidence score.

12. The method of claim 10, wherein consolidating the one or more target tracks comprises completing one or more incomplete target tracks based on the calculated distinct confidence score.

13. The method of claim 1, wherein consolidating the one or more target tracks comprises joining two or more target tracks based on a correspondence loss function.

14. An apparatus, comprising:

a memory; and

a processor operatively coupled to the memory and configured to;

receive a video sequence comprising detection results from one or more detectors, the detection results identifying one or more objects;

apply a clustering framework to the detection results to identify one or more clusters associated with the one or more objects, wherein the clustering framework is applied to the video sequence on a frame-by-frame basis;

determine spatial and temporal information for each of the one or more clusters;

associate the one or more clusters to the detection results based on the spatial and temporal information in consecutive frames of the video sequence to generate tracking information;

generate one or more target tracks based on the tracking information for the one or more clusters; and

consolidate the one or more target tracks to generate refined tracks for the one or more objects.

15. The apparatus of claim 14, wherein applying the cluster framework further comprises:

detecting one or more non-associated detections, wherein the one or more non-associated detections have not been assigned to an existing cluster;

applying a Non-Max Suppression (NMS) method to the one or more non-associated detections;

evaluating the NMS results with a confidence score for the one or more non-associated detections; and

deciding on at least one of;

instantiating a new cluster based on the confidence score; and

eliminating one or more existing clusters.

16. The apparatus of claim 15, wherein the new cluster is instantiated when the confidence score exceeds a threshold.

17. The apparatus of claim 14, wherein at least one of the one or more objects are at least partially occluded by another object.

18. The apparatus of claim 17, wherein the at least one partially occluded object is associated to at least one cluster based on the spatial and temporal information.

19. The apparatus of claim 15, wherein consolidating the one or more target tracks comprises calculating a distinct confidence score for each of the one or more target tracks.

20. An article of manufacture comprising a computer readable storage medium for storing computer readable program code which, when executed, causes a computer to:

receive a video sequence comprising detection results from one or more detectors, the detection results identifying one or more objects;

apply a clustering framework to the detection results to identify one or more clusters associated with the one or more objects, wherein the clustering framework is applied to the video sequence on a frame-by-frame basis;

determine spatial and temporal information for each of the one or more clusters;

associate the one or more clusters to the detection results based on the spatial and temporal information in consecutive frames of the video sequence to generate tracking information;

generate one or more target tracks based on the tracking information for the one or more clusters; and

consolidate the one or more target tracks to generate refined tracks for the one or more objects.