Systems and methods for object tracking and classification

• US 9,959,468 B2
• Filed: 11/06/2015
• Issued: 05/01/2018
• Est. Priority Date: 11/06/2015
• Status: Active Grant

• Current Assignee: The Boeing Co.
• Original Assignee: The Boeing Co.
• Inventors: Mosher, Aaron Y., Mefford, David Keith
• Primary Examiner(s): NAKHJAVAN, SHERVIN K
• Application Number:
  US14/934,773
• Publication Number:
  US 20170132468A1
• Time in Patent Office: 907 Days
• Field of Search: None
• CPC Class Codes: G06F 18/2411 based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes
• CPC Class Codes: G06V 20/70 Labelling scene content, e.g. deriving syntactic or semantic representations

Abstract

A method for classifying at least one object of interest in a video is provided. The method includes accessing, using at least one processing device, a frame of the video, the frame including at least one object of interest to be classified, performing, using the at least one processing device, object detection on the frame to detect the object of interest, tracking, using the at least one processing device, the object of interest over a plurality of frames in the video over time using a persistent tracking capability, isolating, using the at least one processing device, a segment of the frame that includes the object of interest, classifying, using the at least one processing device, the object of interest by processing the segment using deep learning, and generating an output that indicates the classification of the object of interest.

Claims

1. A method of classifying at least one object of interest in a video, the method comprising:

accessing, using at least one processing device, a frame of the video, the frame including at least one object of interest to be classified;

performing, using the at least one processing device, object detection on the frame to detect the object of interest;

tracking, using the at least one processing device, the object of interest over a plurality of frames in the video over time using a persistent tracking capability;

isolating, using the at least one processing device, a segment of the frame that includes the object of interest;

classifying, using the at least one processing device, the object of interest tracked in the tracking step, by processing the segment using deep learning;

generating an output that indicates the classification of the object of interest; and

continuing to track, using the classification of the object of interest, the object of interest after the object of interest has left and returned to a field of view of the video.

2. A method in accordance with claim 1, wherein tracking the object of interest comprising tracking the object of interest using a tracking component, and wherein classifying the object of interest comprises classifying the object of interest using a classification component, wherein the classification component is located remotely from the tracking component.

3. A method in accordance with claim 2, wherein classifying the object of interest using a classification component comprises classifying the object of interest using a classification component of at least one processing device installed on a vehicle.

4. A method in accordance with claim 2, wherein classifying the object of interest using a classification component comprises classifying the object of interest using a classification component that satisfies small size, weight, and power requirements.

5. A method in accordance with claim 1, wherein classifying the object of interest by processing the segment using deep learning comprises classifying the object of interest by processing the segment using at least one of a convolutional neural network and a deep neural network.

6. A method in accordance with claim 1, wherein isolating a segment of the frame that includes the object of interest comprises isolating the segment such that the object of interest is substantially centered within the segment.

7. A method in accordance with claim 1, wherein generating an output comprises:

receiving, from a user at an input device coupled to the at least one processor, a predetermined object type; and

generating, using a presentation interface coupled to the at least one processor, an alert when the classification of the object of interest matches the predetermined object type.

8. A method in accordance with claim 1, further comprising tracking, using the at least one processing device, an object of interest that has been classified in the classifying step, over a plurality of frames in the video over time using the persistent tracking capability.

9. A computer-implemented system operative to classify at least one object of interest in a video, the system comprising:

a tracking component implemented using at least one processing device and configured to;

access a frame of the video, the frame including at least one object of interest to be classified;

perform object detection on the frame to detect the object of interest;

track the object of interest over a plurality of frames in the video over time using a persistent tracking capability;

isolate a segment of the frame that includes the object of interest; and

continue to track, using a classification of the object of interest, the object of interest after the object of interest has left and returned to a field of view of the video; and

a classification component communicatively coupled to said tracking component, said classification component implemented using the at least one processing device and configured to;

classify the object of interest tracked by the tracking component, by processing the segment using deep learning; and

generate an output that indicates the classification of the object of interest.

10. A system in accordance with claim 9, wherein said classification component and said tracking component are implemented using a single processing device.

11. A system in accordance with claim 9, wherein said classification component is located remotely from said tracking component.

12. A system in accordance with claim 11, wherein said classification component is implemented using at least one processing device installed on a vehicle.

13. A system in accordance with claim 11, wherein said classification component satisfies small size, weight, and power requirements.

14. A system in accordance with claim 9, wherein to classify the object of interest by processing the segment using deep learning, said classification component is configured to classify the object of interest using at least one of a convolutional neural network and a deep neural network.

15. A system in accordance with claim 9, wherein to isolate a segment of the frame that includes the object of interest, said tracking component is configured to isolate the segment such that the object of interest is substantially centered within the segment.

16. A system in accordance with claim 9, wherein the tracking component is further configured to track an object of interest that has been classified by the classifying component, over a plurality of frames in the video over time using the persistent tracking capability.

17. An object classification computing device operative to classify at least one object of interest in a video, the object classification computing device comprising:

one or more memory devices; and

one or more processors communicatively coupled to said one or more memory devices, said one or more processors configured to;

access a frame of the video, the frame including at least one object of interest to be classified;

perform object detection on the frame to detect the object of interest;

track the object of interest over a plurality of frames in the video over time using a persistent tracking capability;

isolate a segment of the frame that includes the object of interest;

classify the object of interest tracked by the processor, by processing the segment using deep learning;

generate an output that indicates the classification of the object of interest; and

continue to track, using a classification of the object of interest, the object of interest after the object of interest has left and returned to a field of view of the video.

18. An object classification computing device in accordance with claim 17, wherein at least one of said processors is installed on a vehicle.

19. An object classification computing device in accordance with claim 17, wherein at least one of said processors satisfies small size, weight, and power requirements.

20. An object classification computing device in accordance with claim 17, wherein to classify the object of interest by processing the segment using deep learning, at least one of said processors is configured to classify the object of interest using a convolutional neural network.

21. An object classification computing device in accordance with claim 17, wherein at least one of said processors is further configured to track an object of interest that has been classified by the one or more processors, over a plurality of frames in the video over time using a persistent tracking capability.