Robust object recognition from moving platforms by combining form and motion detection with bio-inspired classification

• US 9,576,214 B1
• Filed: 10/01/2014
• Issued: 02/21/2017
• Est. Priority Date: 01/23/2012
• Status: Active Grant

• Current Assignee: HRL Laboratories LLC (The Boeing Co.)
• Original Assignee: HRL Laboratories LLC (The Boeing Co.)
• Inventors: Zhang, Lei, Khosla, Deepak, Chen, Yang
• Primary Examiner(s): Mistry, Oneal R
• Application Number:
  US14/504,248
• Time in Patent Office: 874 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/2148 characterised by the process organisation or structure, e.g. boosting cascade
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06V 10/446 using Haar-like filters, e.g. using integral image techniques
• CPC Class Codes: G06V 10/7747 Organisation of the process, e.g. bagging or boosting
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames

Abstract

Described is system for object recognition from moving platforms. The system receives a video captured from a moving platform as input. The video is processed with a static object detection module to detect static objects in the video, resulting in a set of static object detections. The video is also processed with a moving object detection module to detect moving objects in the video, resulting in a set of moving object detections. The set of static object detections and the set of moving object detections are fused, resulting in a set of detected objects. The set of detected objects are classified with an object classification module, resulting in a set of recognized objects that are then output.

Claims

1. A system for object recognition from moving platforms, the system comprising:

one or more processors and a non-transitory computer-readable medium having executable instructions encoded thereon such that when executed, the one or more processors perform operations of;

receiving a video captured from a moving platform as input, wherein the video comprises a plurality of image frames;

processing each image frame in the plurality of image frames with a static object detection module to detect at least one of a first type of object, having static object characteristics, in the image frame, resulting in at least one static object detection, wherein the at least one first type of object is stationary relative to a around surface in the image frame;

following processing of the image frame with the static object detection module, processing the image frame with a moving object detection module to detect at least one of a second type of object, having moving object characteristics, in the image frame, resulting in at least one moving object detection;

fusing the at least one static object detection and the at least one moving object detection, resulting in a set of detected objects, wherein an aggregation process removes redundant detections that correspond to the same object;

classifying the set of detected objects with an object classification module, resulting in a set of recognized objects; and

outputting the set of recognized objects.

2. The system as set forth in claim 1, wherein the one or more processors further perform operations of:

extracting Haar features from the plurality of image frames; and

processing the extracted Haar features sequentially with a cascade of classifiers to detect static objects in the plurality of image frames in the static object detection module.

3. The system as set forth in claim 2, wherein each classifier in the cascade of classifiers is pre-trained using an AdaBoost learning technique.

4. The system as set forth in claim 1, wherein the one or more processors further perform operations of:

rotating each image frame in the plurality of image frames gradually by angles between 0 degrees and 360 degrees; and

processing each rotated image frame with the static object detection module to detect a set of static object detections in the rotated image frames.

5. A system for object recognition from moving platforms, the system comprising:

one or more processors and a non-transitory computer-readable medium having executable instructions encoded thereon such that when executed, the one or more processors perform operations of;

receiving a video captured from a moving platform as input, wherein the video comprises a plurality of image frames;

processing the video with a static object detection module to detect static objects in the plurality of image frames, resulting in a set of static object detections;

processing the video with a moving object detection module to detect moving objects in the plurality of image frames, resulting in a set of moving object detections;

fusing the set of static object detections and the set of moving object detections, resulting in a set of detected objects;

classifying the set of detected objects with an object classification module, resulting in a set of recognized objects; and

outputting the set of recognized objects,

wherein the one or more processors further perform operations of;

rotating each image frame in the plurality of image frames gradually by angles between 0 degrees and 360 degrees;

processing each rotated image frame with the static object detection module to detect a set of static object detections in the rotated image frames;

scaling each image frame in the plurality of image frames within a predetermined range;

processing each scaled image frame with the static object detection module to detect a set of static object detections in the scaled image frames; and

aggregating the set of static object detections in the rotated image frames and the set of static object detections in the scaled image frames to avoid redundant static object detections.

6. The system as set forth in claim 5, wherein if the static object detection module and the moving object detection module both detect an object detection in an area of an image frame, the one or more processors further perform an operation of retaining the moving object detection and ignoring the static object detection when fusing the set of static object detections and the set of moving object detections.

7. The system as set forth in claim 6, wherein the object classification module is a convolutional neural network (CNN) based object classification module.

8. A computer-implemented method for object recognition from moving platforms, comprising:

an act of causing a data processor to execute instructions stored on a non-transitory memory such that upon execution, the data processor performs operations of;

receiving a video captured from a moving platform as input, wherein the video comprises a plurality of image frames;

processing each image frame in the plurality of image frames with a static object detection module to detect at least one of a first type of object, having static object characteristics in the image frame, resulting in at least one static object detection, wherein the at least one first type of object is stationary relative to a ground surface in the image frame;

following processing of the image frame with the static object detection module, processing the image frame with a moving object detection module to detect at least one of a second type of object, having moving object characteristics, in the image frame, resulting in at least one moving object detection;

fusing the at least one static object detection and the at least one moving object detection, resulting in a set of detected objects, wherein an aggregation process removes redundant detections that correspond to the same object;

classifying the set of detected objects with an object classification module, resulting in a set of recognized objects; and

outputting the set of recognized objects.

9. The method as set forth in claim 8, wherein the one or more processors further perform operations of:

extracting Haar features from the plurality of image frames; and

processing the extracted Haar features sequentially with a cascade of classifiers sequentially to detect static objects in the plurality of image frames in the static object detection module.

10. The method as set forth in claim 9, wherein each classifier in the cascade of classifiers is pre-trained using an AdaBoost learning technique.

11. A computer-implemented method for object recognition from moving platforms, comprising:

an act of causing a data processor to execute instructions stored on a non-transitory memory such that upon execution, the data processor performs operations of;

receiving a video captured from a moving platform as input, wherein the video comprises a plurality of image frames;

processing the video with a static object detection module to detect static objects in the plurality of image frames, resulting in a set of static object detections;

processing the video with a moving object detection module to detect moving objects in the plurality of image frames, resulting in a set of moving object detections;

fusing the set of static object detections and the set of moving object detections, resulting in a set of detected objects;

classifying the set of detected objects with an object classification module, resulting in a set of recognized objects; and

outputting the set of recognized objects,

wherein the one or more processors further perform operations of;

rotating each image frame in the plurality of image frames gradually by angles spanned evenly between 0 degrees and 360 degrees;

processing each rotated image frame with the static object detection module to detect a set of static object detections in the rotated image frames;

scaling each image frame in the plurality of image frames within a predetermined range;

processing each scaled image frame with the static object detection module to detect a set of static object detections in the scaled image frames; and

aggregating the set of static object detections in the rotated image frames and the set of static object detections in the scaled image frames to avoid redundant static object detections.

12. The method as set forth in claim 11, wherein if the static object detection module and the moving object detection module both detect an object detection in an area of an image frame, the one or more processors further perform an operation of retaining the moving object detection and ignoring the static object detection when fusing the set of static object detections and the set of moving object detections.

13. The method as set forth in claim 12, wherein the object classification module is a convolutional neural network (CNN) based object classification module.

14. A computer program product for object recognition from moving platforms, the computer program product comprising computer-readable instructions stored on a non-transitory computer-readable medium that are executable by a computer having a processor for causing the processor to perform operations of:

receiving a video captured from a moving platform as input, wherein the video comprises a plurality of image frames;

processing each image frame in the plurality of image frames with a static object detection module to detect at least one of a first type of object, having static object characteristics, in the image frames, resulting in at least one static object detection, wherein the at least one first type of object is stationary relative to a ground surface in the image frame;

following processing of the image frame with the static object detection module, processing the image frame with a moving object detection module to detect at least one of a second type of object, having moving object characteristics, in the image frame, resulting in at least one moving object detection;

fusing the at least one static object detection and the at least one moving object detection, resulting in a set of detected objects, wherein an aggregation process removes redundant detections that correspond to the same object;

classifying the set of detected objects with an object classification module, resulting in a set of recognized objects; and

outputting the set of recognized objects.

15. The computer program product as set forth in claim 14, further comprising instructions for causing the processor to perform operations of:

extracting Haar features from the plurality of image frames; and

processing the extracted Haar features sequentially with a cascade of classifiers sequentially to detect static objects in the plurality of image frames in the static object detection module.

16. The computer program product as set forth in claim 15, wherein each classifier in the cascade of classifiers is pre-trained using an AdaBoost learning technique.

17. A computer program product for object recognition from moving platforms, the computer program product comprising computer-readable instructions stored on a non-transitory computer-readable medium that are executable by a computer having a processor for causing the processor to perform operations of:

receiving a video captured from a moving platform as input, wherein the video comprises a plurality of image frames;

processing the video with a static object detection module to detect static objects in the plurality of image frames, resulting in a set of static object detections;

processing the video with a moving object detection module to detect moving objects in the plurality of image frames, resulting in a set of moving object detections;

fusing the set of static object detections and the set of moving object detections, resulting in a set of detected objects;

classifying the set of detected objects with an object classification module, resulting in a set of recognized objects; and

outputting the set of recognized objects,

wherein the computer program product further comprises instructions for causing the processor to perform operations of;

rotating each image frame in the plurality of image frames gradually by angles spanned evenly between 0 degrees and 360 degrees;

processing each rotated image frame with the static object detection module to detect a set of static object detections in the rotated image frames;

scaling each image frame in the plurality of image frames within a predetermined range;

processing each scaled image frame with the static object detection module to detect a set of static object detections in the scaled image frames; and

aggregating the set of static object detections in the rotated image frames and the set of static object detections in the scaled image frames to avoid redundant static object detections.

18. The computer program product as set forth in claim 17, wherein if the static object detection module and the moving object detection module both detect an object detection in an area of an image frame, the computer program product further comprises instructions for causing the processor to perform an operation of retaining the moving object detection and ignoring the static object detection when fusing the set of static object detections and the set of moving object detections.

19. The computer program product as set forth in claim 18, wherein the object classification module is a convolutional neural network (CNN) based object classification module.

20. A system for object recognition from moving platforms, the system comprising:

a moving platform;

at least one video camera connected with the moving platform;

one or more processors and a non-transitory computer-readable medium having executable instructions encoded thereon such that when executed, the one or more processors perform operations of;

receiving a video captured from the video camera, wherein the video comprises a plurality of image frames;

processing each image frame in the plurality of image frames with a static object detection module to detect at least one of a first type of object, having static object characteristics in the image frame, resulting in at least one static object detection, wherein the at least one first type of object is stationary relative to a around surface in the image frame;

following processing of the image frame with the static object detection module, processing the image frame with a moving object detection module to detect at least one of a second type of object, having moving object characteristics in the image frame, resulting in at least one moving object detection;

fusing the at least one static object detection and the at least one moving object detection, resulting in a set of detected objects, wherein an aggregation process removes redundant detections that correspond to the same object;

classifying the set of detected objects with an object classification module, resulting in a set of recognized objects; and

outputting the set of recognized objects.

21. A system for object recognition from moving platforms, the system comprising:

one or more processors and a non-transitory computer-readable medium having executable instructions encoded thereon such that when executed, the one or more processors perform operations of;

receiving a video captured from a moving platform as input, wherein the video comprises a plurality of image frames;

processing the video with a static object detection module to detect static objects in the plurality of image frames, resulting in a set of static object detections;

processing the video with a moving object detection module to detect moving objects in the plurality of image frames, resulting in a set of moving object detections;

fusing the set of static object detections and the set of moving object detections, resulting in a set of detected objects;

classifying the set of detected objects with an object classification module, resulting in a set of recognized objects; and

outputting the set of recognized objects,

wherein the one or more processors further perform operations of;

rotating each image frame in the plurality of image frames gradually by angles between 0 degrees and 360 degrees;

processing each rotated image frame with the static object detection module to detect a set of static object detections in the rotated image frames;

scaling each image frame in the plurality of image frames within a predetermined range; and

processing each scaled image frame with the static object detection module to detect a set of static object detections in the scaled image frames.

22. The system as set forth in claim 21, wherein the one or more processors further perform an operation of:

aggregating the set of static object detections in the rotated image frames and the set of static object detections in the scaled image frames to avoid redundant static object detections.