System for object detection and recognition in videos using stabilization

• US 8,885,887 B1
• Filed: 01/23/2013
• Issued: 11/11/2014
• Est. Priority Date: 01/23/2012
• Status: Active Grant

• Current Assignee: HRL Laboratories LLC (The Boeing Co.)
• Original Assignee: HRL Laboratories LLC (The Boeing Co.)
• Inventors: Chen, Yang, Kim, Kyungnam, Khosla, Deepak, Cheng, Shinko Y.
• Primary Examiner(s): BHATNAGAR, ANAND P
• Application Number:
  US13/748,522
• Time in Patent Office: 657 Days
• Field of Search: 382/100, 382/103, 382/107, 382/170, 382/181, 382/190, 382/195, 382/254, 382/275, 348/135, 348/143, 348/144, 348/497, 345/611, 702/69
• US Class Current: 382/103
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10032 Satellite or aerial image; Remote sensing
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 5/002 Denoising; Smoothing noise processing or correction adapted to be used in an image pickup device containing and electronic image sensor H04N23/81, H04N25/60 - H04N25/67
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments

Abstract

Described is a system for stabilizing, detecting, and recognizing objects in video captured from a mobile platform. The system first receives a video (with a plurality of image frames) captured from a mobile platform. The video is stabilized by registering the image frames to a global coordinate system to generate stabilized image frames. A bio-inspired attention algorithm is applied to the stabilized image frames to produce a set of locations in the stabilized image frames that are salient points representative of an object of interest. An image chip is generated that surrounds each salient point. High-dimensional feature vectors are extracted from the image chip. The feature vectors are then classified as an object class. Thus, through classifying the feature vectors, an object of interest can be identified in the video as captured from the mobile platform.

Claims

1. A system for stabilizing, detecting, and recognizing objects in video captured from a mobile platform, comprising:

one or more processors and a memory, the memory having instructions encoded thereon such that upon execution of the instructions, the one or more processors perform operations of;

receiving a video captured from a mobile platform, the video having a plurality of image frames;

stabilizing the video by registering the image frames to a global coordinate system to generate stabilized image frames;

applying a bio-inspired attention algorithm to the stabilized image frames to produce a set of locations in the stabilized image frames that are salient points representative of an object of interest;

generating an image chip that surrounds each salient point;

extracting high-dimensional feature vectors from the image chip; and

classifying the feature vectors as an object class, whereby through classifying the feature vectors, an object of interest can be identified in the video as captured from the mobile platform.

2. The system as set forth in claim 1, wherein stabilizing the video further comprises operations of:

extracting key points and feature descriptors in the image frames;

matching key points between two images of the image frames to generate a list of candidate set of matching points from the two images;

generating a homography transformation between the two images that fits the candidate set of matching points; and

warping the image frames using the homography transformation to generate the stabilized image frames.

3. The system as set forth in claim 2, wherein a match for a key point in a first image is defined as the key point in the second image with a minimum Euclidean distance based on descriptor vectors of the key points.

4. The system as set forth in claim 2, wherein in warping the image frames, a block size of N frames is selected, in which each frame is warped to the coordinates of a first frame in the block using the homography transformation.

5. The system as set forth in claim 2, wherein in warping the image frames, each image frame is warped to a previous image frame using the homography transformation.

6. The system as set forth in claim 2, wherein in receiving the video captured from the mobile platform, the mobile platform is an airplane, such that the video is an aerial video.

7. The system as set forth in claim 1, wherein in receiving the video captured from the mobile platform, the mobile platform is an airplane, such that the video is an aerial video.

8. A computer implemented method for stabilizing, detecting, and recognizing objects in video captured from a mobile platform, comprising an act of causing a data processor to execute instructions stored on a non-transitory computer-readable medium such that upon execution, the data processor performs operations of:

receiving a video captured from a mobile platform, the video having a plurality of image frames;

stabilizing the video by registering the image frames to a global coordinate system to generate stabilized image frames;

applying a bio-inspired attention algorithm to the stabilized image frames to produce a set of locations in the stabilized image frames that are salient points representative of an object of interest;

generating an image chip that surrounds each salient point;

extracting high-dimensional feature vectors from the image chip; and

classifying the feature vectors as an object class, whereby through classifying the feature vectors, an object of interest can be identified in the video as captured from the mobile platform.

9. The method as set forth in claim 8, wherein stabilizing the video further comprises operations of:

extracting key points and feature descriptors in the image frames;

matching key points between two images of the image frames to generate a list of candidate set of matching points from the two images;

generating a homography transformation between the two images that fits the candidate set of matching points; and

warping the image frames using the homography transformation to generate the stabilized image frames.

10. The method as set forth in claim 9, wherein a match for a key point in a first image is defined as the key point in the second image with a minimum Euclidean distance based on descriptor vectors of the key points.

11. The method as set forth in claim 9, wherein in warping the image frames, a block size of N frames is selected, in which each frame is warped to the coordinates of a first frame in the block using the homography transformation.

12. The method as set forth in claim 9, wherein in warping the image frames, each image frame is warped to a previous image frame using the homography transformation.

13. The method as set forth in claim 9, wherein in receiving the video captured from the mobile platform, the mobile platform is an airplane, such that the video is an aerial video.

14. The method as set forth in claim 8, wherein in receiving the video captured from the mobile platform, the mobile platform is an airplane, such that the video is an aerial video.

15. A computer program product for stabilizing, detecting, and recognizing objects in video captured from a mobile platform, the computer program product comprising computer-readable instructions stored on a non-transitory computer-readable medium that are executable by a computer having a processor for causing the processor to perform operations of:

receiving a video captured from a mobile platform, the video having a plurality of image frames;

stabilizing the video by registering the image frames to a global coordinate system to generate stabilized image frames;

applying a bio-inspired attention algorithm to the stabilized image frames to produce a set of locations in the stabilized image frames that are salient points representative of an object of interest;

generating an image chip that surrounds each salient point;

extracting high-dimensional feature vectors from the image chip; and

classifying the feature vectors as an object class, whereby through classifying the feature vectors, an object of interest can be identified in the video as captured from the mobile platform.

16. The computer program product as set forth in claim 15, wherein stabilizing the video further comprises operations of:

extracting key points and feature descriptors in the image frames;

matching key points between two images of the image frames to generate a list of candidate set of matching points from the two images;

generating a homography transformation between the two images that fits the candidate set of matching points; and

warping the image frames using the homography transformation to generate the stabilized image frames.

17. The computer program product as set forth in claim 16, wherein a match for a key point in a first image is defined as the key point in the second image with a minimum Euclidean distance based on descriptor vectors of the key points.

18. The computer program product as set forth in claim 16, wherein in warping the image frames, a block size of N frames is selected, in which each frame is warped to the coordinates of a first frame in the block using the homography transformation.

19. The computer program product as set forth in claim 16, wherein in warping the image frames, each image frame is warped to a previous image frame using the homography transformation.

20. The computer program product as set forth in claim 16, wherein in receiving the video captured from the mobile platform, the mobile platform is an airplane, such that the video is an aerial video.

21. The computer program product as set forth in claim 15, wherein in receiving the video captured from the mobile platform, the mobile platform is an airplane, such that the video is an aerial video.