System for detecting an object of interest in a scene

• US 9,378,420 B1
• Filed: 05/30/2014
• Issued: 06/28/2016
• Est. Priority Date: 06/04/2013
• Status: Active Grant

• Current Assignee: HRL Laboratories LLC (The Boeing Co.)
• Original Assignee: HRL Laboratories LLC (The Boeing Co.)
• Inventors: Flores, A. Arturo, Cheng, Shinko Y, Mundhenk, Terrell N., Neely, Howard III, Daily, Michael J.
• Primary Examiner(s): Koziol, Stephen R
• Assistant Examiner(s): Gilliard, Delomia L
• Application Number:
  US14/292,416
• Time in Patent Office: 760 Days
• Field of Search: 382/103
• US Class Current: 1/1
• CPC Class Codes: G06F 18/2411 based on the proximity to a decision surface, e.g. support vector machines
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/464 using a plurality of salient features, e.g. bag-of-words [BoW] representations
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation

Abstract

The present invention relates to a system for detecting an object of interest in a scene. The system operates by receiving an image frame of a scene and extracting features from the image frame, the features being descriptors. The descriptors are quantized to generate PHOW features. A sliding window protocol is implemented to slide a window over the image and analyze the PHOW features that fall inside the window. Finally, the system determines if the PHOW features represent the object of interest and, if so, then designates the window as a location in the image with a detected object of interest.

Claims

1. An system for detecting an object of interest in a scene, comprising:

one or more processors and a memory, the memory having executable instructions encoded thereon, such that upon execution of the instructions, the one or more processors perform operations of;

receiving an image frame of a scene;

extracting features from the image frame, the features being descriptors, wherein the features are extracted from the image frame by performing operations of;

creating three threads, one for each of three independent scales;

providing, for each thread, a link to the image frame and a set of running parameters;

running each thread, in parallel, to identify descriptors in each of the three threads;

upon completion of the three threads, compiling the results from each of the three threads into a full set of dense Scale Invariant Feature Transform (SIFT) descriptors (DSIFT);

quantizing the DSIFT descriptors to generate a pyramid histogram of visual word (PHOW) features;

implementing a sliding window protocol to slide a window over the image and analyze PHOW features that fall inside the window; and

determining if the PHOW features represent the object of interest and, if so, then designating the window as a location in the image with a detected object of interest.

2. The system as set forth in claim 1, wherein determining if the PHOW features represent the object of interest further comprises operations of:

subdividing the window into a set of bins;

compiling, in each bin, a histogram of the PHOW features, the histograms representing a count of PHOW features found in each bin;

transforming the histograms by a Chi-Square homogeneous kernel transform;

feeding the transformed histograms into a support vector machine (SVM) classifier to generate an SVM score for each feature, the SVM scores representing a similarity of an object in the image to a target object;

squashing the SVM scores with a tangential sigmoid so that the SVM scores range from −

1.0 to 1.0;

collecting the SMV scores into a list of scores;

reducing the list of scores using Non-Maximal Suppression to generate a reduced list of scores; and

applying a threshold to the reduced list of scores to eliminate any scores below a predetermined threshold and generate object scores, the object scores representing a likelihood that a particular object in the scene is the object of interest.

3. The system as set forth in claim 2,

wherein quantizing the descriptors to generate PHOW features further comprises an operation of:

for each descriptor, performing a KD-tree query to identify a learned feature that the descriptor is most similar to, the identified features being the PHOW features;

wherein in implementing the sliding window protocol, the image frame is divided into a plurality of stripes, such that within each stripe are all the windows which might be run at that location in the image frame; and

further comprising an operation of implementing a saliency mask to determine attention zones in the image frame for processing with the sliding window protocol and zones in the image frame that are to be ignored by the sliding window protocol.

4. The system as set forth in claim 1, wherein quantizing the descriptors to generate PHOW features further comprises an operation of:

for each descriptor, performing a KD-tree query to identify a learned feature that the descriptor is most similar to, the identified features being the PHOW features.

5. The system as set forth in claim 1, wherein in implementing the sliding window protocol, the image frame is divided into a plurality of stripes, such that within each stripe are all the windows which might be run at that location in the image frame.

6. The system as set forth in claim 1, further comprising an operation of implementing a saliency mask to determine attention zones in the image frame for processing with the sliding window protocol and zones in the image frame that are to be ignored by the sliding window protocol.

7. A computer program product for detecting an object of interest in a scene, the computer program product comprising:

a non-transitory computer-readable medium having executable instructions encoded thereon, such that upon execution of the instructions by one or more processors, the one or more processors perform operations of;

receiving an image frame of a scene;

extracting features from the image frame, the features being descriptors, wherein the features are extracted from the image frame by performing operations of;

creating three threads, one for each of three independent scales;

providing, for each thread, a link to the image frame and a set of running parameters;

running each thread, in parallel, to identify descriptors in each of the three threads;

upon completion of the three threads, compiling the results from each of the three threads into a full set of dense Scale Invariant Feature Transform (SIFT) descriptors (DSIFT);

quantizing the DSIFT descriptors to generate a pyramid histogram of visual word (PHOW) features;

implementing a sliding window protocol to slide a window over the image and analyze PHOW features that fall inside the window; and

determining if the PHOW features represent the object of interest and, if so, then designating the window as a location in the image with a detected object of interest.

8. The computer program product as set forth in claim 7, wherein determining if the PHOW features represent the object of interest further comprises operations of:

subdividing the window into a set of bins;

compiling, in each bin, a histogram of the PHOW features, the histograms representing a count of PHOW features found in each bin;

transforming the histograms by a Chi-Square homogeneous kernel transform;

feeding the transformed histograms into a support vector machine (SVM) classifier to generate an SVM score for each feature, the SVM scores representing a similarity of an object in the image to a target object;

squashing the SVM scores with a tangential sigmoid so that the SVM scores range from −

1.0 to 1.0;

collecting the SMV scores into a list of scores;

reducing the list of scores using Non-Maximal Suppression to generate a reduced list of scores; and

applying a threshold to the reduced list of scores to eliminate any scores below a predetermined threshold and generate object scores, the object scores representing a likelihood that a particular object in the scene is the object of interest.

9. The computer program product as set forth in claim 8,

wherein quantizing the descriptors to generate PHOW features further comprises an operation of:

for each descriptor, performing a KD-tree query to identify a learned feature that the descriptor is most similar to, the identified features being the PHOW features;

wherein in implementing the sliding window protocol, the image frame is divided into a plurality of stripes, such that within each stripe are all the windows which might be run at that location in the image frame; and

further comprising an operation of implementing a saliency mask to determine attention zones in the image frame for processing with the sliding window protocol and zones in the image frame that are to be ignored by the sliding window protocol.

10. The computer program product as set forth in claim 7, wherein quantizing the descriptors to generate PHOW features further comprises an operation of:

for each descriptor, performing a KD-tree query to identify a learned feature that the descriptor is most similar to, the identified features being the PHOW features.

11. The computer program product as set forth in claim 7, wherein in implementing the sliding window protocol, the image frame is divided into a plurality of stripes, such that within each stripe are all the windows which might be run at that location in the image frame.

12. The computer program product as set forth in claim 7, further comprising an operation of implementing a saliency mask to determine attention zones in the image frame for processing with the sliding window protocol and zones in the image frame that are to be ignored by the sliding window protocol.

13. A computer implemented method for detecting an object of interest in a scene, the method comprising an act of causing one or more processors to execute instructions encoded on a non-transitory computer-readable medium, such that upon execution of the instructions, the one or more processors perform operations of:

receiving an image frame of a scene;

extracting features from the image frame, the features being descriptors, wherein the features are extracted from the image frame by performing operations of;

creating three threads, one for each of three independent scales;

providing, for each thread, a link to the image frame and a set of running parameters;

running each thread, in parallel, to identify descriptors in each of the three threads;

upon completion of the three threads, compiling the results from each of the three threads into a full set of dense Scale Invariant Feature Transform (SIFT) descriptors (DSIFT);

quantizing the DSIFT descriptors to generate a pyramid histogram of visual word (PHOW) features;

implementing a sliding window protocol to slide a window over the image and analyze PHOW features that fall inside the window; and

determining if the PHOW features represent the object of interest and, if so, then designating the window as a location in the image with a detected object of interest.

14. The computer implemented method as set forth in claim 13, wherein determining if the PHOW features represent the object of interest further comprises operations of:

subdividing the window into a set of bins;

compiling, in each bin, a histogram of the PHOW features, the histograms representing a count of PHOW features found in each bin;

transforming the histograms by a Chi-Square homogeneous kernel transform;

feeding the transformed histograms into a support vector machine (SVM) classifier to generate an SVM score for each feature, the SVM scores representing a similarity of an object in the image to a target object;

squashing the SVM scores with a tangential sigmoid so that the SVM scores range from −

1.0 to 1.0;

collecting the SMV scores into a list of scores;

reducing the list of scores using Non-Maximal Suppression to generate a reduced list of scores; and

applying a threshold to the reduced list of scores to eliminate any scores below a predetermined threshold and generate object scores, the object scores representing a likelihood that a particular object in the scene is the object of interest.

15. The computer implemented method as set forth in claim 14,

wherein quantizing the descriptors to generate PHOW features further comprises an operation of:

for each descriptor, performing a KD-tree query to identify a learned feature that the descriptor is most similar to, the identified features being the PHOW features;

wherein in implementing the sliding window protocol, the image frame is divided into a plurality of stripes, such that within each stripe are all the windows which might be run at that location in the image frame; and

further comprising an operation of implementing a saliency mask to determine attention zones in the image frame for processing with the sliding window protocol and zones in the image frame that are to be ignored by the sliding window protocol.

16. The computer implemented method as set forth in claim 13, wherein quantizing the descriptors to generate PHOW features further comprises an operation of:

for each descriptor, performing a KD-tree query to identify a learned feature that the descriptor is most similar to, the identified features being the PHOW features.

17. The computer implemented method as set forth in claim 13, wherein in implementing the sliding window protocol, the image frame is divided into a plurality of stripes, such that within each stripe are all the windows which might be run at that location in the image frame.

18. The computer implemented method as set forth in claim 13, further comprising an operation of implementing a saliency mask to determine attention zones in the image frame for processing with the sliding window protocol and zones in the image frame that are to be ignored by the sliding window protocol.