System for automatic object localization based on visual simultaneous localization and mapping (SLAM) and cognitive swarm recognition

• US 8,649,565 B1
• Filed: 06/22/2010
• Issued: 02/11/2014
• Est. Priority Date: 06/18/2009
• Status: Expired due to Fees

• Current Assignee: HRL Laboratories LLC (The Boeing Co.)
• Original Assignee: HRL Laboratories LLC (The Boeing Co.)
• Inventors: Kim, Kyungnam, Daily, Michael
• Primary Examiner(s): Abdi, Amara
• Application Number:
  US12/821,063
• Time in Patent Office: 1,330 Days
• Field of Search: 340/995.1, 342/145, 382/106, 700/253
• US Class Current: 382/106
• CPC Class Codes: G06F 18/2111 by using evolutionary computational techniques, e.g. genetic algorithms
• CPC Class Codes: G06F 18/254 of classification results, e.g. of results related to same input data
• CPC Class Codes: G06V 10/771 Feature selection, e.g. selecting representative features from a multi-dimensional feature space
• CPC Class Codes: G06V 10/809 of classification results, e.g. where the classifiers operate on the same input data
• CPC Class Codes: G06V 40/103 Static body considered as a whole, e.g. static pedestrian or occupant recognition

Abstract

Described is a system for automatic object localization based on visual simultaneous localization and mapping (SLAM) and cognitive swarm recognition. The system is configured to detect a set of location data corresponding to a current location of a sensor positioned on a platform. A map model of an environment surrounding the sensor is generated based on an input image from the sensor and the location data. In a desired aspect, a cognitive swarm object detection module is used to search for and detect an object of interest. The three-dimensional location of the object of interest relative to the platform is then estimated based on the map model and the location data regarding the sensor. The system described allows for real-time, continuous three-dimensional location updating for moving objects of interest from a mobile platform. A computer-implemented method and computer program product are also described.

Claims

1. A system for automatic object localization, the system comprising one or more processors configured to perform operations of:

detecting a set of three-dimensional location data, wherein the three-dimensional location data corresponds to a current location of at least one sensor positioned on a platform;

receiving input from the at least one sensor, resulting in an input image;

generating a map model of an environment surrounding the at least one sensor based on the input image and the three-dimensional location data;

detecting and classifying an object of interest in the input image using a cognitive swarm detection module;

estimating a three-dimensional location of the object of interest relative to the platform based on the map model and the three-dimensional location data of the at least one sensor using a localization module;

estimating a pose of the object of interest relative to the platform based on the map model and the three-dimensional location data of the at least on sensor using the localization module;

providing the three-dimensional location data from the localization module as feedback to the cognitive swarm detection module to refine the input image;

wherein the system is further configured to estimate the three-dimensional location and pose of the object of interest while the platform and the object of interest are moving, and continuously update the estimated location of the object of interest in the map model in real-time; and

wherein the cognitive swarm detection module runs in parallel with the localization module for real-time processing.

2. The system for automatic object localization as set forth in claim 1, wherein in the act of generating a map model, the system is further configured to perform operations of:

extracting a set of feature points from the input image;

storing the set of feature points in a map database;

comparing the set of feature points extracted from the input image with a set of feature points extracted from a previous image; and

updating the map model continuously, such that the locations of matching feature points between the input image and the previous image are updated in the map model.

3. The system for automatic object localization as set forth in claim 2, wherein the system is further configured to detect and track a set of feature points representing the object of interest.

4. The system for automatic object localization as set forth in claim 1, further comprising multiple sensors positioned on the platform, wherein the multiple sensors are positioned such that a 360 degree view of the environment is obtained.

5. The system for automatic object localization as set forth in claim 1, wherein the system is further configured to continuously update a six degree-of-freedom location and pose of the platform in the map model.

6. The system for automatic object localization as set forth in claim 1, further comprising a computation platform using parallelized computing for real-time processing on multiple sensors.

7. A computer-implemented method for automatic object localization, the method comprising an act of causing a processor to perform operations of:

detecting a set of three-dimensional location data, wherein the three-dimensional location data corresponds to a current location of at least one sensor positioned on a platform;

receiving input from the at least one sensor, resulting in an input image;

generating a map model of an environment surrounding the at least one sensor based on the input image and the three-dimensional location data;

detecting and classifying an object of interest in the input image using a cognitive swarm detection module;

estimating a three-dimensional location of the object of interest relative to the platform based on the map model and the three-dimensional location data of the at least one sensor using a localization module;

estimating a pose of the object of interest relative to the platform based on the map model and the three-dimensional location data of the at least on sensor using the localization module;

providing the three-dimensional location data from the localization module as feedback to the cognitive swarm detection module to refine the input image;

wherein the system is further configured to estimate the three-dimensional location and pose of the object of interest while the platform and the object of interest are moving, and continuously update the estimated location of the object of interest in the map model in real-time; and

wherein the cognitive swarm detection module runs in parallel with the localization module for real-time processing.

8. The method for automatic object localization as set forth in claim 7, wherein in the act of generating a map model, the method further comprises acts of:

extracting a set of feature points from the input image;

storing the set of feature points in a map database; and

comparing the set of feature points extracted from the input image with a set of feature points extracted from a previous image; and

updating the map model continuously, such that the locations of matching feature points between the input image and the previous image are updated in the map model.

9. The method for automatic object localization as set forth in claim 8, further comprising an act of detecting and tracking a set of feature points representing the object of interest.

10. The method for automatic object localization as set forth in claim 7, further comprising an act of positioning multiple sensors on the platform, wherein the multiple sensors are positioned such that a 360 degree view of the environment is obtained.

11. The method for automatic object localization as set forth in claim 7, further comprising an act of continuously updating a six degree-of-freedom location and pose of the platform in the map model.

12. The method for automatic object localization as set forth in claim 7, further comprising an act of using parallelized computing of a computing platform for real-time processing on multiple sensors.

13. A computer program product for automatic object localization, the computer program product comprising non-transitory computer-readable instruction means stored on a computer-readable medium that are executable by a computer having a processor for causing the processor to perform operations of:

detecting a set of three-dimensional location data, wherein the three-dimensional location data corresponds to a current location of at least one sensor positioned on a platform;

receiving input from the at least one sensor, resulting in an input image;

generating a map model of an environment surrounding the at least one sensor based on the input image and the three-dimensional location data;

detecting and classifying an object of interest in the input image using a cognitive swarm detection module;

estimating a three-dimensional location of the object of interest relative to the platform based on the map model and the three-dimensional location data of the at least one sensor using a localization module;

estimating a pose of the object of interest relative to the platform based on the may model and the three-dimensional location data of the at least on sensor using the localization module;

providing the three-dimensional location data from the localization module as feedback to the cognitive swarm detection module to refine the input image;

wherein the system is further configured to estimate the three-dimensional location and pose of the object of interest while the platform and the object of interest are moving, and continuously update the estimated location of the object of interest in the map model in real-time; and

wherein the cognitive swarm detection module runs in parallel with the localization module for real-time processing.

14. The computer program product for automatic object localization as set forth in claim 13, wherein in the act of generating a map model, further comprising instruction means for causing the processor to perform operations of:

extracting a set of feature points from the input image;

storing the set of feature points in a map database; and

comparing the set of feature points extracted from the input image with a set of feature points extracted from a previous image; and

updating the map model continuously, such that the locations of matching feature points between the input image and the previous image are updated in the map model.

15. The computer program product for automatic object localization as set forth in claim 14, further comprising instruction means for detecting and tracking a set of feature points representing the object of interest.

16. The computer program product for automatic object localization as set forth in claim 13 further comprising instruction means for continuously updating a six degree-of-freedom location and pose of the platform in the map model.

17. The computer program product for automatic object localization as set forth in claim 13, further comprising instruction means for parallelized computing of a computing platform for real-time processing on multiple sensors.