System for object recognition in colorized point clouds

• US 8,488,877 B1
• Filed: 12/02/2009
• Issued: 07/16/2013
• Est. Priority Date: 12/02/2009
• Status: Expired due to Fees

• Current Assignee: HRL Laboratories LLC (The Boeing Co.)
• Original Assignee: HRL Laboratories LLC (The Boeing Co.)
• Inventors: Owechko, Yuri, Medasani, Swarup, Azuma, Ronald T., Nelson, Jim
• Primary Examiner(s): Park, Chan S
• Assistant Examiner(s): MOTSINGER, SEAN T
• Application Number:
  US12/592,836
• Time in Patent Office: 1,322 Days
• Field of Search: None
• US Class Current: 382/165
• CPC Class Codes: G06V 20/39 Urban scenes
• CPC Class Codes: G06V 20/64 Three-dimensional objects

Abstract

Described is a system for object recognition in colorized point clouds. The system includes an implicit geometry engine that is configured to receive three-dimensional (3D) colorized cloud point data regarding a 3D object of interest and to convert the cloud point data into implicit representations. The engine also generates geometric features. A geometric grammar block is included to generate object cues and recognize geometric objects using geometric tokens and grammars based on object taxonomy. A visual attention cueing block is included to generate object cues based on 3D geometric properties. Finally, an object recognition block is included to perform a local search for objects using cues from the cueing block and the geometric grammar block and to classify the 3D object of interest as a particular object upon a classifier reaching a predetermined threshold.

Claims

1. A system for object recognition in colorized point clouds, comprising one or more processors that include:

an implicit geometry engine, the implicit geometry engine configured to receive three-dimensional (3D) colorized point cloud data regarding a 3D object of interest, to convert the point cloud data into implicit representations, and to generate geometric features;

a geometric grammar block, the geometric grammar block configured to generate object cues and recognize geometric objects using geometric tokens and finite state machines;

a visual attention cueing block, the visual attention cueing block configured to generate object cues based on two-dimensional (2D) visually salient properties;

an object recognition block, the object recognition block being configured to perform a local search for objects using cues from the cueing block and the geometric grammar block and to classify the 3D object of interest as a particular object upon a classifier reaching a predetermined threshold.

2. A system as set forth in claim 1, wherein the object recognition block further comprises:

a search module, the search module configured to perform a local search for objects using cues from the visual attention cueing block and the geometric grammar block to generate candidate regions of interest (ROIs); and

a statistical classifier block, the statistical classifier block being, configured to, within the candidate ROIs, implement multiple machine learning based object classifiers using geometric features generated by the implicit geometry engine, the statistical classifier block further configured to classify the 3D object of interest as a particular object upon the output of the object classifiers reaching a predetermined threshold.

3. A system as set forth in claim 2, wherein the implicit geometry engine is further configured to receive the 3D colorized point cloud data and process the 3D colorized point cloud data along separate 2D and 3D pathways, such that in the 3D pathway, the 3D colorized point cloud data is converted into implicit representations, while in the 2D pathway, color information in the 3D colorized cloud point data is projected into 2D representations for cueing and recognition of potential objects.

4. A system as set forth in claim 3, wherein the object recognition block is further configured to perform operations of:

performing an object specific search of salient 2D (x, y) locations within the implicit representations to generate candidate ROis;

performing an object specific search of salient locations in (x,y) and altitude (z) if required for the 3D object of interest to generate candidate ROIs;

applying object classifiers to classify the 3D object of interest as a particular object upon the output of the object classifiers reaching a predetermined threshold;

determining if a spatial search is completed within the implicit representation so that there remains no unsearched portions of the implicit representation;

if yes, determining if the 3D object of interest is the last object of interest in the implicit representations;

if yes, then providing a user the classifications of the recognized objects and stopping;

if no, then selecting a next object of interest within the implicit representations and repeating the process above beginning with performing an object specific search; and

if no, then updating search parameters with the unsearched portions of the implicit representation and repeating the process above beginning with performing an object specific search.

5. A system as set forth in claim 4, wherein the search module is further configured to use particle swarm optimization to perform a local search for objects.

6. A system as set forth in claim 2, wherein the search module is further configured to use particle swarm optimization to perform a local search for objects.

7. A system as set forth in claim 1, wherein the implicit geometry engine is further configured to receive the 3D colorized point cloud data and process the 3D colorized point cloud data along separate 2D and 3D pathways, such that in the 3D pathway, the 3D colorized point cloud data is converted into implicit representations, while in the 2D pathway, color information in the 3D colorized cloud point, data is projected into 2D representations for cueing and recognition of potential objects.

8. A system as set forth in claim 1, wherein the object recognition block is further configured to perform operations of:

performing an object specific search of salient 2D (x, y) locations within the implicit representations to generate candidate ROIs;

performing an object specific search of salient locations in (x,y) and altitude (z) if required for the 3D object of interest to generate candidate ROIs;

applying object classifiers to classify the 3D object of interest as a particular object upon the output of the object classifiers reaching a predetermined threshold;

determining if a spatial search is completed within the implicit representation so that there remains no unsearched portions of the implicit representation;

if yes, determining if the 3D object of interest is the last object of interest in the implicit representations;

if yes, then providing a user the classifications of the recognized objects and stopping;

if no, then selecting a next object of interest within the implicit representations and repeating the process above beginning with performing an object specific search; and

if no, then updating search parameters with the unsearched portions of the implicit representation and repeating the process above beginning with performing an object specific search.

19. The computer program product as set forth in claim 1$, further comprising instruction means for causing the processor to perform an operation of receiving, in the implicit geometry engine, the 3D colorized point cloud data and processing the 3D colorized point cloud data along separate 2D and 3D pathways, such that in the 3D pathway, the 3D colorized point cloud data is converted into implicit representations, while in the 2D pathway, color information in the 3D colorized cloud point data is projected into 2D representations for cueing and recognition of potential objects.

20. The computer program product as set forth in claim 19, further comprising instruction means for causing the processor to perform operations of:

performing an object specific search of salient 2D (x, y) locations within the implicit representations to generate candidate ROIs;

performing an object specific search of salient locations in (x, y) and altitude (z) if required for the 3D object of interest to generate candidate ROIs;

applying object classifiers to classify the 3D object of interest as a particular object upon the output of the object classifiers reaching, a predetermined threshold;

determining if a spatial search is completed within the implicit representation so that there remains no unsearched portions of the implicit representation;

if yes, determining if the 3D object of interest is the last object of interest the implicit representations;

if yes, then providing a user the classifications of the recognized objects and stopping;

if no, then selecting a next object of interest within the implicit representations and repeating the process above beginning with performing an object specific search; and

if no, then updating search parameters with the unsearched portions of the implicit representation and repeating the process above beginning with performing an object specific search.

21. The computer program product as set forth in claim 20, further comprising instruction means tier causing the processor to perform an operation of using, in the search module, particle swarm optimization to perform a local search for objects.

9. A computer implemented method for object recognition in colorized point clouds, comprising:

an act of causing a computer having a processor to perform acts of;

receiving, in an implicit geometry engine, a three-dimensional (3D) colorized point cloud data regarding a 3D object of interest, to convert the point cloud data into implicit representations, and to generate geometric features;

generating, in a geometric grammar block, object cues and recognizing geometric objects using geometric tokens and finite state machines;

generating, in a visual attention cueing block, object cues based on two-dimensional (2D) visually salient properties;

performing, in an object recognition block, a local search for objects using cues from the cueing block and the geometric grammar block; and

classifying the 3D object of interest as a particular object upon a classifier reaching a predetermined threshold.

10. The method as set forth in claim 9, further comprising acts of:

performing, in a search module, a local search for objects using cues from the visual attention cueing block and the geometric grammar block to generate candidate regions of interest (ROIs); and

implementing, in a statistical classifier block and within the candidate ROIs, multiple machine learning based object classifiers using, geometric features generated by the implicit geometry engine; and

classifying the 3D object of interest as a particular object upon an output of the object classifiers reaching a predetermined threshold.

11. The method as set forth in claim 10, further comprising an act of receiving, in the implicit geometry engine, the 3D colorized point cloud data and processing the 3D colorized point cloud data along, separate 2D and 3D pathways, such that in the 3D pathway, the 3D colorized point cloud data is converted into implicit representations, while in the 2D pathway, color information in the 3D colorized cloud point, data is projected into 2D representations for cueing and recognition of potential objects.

12. The method as set forth in claim 11, limber comprising acts of:

performing an object specific search of salient 2D (x, y) locations within the implicit representations to generate candidate ROIs;

performing an object specific search of salient locations in (x,y) and altitude (z) if required for the 3D object of interest to generate candidate ROIs;

applying object classifiers to classify the 3D object of interest as a particular object upon the output of the object classifiers reaching a predetermined threshold;

determining if a spatial search is completed within the implicit representation so that there remains no unsearched portions of the implicit representation;

if yes, determining if the 3D object of interest is the last object of interest in the implicit representations;

if yes, then providing a user the classifications of the recognized objects and stopping;

if no, then selecting a next object of interest within the implicit representations and repeating the process above beginning with performing an object specific search; and

if no, then updating search parameters with the unsearched portions of the implicit representation and repeating the process above beginning with performing an object specific search.

13. The method as set forth in claim 12, further comprising an act of using, in the search module, particle swarm optimization to perform a local search for objects.

14. The method as set forth in claim 10, further comprising an act of using, in the search module, particle swarm optimization to perform a local search for objects.

15. The method as set forth in claim 9, further comprising an act of receiving, in the implicit geometry engine, the 3D colorized point cloud data and processing the 3D colorized point cloud data along separate 2D and 3D pathways, such that in the 3D pathway, the 3D colorized point d cloud data is converted into implicit representations, while in the 2D pathway, color information in the 3D colorized cloud point data is projected into 2D representations for cueing and recognition of potential objects.

16. The method as set forth in claim 9, further comprising acts of:

performing an object specific search of salient 2D (x, y) locations within the implicit representations to generate candidate ROIs;

performing, an object specific search of salient locations in (x, y) and altitude (z) if required for the 3D object of interest to generate candidate ROIs;

applying object classifiers to classify the 3D object of interest as a particular object upon the output of the object classifiers reaching a predetermined threshold;

determining if a spatial search is completed within the implicit representation so that there remains no unsearched portions of the implicit representation;

if yes, determining if the 3D object of interest is the last object of interest the implicit representations;

if yes, then providing a user the classifications of the recognized objects and stopping;

if no, then selecting a next object of interest within the implicit representations and repeating the process above beginning with performing an object specific search; and

if no, then updating search parameters with the unsearched portions of the implicit representation and repeating the process above beginning with performing an object specific search.

17. A computer program product for object recognition in colorized point clouds, the computer program product comprising computer-readable instruction means stored on a non-transitory computer-readable medium that are executable by a computer having a processor for causing, the processor to perform operations of:

receiving, in an implicit geometry engine, a three-dimensional (3D) colorized cloud point data regarding a 3D object of interest, to convert the cloud point data into implicit representations, and to generate geometric features;

generating, in a geometric grammar block, object cues and recognizing geometric objects using geometric tokens and finite state machines;

generating, in a visual attention cueing block, object cues based on two-dimensional (2D) visually salient properties;

performing, in an object recognition block, a local search for objects using cues from the cueing block and the geometric grammar block; and

classifying the 3D object of interest as a particular object upon a classifier reaching a predetermined threshold.

18. The computer program product as set forth in claim 17, further comprising instruction means for causing the processor to perform operations of

performing, in a search module, at local search for objects using cues from the visual attention cueing block and the geometric grammar block to generate candidate regions of interest (ROIs);

and

implementing, in a statistical classifier block and within the candidate ROIs, multiple machine learning based object classifiers using geometric;

features generated by the implicit geometry engine; and

classifying the 3D object of interest as a particular object upon an output of the object classifiers reaching a predetermined threshold.

22. The computer program product as set forth in claim 18, further comprising instruction means for causing the processor to perform an operation of using, in the search module, particle swarm optimization to perform a local search for objects.

23. The computer program product as set forth in claim 17, further comprising instruction means for causing the processor to perform an operation of receiving, in the implicit geometry engine, the 3D colorized point cloud data and processing the 3D colorized point cloud data along separate 2D and 3D pathways, such that in the 3D pathway, the 3D colorized point cloud data is converted into implicit representations, while in the 2D pathway, color information in the 3D colorized point cloud data is projected into 2D representations for cueing and recognition of potential objects.

24. The computer program product as set forth in claim 17, further comprising instruction means for causing the processor to perform operations of:

performing an object specific search of salient 2D (x, y) locations within the implicit representations to generate candidate ROIs;

performing an object specific search of salient locations in (x, y) and altitude (z) if required for the 3D object of interest to generate candidate ROIs;

applying object classifiers to classify the 3D object of interest as a particular object upon the output of the object classifiers reaching a predetermined threshold;

determining if a spatial search is completed within the implicit representation so that there remains no unsearched portions of the implicit representation;

if yes, determining if the 3D object of interest is the last object of interest in the implicit representations;

if yes, then providing a user the classifications of the recognized objects and stopping;

if no, then selecting a next object of interest within the implicit representations and repeating the process above beginning;

with performing an object specific search; and

if no, then updating search parameters with the unsearched portions of the implicit representation and repeating the process above beginning with performing an object specific search.