Real-time system for multi-modal 3D geospatial mapping, object recognition, scene annotation and analytics

• US 9,476,730 B2
• Filed: 12/18/2014
• Issued: 10/25/2016
• Est. Priority Date: 03/18/2014
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Samarasekera, Supun, Hadsell, Raia, Kumar, Rakesh, Sawhney, Harpreet S., Matei, Bogdan C., Villamil, Ryan
• Primary Examiner(s): Werner, Brian P
• Application Number:
  US14/575,495
• Publication Number:
  US 20150269438A1
• Time in Patent Office: 677 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G01C 11/02 Picture taking arrangements specially adapted for photogrammetry or photographic surveying, e.g. controlling overlapping of pictures
• CPC Class Codes: G01C 21/3673 Labelling using text of road map data items, e.g. road names, POI names
• CPC Class Codes: G01C 21/3848 Data obtained from both position sensors and additional sensors
• CPC Class Codes: G01C 21/3852 Data derived from aerial or satellite images
• CPC Class Codes: G01C 21/3867 Geometry of map features, e.g. shape points, polygons or for simplified maps
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06V 10/811 the classifiers operating on different input data, e.g. multi-modal recognition
• CPC Class Codes: G06V 20/176 Urban or other man-made structures
• CPC Class Codes: G08G 5/0004 Transmission of traffic-related information to or from an aircraft airborne radio transmission systems in general H04B7/185; airborne wireless networks H04W84/06
• CPC Class Codes: G08G 5/003 Flight plan management
• CPC Class Codes: G08G 5/0069 specially adapted for an unmanned aircraft
• CPC Class Codes: G08G 5/0073 Surveillance aids scene image recognition G06V20/00

Abstract

A multi-sensor, multi-modal data collection, analysis, recognition, and visualization platform can be embodied in a navigation capable vehicle. The platform provides an automated tool that can integrate multi-modal sensor data including two-dimensional image data, three-dimensional image data, and motion, location, or orientation data, and create a visual representation of the integrated sensor data, in a live operational environment. An illustrative platform architecture incorporates modular domain-specific business analytics “plug ins” to provide real-time annotation of the visual representation with domain-specific markups.

Claims

1. A mobile computing device, comprising:

one or more processors, and, in communication with the one or more processors;

one or more image sensors configured to obtain two-dimensional image data and three-dimensional image data;

one or more non-transitory machine accessible storage media comprising instructions to cause the mobile computing device to;

temporally and spatially align the two-dimensional image data and three-dimensional image data;

generate a map representation of a geo-spatial area of the real world surroundings of the mobile computing device based on the temporally and spatially aligned two-dimensional and three-dimensional image data; and

recognize a plurality of visual features in the map representation, using one or more computer vision algorithms to;

recognize larger-scale objects;

recognize smaller-scale objects by iteratively performing context-free object identification and contextual object identification; and

recognize a complex object comprising a plurality of the smaller-scale objects, using a classifier.

2. The mobile computing device of claim 1, comprising instructions to cause the mobile computing device to detect the larger-scale objects by determining a contextual frame of reference, and use the contextual frame of reference to identify the larger-scale objects.

3. The mobile computing device of claim 1, comprising instructions to cause the mobile computing device to recognize the larger-scale objects by executing an invariant three-dimensional feature detection algorithm directly on point cloud data obtained from at least one of the image sensors.

4. The mobile computing device of claim 3, comprising instructions to cause the mobile computing device to recognize the larger-scale objects by executing an invariant two-dimensional feature detection algorithm.

5. The mobile computing device of claim 1, comprising instructions to cause the mobile computing device to recognize the larger-scale objects by executing an invariant two-dimensional feature detection algorithm.

6. The mobile computing device of claim 2, comprising instructions to cause the mobile computing device to recognize the smaller-scale objects by executing a context-free feature-sharing algorithm.

7. The mobile computing device of claim 6, comprising instructions to cause the mobile computing device to recognize the smaller-scale objects by obtaining context information and classifying the smaller-scale objects based on the context information.

8. The mobile computing device of claim 6, comprising instructions to cause the mobile computing device to recognize the complex objects by executing a contextual bag of objects algorithm.

9. An object/scene recognition system comprising instructions embodied in one or more non-transitory computer readable storage media to and executable by one or more processors to cause a mobile computing device to:

obtain two-dimensional image data and three-dimensional image data from one or more image sensors;

temporally and spatially align the two-dimensional image data and three-dimensional image data;

recognize a plurality of visual features in the image data, using one or more computer vision algorithms to;

recognize larger-scale objects;

recognize smaller-scale objects by iteratively performing context-free object identification and contextual object identification; and

recognize a complex object comprising a plurality of the recognized smaller-scale objects, using a classifier.

10. The system of claim 9, comprising instructions to cause the mobile computing device to detect the larger-scale objects by determining a contextual frame of reference and use the contextual frame of reference to identify the larger-scale objects.

11. The system of claim 9, comprising instructions to cause the mobile computing device to recognize the larger-scale objects by executing an invariant three-dimensional feature detection algorithm directly on point cloud data obtained from at least one of the image sensors.

12. The system of claim 11, comprising instructions to cause the mobile computing device to recognize the larger-scale objects by executing an invariant two-dimensional feature detection algorithm.

13. The system of claim 9, comprising instructions to cause the mobile computing device to recognize the larger-scale objects by executing an invariant two-dimensional feature detection algorithm.

14. The system of claim 10, comprising instructions to cause the mobile computing device to recognize the smaller-scale objects by executing a context-free feature-sharing algorithm.

15. The mobile computing device of claim 14, comprising instructions to cause the mobile computing device to recognize the smaller-scale objects by obtaining context information and classifying the smaller-scale objects based on the context information.

16. The mobile computing device of claim 14, comprising instructions to cause the mobile computing device to recognize the complex objects by executing a contextual bag of objects algorithm.

17. An object/scene recognition method comprising, with one or more mobile computing devices:

obtaining two-dimensional image data and three-dimensional image data from one or more image sensors;

temporally and spatially aligning the two-dimensional image data and the three-dimensional image data; and

recognizing a plurality of visual features in the image data by;

recognizing larger-scale objects;

recognizing smaller-scale objects by iteratively performing context-free object identification and contextual object identification; and

recognizing a complex object comprising a plurality of the smaller-scale objects using a classifier.

18. The method of claim 17, comprising determining a contextual frame of reference and using the contextual frame of reference to identify the larger-scale objects.

19. The method of claim 17, comprising executing an invariant three-dimensional feature detection algorithm directly on point cloud data obtained from at least one of the image sensors.

20. The method of claim 18, comprising, iteratively:

executing a context-free feature-sharing algorithm to recognize the smaller-scale objects, obtaining context information, and classifying the smaller-scale objects based on the context information.