Real-time system for multi-modal 3D geospatial mapping, object recognition, scene annotation and analytics

• US 9,911,340 B2
• Filed: 11/07/2016
• Issued: 03/06/2018
• Est. Priority Date: 03/18/2014
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Samarasekera, Supun, Hadsell, Raia, Kumar, Rakesh, Sawhney, Harpreet S., Matei, Bogdan C., Villamil, Ryan
• Primary Examiner(s): Dunn, Alex C
• Application Number:
  US15/344,900
• Publication Number:
  US 20170053538A1
• Time in Patent Office: 484 Days
• CPC Class Codes: G01C 11/02 Picture taking arrangements specially adapted for photogrammetry or photographic surveying, e.g. controlling overlapping of pictures
• CPC Class Codes: G01C 21/3673 Labelling using text of road map data items, e.g. road names, POI names
• CPC Class Codes: G01C 21/3848 Data obtained from both position sensors and additional sensors
• CPC Class Codes: G01C 21/3852 Data derived from aerial or satellite images
• CPC Class Codes: G01C 21/3867 Geometry of map features, e.g. shape points, polygons or for simplified maps
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06V 10/811 the classifiers operating on different input data, e.g. multi-modal recognition
• CPC Class Codes: G06V 20/176 Urban or other man-made structures
• CPC Class Codes: G08G 5/0004 Transmission of traffic-related information to or from an aircraft airborne radio transmission systems in general H04B7/185; airborne wireless networks H04W84/06
• CPC Class Codes: G08G 5/003 Flight plan management
• CPC Class Codes: G08G 5/0069 specially adapted for an unmanned aircraft
• CPC Class Codes: G08G 5/0073 Surveillance aids scene image recognition G06V20/00

Abstract

A multi-sensor, multi-modal data collection, analysis, recognition, and visualization platform can be embodied in a navigation capable vehicle. The platform provides an automated tool that can integrate multi-modal sensor data including two-dimensional image data, three-dimensional image data, and motion, location, or orientation data, and create a visual representation of the integrated sensor data, in a live operational environment. An illustrative platform architecture incorporates modular domain-specific business analytics “plug ins” to provide real-time annotation of the visual representation with domain-specific markups.

Claims

1. A mobile computing device, comprising:

one or more processors;

one or more image sensors in communication with the one or more processors, the one or more image sensors configured to obtain multi-dimensional image data including at least one of two-dimensional image data and three-dimensional image data; and

one or more non-transitory machine accessible storage media in communication with the one or more processors, the one or more non-transitory machine accessible storage media comprising instructions to cause the mobile computing device to perform recognition of a plurality of visual features based on a map representation of a geo-spatial area of real world surroundings of the mobile computing device generated based on temporal and spatial alignment of the multi-dimensional image data, and

wherein the recognition of the plurality of visual features includes recognition of larger-scale objects, recognition of smaller-scale objects by performing context-free object identification and contextual object identification, and recognition of a complex object comprising a plurality of the smaller-scale objects.

2. The mobile computing device of claim 1, wherein, the one or more non-transitory machine accessible storage media comprise instructions to cause the mobile computing device to temporally and spatially align the multi-dimensional image data and generate the map representation of the geo-spatial area of the real world surroundings of the mobile computing device based on the temporally and spatially aligned multi-dimensional image data.

3. The mobile computing device of claim 1, wherein the recognition of the complex object is performed using a classifier.

4. The mobile computing device of claim 1, wherein the recognition of the larger-scale objects is performed by determining a contextual frame of reference, and using the contextual frame of reference to identify the larger-scale objects.

5. The mobile computing device of claim 1, wherein the recognition of the larger-scale objects is performed by executing an invariant multi-dimensional feature detection algorithm on data from at least one of the image sensors.

6. The mobile computing device of claim 1, wherein the recognition of the smaller-scale objects is performed by executing a context-free feature-sharing algorithm.

7. The mobile computing device of claim 6, wherein the recognition of the smaller-scale objects is performed by obtaining context information and classifying the smaller-scale objects based on the context information.

8. The mobile computing device of claim 7, wherein the recognition of the complex objects is performed by executing a contextual bag of objects algorithm.

9. An object/scene recognition system comprising instructions embodied in one or more non-transitory computer readable storage media executable by one or more processors to cause a mobile computing device to:

perform recognition of a plurality of visual features based on a map representation of a geo-spatial area of real world surroundings of the mobile computing device generated based on temporal and spatial alignment of multi-dimensional image data obtained by one or more sensors,

wherein the multi-dimensional image data includes at least one of two-dimensional image data and three-dimensional image data, and

wherein the recognition of the plurality of visual features includes recognition of larger-scale objects, recognition of smaller-scale objects by performing context-free object identification and contextual object identification, and recognition of a complex object comprising a plurality of the smaller-scale objects.

10. The object/scene recognition system of claim 9 further comprising instructions embodied in the one or more non-transitory computer readable storage media executable by one or more processors to obtain the multi-dimensional image data from the one or more image sensors, perform temporal and spatial alignment of the multi-dimensional image data, and generate the map representation of the geo-spatial area of the real world surroundings of the mobile computing device based on the temporally and spatially aligned multi-dimensional image data.

11. The object/scene recognition system of claim 9, wherein the recognition of the complex object is performed using a classifier.

12. The object/scene recognition system of claim 9, wherein the recognition of the larger-scale objects is performed by determining a contextual frame of reference, and using the contextual frame of reference to identify the larger-scale objects.

13. The object/scene recognition system of claim 9, wherein the recognition of the larger-scale objects is performed by executing an invariant multi-dimensional feature detection algorithm on data from at least one of the image sensors.

14. The object/scene recognition system of claim 9, wherein the recognition of the smaller-scale objects is performed by executing a context-free feature-sharing algorithm.

15. The object/scene recognition system of claim 14, wherein the recognition of the smaller-scale objects is performed by obtaining context information and classifying the smaller-scale objects based on the context information.

16. The object/scene recognition system of claim 15, wherein the recognition of the complex objects is performed by executing a contextual bag of objects algorithm.

17. An object/scene recognition method comprising, with one or more mobile computing devices:

recognizing a plurality of visual features based on a map representation of a geo-spatial area of real world surroundings the one or more mobile computing devices generated based on temporal and spatial alignment of multi-dimensional image data obtained by one or more sensors, wherein the multi-dimensional image data includes at least one of two-dimensional image data and three-dimensional image data,

wherein the recognizing of the plurality of visual features includes recognizing of larger-scale objects, recognizing of smaller-scale objects by performing context-free object identification and contextual object identification, and recognizing of a complex object comprising a plurality of the smaller-scale objects.

18. The object/scene recognition method of claim 17, further comprising:

temporally and spatially aligning the multi-dimensional image data; and

generating the map representation of the geo-spatial area of the real world surroundings of the mobile computing device based on the temporally and spatially aligned multi-dimensional image data.

19. The object/scene recognition method of claim 17, wherein the recognizing of the complex object is performed using a classifier.

20. The object/scene recognition method of claim 17, wherein the recognizing of the larger-scale objects is performed by determining a contextual frame of reference and using the contextual frame of reference to identify the larger-scale objects.

21. The object/scene recognition method of claim 17, wherein the recognizing of the larger-scale objects is performed by executing an invariant multi-dimensional feature detection algorithm on data from at least one of the image sensors.

22. The object/scene recognition method of claim 17, wherein the recognizing of the smaller-scale objects is performed by executing a context-free feature-sharing algorithm.

23. The object/scene recognition method of claim 22, wherein the recognizing of the smaller-scale objects is performed by obtaining context information and classifying the smaller-scale objects based on the context information.

24. The object/scene recognition method of claim 23, wherein the recognition of the complex objects is performed by executing a contextual bag of objects algorithm.