REAL-TIME SYSTEM FOR MULTI-MODAL 3D GEOSPATIAL MAPPING, OBJECT RECOGNITION, SCENE ANNOTATION AND ANALYTICS

• US 20150268058A1
• Filed: 12/18/2014
• Published: 09/24/2015
• Est. Priority Date: 03/18/2014
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Samarasekera, Supun, Hadsell, Raia, Kumar, Rakesh, Sawhney, Harpreet S., Matei, Bogdan C., Villamil, Ryan
• Granted Patent: US 9,488,492 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G01C 11/02 Picture taking arrangements specially adapted for photogrammetry or photographic surveying, e.g. controlling overlapping of pictures
• CPC Class Codes: G01C 21/3673 Labelling using text of road map data items, e.g. road names, POI names
• CPC Class Codes: G01C 21/3848 Data obtained from both position sensors and additional sensors
• CPC Class Codes: G01C 21/3852 Data derived from aerial or satellite images
• CPC Class Codes: G01C 21/3867 Geometry of map features, e.g. shape points, polygons or for simplified maps
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06V 10/811 the classifiers operating on different input data, e.g. multi-modal recognition
• CPC Class Codes: G06V 20/176 Urban or other man-made structures
• CPC Class Codes: G08G 5/0004 Transmission of traffic-related information to or from an aircraft airborne radio transmission systems in general H04B7/185; airborne wireless networks H04W84/06
• CPC Class Codes: G08G 5/003 Flight plan management
• CPC Class Codes: G08G 5/0069 specially adapted for an unmanned aircraft
• CPC Class Codes: G08G 5/0073 Surveillance aids scene image recognition G06V20/00

Abstract

A multi-sensor, multi-modal data collection, analysis, recognition, and visualization platform can be embodied in a navigation capable vehicle. The platform provides an automated tool that can integrate multi-modal sensor data including two-dimensional image data, three-dimensional image data, and motion, location, or orientation data, and create a visual representation of the integrated sensor data, in a live operational environment. An illustrative platform architecture incorporates modular domain-specific business analytics “plug ins” to provide real-time annotation of the visual representation with domain-specific markups.

Claims

1. A navigation-capable vehicle, comprising:

one or more processors, and, in communication with the one or more processors;

a two-dimensional image sensor;

a three-dimensional image sensor;

one or more sensors to determine motion, location, and orientation of the navigation-capable vehicle; and

one or more non-transitory machine accessible storage media comprising instructions to cause the navigation-capable vehicle to;

temporally and spatially align sensor data received from the two-dimensional sensor, the three-dimensional sensor, and the one or more motion, location, and orientation sensors;

generate a map representation of a real world environment in a frame of reference of the navigation-capable vehicle based on the temporally and spatially aligned sensor data;

recognize a plurality of visual features in the map representation using one or more computer vision algorithms; and

annotate one or more of the visual features in accordance with domain-specific business logic.

2. The navigation-capable vehicle of claim 1, comprising instructions to cause the navigation-capable vehicle to present a visualization of the annotated visual features on the navigation-capable vehicle.

3. The navigation-capable vehicle of claim 1, comprising instructions to cause the navigation-capable vehicle to estimate a navigation path for the navigation-capable vehicle.

4. The navigation-capable vehicle of claim 1, comprising instructions to cause the navigation-capable vehicle to estimate a navigation path for the navigation-capable vehicle.

5. The navigation-capable vehicle of claim 1, comprising instructions to cause the navigation-capable vehicle to algorithmically detect changes in the visual features of the map representation over time.

6. The navigation-capable vehicle of claim 5, comprising instructions to cause the navigation-capable vehicle to transmit data indicative of the detected changes to a ground-based computing device in real time.

7. The navigation-capable vehicle of claim 5, wherein the domain-specific business logic is to cause the navigation-capable vehicle to detect a change in a visual feature of a physical component of a ground-based utility service.

8. The navigation-capable vehicle of claim 5, wherein the domain-specific business logic is to cause the navigation-capable vehicle to detect a change in a visual feature of a natural resource.

9. The navigation-capable vehicle of claim 5, wherein the domain-specific business logic is to cause the navigation-capable vehicle to detect a change in a visual feature of a man-made physical structure.

10. The navigation-capable vehicle of claim 1, wherein the navigation-capable vehicle comprises an unmanned aerial vehicle.

11. A multi-sensor data collection, analysis, recognition, and visualization platform comprising instructions embodied in one or more non-transitory computer readable storage media and executable by one or more processors to cause a navigation-capable vehicle to:

receive sensor data from a plurality of sensors comprising a two dimensional image sensor, a three-dimensional image sensor, and one or more sensors to determine motion, location, and orientation of the navigation-capable vehicle;

temporally and spatially align the sensor data received from the two-dimensional sensor, the three-dimensional sensor, and the one or more motion, location, and orientation sensors;

generate a map representation of the real world surroundings of the navigation-capable vehicle based on the temporally and spatially aligned sensor data;

recognize a plurality of visual features in the map representation by executing one or more computer vision algorithms;

annotate one or more of the visual features in accordance with domain-specific business logic; and

present a visualization of the annotated visual features on the navigation-capable vehicle.

12. The platform of claim 11, comprising instructions to cause the navigation-capable vehicle to present the visualization of the annotated visual features on the navigation-capable vehicle in real time.

13. The platform of claim 11, comprising instructions to cause the navigation-capable vehicle to transmit the visualization of the annotated visual features to a ground-based computing device in real time.

14. The platform of claim 11, wherein the domain-specific business logic comprises a change detection algorithm to detect one or more domain-specific changes in the visual features over time, and the platform comprises instructions to annotate the visual features to identify the detected domain-specific changes on the visualization.

15. The platform of claim 11, wherein the domain-specific business logic comprises an anomaly detection algorithm to detect one or more domain-specific anomalies in the visual features over time, and the platform comprises instructions to annotate the visual features to identify the detected domain-specific anomalies on the visualization.

16. A system for multi-sensor data collection, analysis, recognition, and visualization by a navigation-capable vehicle, the system comprising one or more computing devices configured to:

temporally and spatially align data received from a two-dimensional sensor, a three-dimensional sensor, and one or more motion, location, and orientation sensors;

generate a map representation of the real world surroundings of the navigation-capable vehicle based on the temporally and spatially aligned sensor data;

recognize a plurality of visual features in the map representation by executing one or more computer vision algorithms;

estimate a navigation path for the navigation-capable vehicle;

annotate one or more of the visual features in accordance with domain-specific business logic; and

present a visualization of the annotated visual features on the navigation-capable vehicle.

17. The system of claim 16, configured to interactively modify the visualization of the annotated visual features in response to user input.

18. The system of claim 16, configured to tag one or more of the annotated visual features in the visualization in response to user input.

19. The system of claim 16, configured to execute domain-specific anomaly detection logic on the visual features and annotate the visualization based on the execution of the domain-specific anomaly detection logic.

20. The system of claim 16, configured to execute domain-specific change detection logic on the visual features and annotate the visualization based on the execution of the domain-specific change detection logic.