Real-time system for multi-modal 3D geospatial mapping, object recognition, scene annotation and analytics

• US 9,488,492 B2
• Filed: 12/18/2014
• Issued: 11/08/2016
• Est. Priority Date: 03/18/2014
• Status: Active Grant

• Current Assignee: SRI International, Inc.
• Original Assignee: SRI International, Inc.
• Inventors: Samarasekera, Supun, Hadsell, Raia, Kumar, Rakesh, Sawhney, Harpreet S., Matei, Bogdan C., Villamil, Ryan
• Primary Examiner(s): Tarcza, Thomas
• Assistant Examiner(s): Dunn, Alex C
• Application Number:
  US14/575,472
• Publication Number:
  US 20150268058A1
• Time in Patent Office: 691 Days
• US Class Current: 1/1
• CPC Class Codes: G01C 11/02 Picture taking arrangements specially adapted for photogrammetry or photographic surveying, e.g. controlling overlapping of pictures
• CPC Class Codes: G01C 21/3673 Labelling using text of road map data items, e.g. road names, POI names
• CPC Class Codes: G01C 21/3848 Data obtained from both position sensors and additional sensors
• CPC Class Codes: G01C 21/3852 Data derived from aerial or satellite images
• CPC Class Codes: G01C 21/3867 Geometry of map features, e.g. shape points, polygons or for simplified maps
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06V 10/811 the classifiers operating on different input data, e.g. multi-modal recognition
• CPC Class Codes: G06V 20/176 Urban or other man-made structures
• CPC Class Codes: G08G 5/0004 Transmission of traffic-related information to or from an aircraft airborne radio transmission systems in general H04B7/185; airborne wireless networks H04W84/06
• CPC Class Codes: G08G 5/003 Flight plan management
• CPC Class Codes: G08G 5/0069 specially adapted for an unmanned aircraft
• CPC Class Codes: G08G 5/0073 Surveillance aids scene image recognition G06V20/00

Abstract

A multi-sensor, multi-modal data collection, analysis, recognition, and visualization platform can be embodied in a navigation capable vehicle. The platform provides an automated tool that can integrate multi-modal sensor data including two-dimensional image data, three-dimensional image data, and motion, location, or orientation data, and create a visual representation of the integrated sensor data, in a live operational environment. An illustrative platform architecture incorporates modular domain-specific business analytics “plug ins” to provide real-time annotation of the visual representation with domain-specific markups.

Claims

1. A navigation-capable vehicle, comprising:

one or more processors, and, in communication with the one or more processors;

one or more two-dimensional image sensors;

one or more sensors to determine motion, location, and orientation of the navigation-capable vehicle; and

one or more non-transitory machine accessible storage media comprising instructions to cause the navigation-capable vehicle to;

temporally and spatially align sensor data received from the one or more two-dimensional image sensors, and the one or more motion, location, and orientation sensors;

generate a map representation of a real world environment in a frame of reference of the navigation-capable vehicle based on the temporally and spatially aligned sensor data;

recognize a plurality of visual features in the map representation using one or more computer vision algorithms;

annotate one or more of the visual features in accordance with domain-specific business logic;

wherein the domain-specific business logic is to cause the navigation-capable vehicle to detect a change in a visual feature of a natural resource.

2. A navigation-capable vehicle, comprising:

one or more processors, and, in communication with the one or more processors;

one or more two-dimensional image sensors;

one or more sensors to determine motion, location, and orientation of the navigation-capable vehicle; and

one or more non-transitory machine accessible storage media comprising instructions to cause the navigation-capable vehicle to;

temporally and spatially align sensor data received from the one or more two-dimensional image sensors, and the one or more motion, location, and orientation sensors;

generate a map representation of a real world environment in a frame of reference of the navigation-capable vehicle based on the temporally and spatially aligned sensor data;

recognize a plurality of visual features in the map representation using one or more computer vision algorithms; and

annotate one or more of the visual features in accordance with domain-specific business logic;

wherein the navigation-capable vehicle comprises an unmanned aerial vehicle.

8. The navigation-capable vehicle of claim 2, wherein one of the one or more two-dimensional image sensors generates two-dimensional image sensor data.

9. The navigation-capable vehicle of claim 2, wherein one of the one or more two-dimensional image sensors generates three-dimensional image sensor data.

10. The navigation-capable vehicle of claim 2, further comprising a three-dimensional image sensor.

3. A multi-sensor data collection, analysis, recognition, and visualization platform comprising instructions embodied in one or more non-transitory computer readable storage media and executable by one or more processors to cause a navigation-capable vehicle to:

receive sensor data from a plurality of sensors comprising one or more two-dimensional image sensors, and one or more sensors to determine motion, location, and orientation of the navigation-capable vehicle;

temporally and spatially align the sensor data received from the one or more two-dimensional image sensors, and the one or more motion, location, and orientation sensors;

generate a map representation of the real world surroundings of the navigation-capable vehicle based on the temporally and spatially aligned sensor data;

recognize a plurality of visual features in the map representation by executing one or more computer vision algorithms;

annotate one or more of the visual features in accordance with domain-specific business logic; and

present a visualization of the annotated visual features on the navigation-capable vehicle;

wherein the domain-specific business logic comprises a change detection algorithm to detect one or more domain-specific changes in the visual features over time, and the platform comprises instructions to annotate the visual features to identify the detected domain-specific changes on the visualization.

11. The navigation-capable vehicle of claim 3, wherein one of the one or more two-dimensional image sensors generates two-dimensional image sensor data.

12. The navigation-capable vehicle of claim 3, wherein one of the one or more two-dimensional image sensors generates three-dimensional image sensor data.

13. The navigation-capable vehicle of claim 3, further comprising a three-dimensional image sensor.

4. A multi-sensor data collection, analysis, recognition, and visualization platform comprising instructions embodied in one or more non-transitory computer readable storage media and executable by one or more processors to cause a navigation-capable vehicle to:

receive sensor data from a plurality of sensors comprising one or more two-dimensional image sensors, and one or more sensors to determine motion, location, and orientation of the navigation-capable vehicle;

temporally and spatially align the sensor data received from the one or more two-dimensional sensors, and the one or more motion, location, and orientation sensors;

generate a map representation of the real world surroundings of the navigation-capable vehicle based on the temporally and spatially aligned sensor data;

recognize a plurality of visual features in the map representation by executing one or more computer vision algorithms;

annotate one or more of the visual features in accordance with domain-specific business logic; and

present a visualization of the annotated visual features on the navigation-capable vehicle;

wherein the domain-specific business logic comprises an anomaly detection algorithm to detect one or more domain-specific anomalies in the visual features over time, and the platform comprises instructions to annotate the visual features to identify the detected domain-specific anomalies on the visualization.

14. The navigation-capable vehicle of claim 4, wherein one of the one or more two-dimensional image sensors generates two-dimensional image sensor data.

15. The navigation-capable vehicle of claim 4, wherein one of the one or more two-dimensional image sensors generates three-dimensional image sensor data.

16. The navigation-capable vehicle of claim 4, further comprising a three-dimensional image sensor.

5. A system for multi-sensor data collection, analysis, recognition, and visualization by a navigation-capable vehicle, the system comprising one or more computing devices configured to:

temporally and spatially align data received from one or more two-dimensional sensors and one or more motion, location, and orientation sensors;

generate a map representation of the real world surroundings of the navigation-capable vehicle based on the temporally and spatially aligned sensor data;

recognize a plurality of visual features in the map representation by executing one or more computer vision algorithms;

estimate a navigation path for the navigation-capable vehicle;

annotate one or more of the visual features in accordance with domain-specific business logic;

present a visualization of the annotated visual features on the navigation-capable vehicle; and

tag one or more of the annotated visual features in the visualization in response to user input.

6. The system of claim 5, configured to execute domain-specific anomaly detection logic on the visual features and annotate the visualization based on the execution of the domain-specific anomaly detection logic.

7. The system of claim 5, configured to execute domain-specific change detection logic on the visual features and annotate the visualization based on the execution of the domain-specific change detection logic.