Real time machine vision and point-cloud analysis for remote sensing and vehicle control

• US 10,549,768 B2
• Filed: 10/23/2017
• Issued: 02/04/2020
• Est. Priority Date: 11/27/2013
• Status: Active Grant

• Current Assignee: Condor Acquisition SUB II Incorporated (Luminar Technologies, Inc.)
• Original Assignee: Solfice Research Inc. (Luminar Technologies, Inc.)
• Inventors: Puttagunta, Shanmukha Sravan, Chraim, Fabien, Gupta, Anuj, Harvey, Scott, Creadore, Jason, Mills, Graham
• Primary Examiner(s): Dunn, Alex C
• Application Number:
  US15/790,968
• Publication Number:
  US 20180057030A1
• Time in Patent Office: 834 Days
• CPC Class Codes: B61L 2205/04 Satellite based navigation systems, e.g. global positioning system [GPS]
• CPC Class Codes: B61L 23/041 Obstacle detection
• CPC Class Codes: B61L 23/34 for indicating the distance between vehicles or trains by the transmission of signals therebetween
• CPC Class Codes: B61L 25/025 Absolute localisation, e.g. providing geodetic coordinates
• CPC Class Codes: B61L 25/04 Indicating or recording train identities
• CPC Class Codes: B61L 27/04 Automatic systems, e.g. controlled by train; Change-over to manual control

Abstract

Methods and apparatus for real time machine vision and point-cloud data analysis are provided, for remote sensing and vehicle control. Point cloud data can be analyzed via scalable, centralized, cloud computing systems for extraction of asset information and generation of semantic maps. Machine learning components can optimize data analysis mechanisms to improve asset and feature extraction from sensor data. Optimized data analysis mechanisms can be downloaded to vehicles for use in on-board systems analyzing vehicle sensor data. Semantic map data can be used locally in vehicles, along with onboard sensors, to derive precise vehicle localization and provide input to vehicle to control systems.

Claims

1. A method of localizing a vehicle by a computing platform installed within the vehicle, the method comprising:

determining an initial geographical position of the vehicle based on output of a global positioning system receiver installed in the vehicle;

querying a local map cache stored within the vehicle, the local map cache storing a local map of assets comprising, for each asset;

a location of the asset, properties associated with the asset, and one or more relationships relative to other assets;

the local map cache queried to identify assets previously mapped near the initial geographical position;

extracting one or more features, and a location relative to the vehicle, of assets observed within the vicinity of the vehicle using local environment sensors installed in the vehicle; and

determining a second vehicle position that is refined relative to the initial geographical position of the vehicle, by comparing the extracted features of said assets observed within the vicinity of the vehicle with asset information retrieved from the local map cache.

2. The method of claim 1, further comprising, prior to the step of querying a local map cache:

downloading local map data to the vehicle during vehicle operation from a remote map store, via a wireless communication device.

10. The method of claim 1, further comprising:

transmitting said second vehicle position to a vehicle decision engine.

3. A method of localizing a vehicle by a computing platform installed within the vehicle, the method comprising:

determining an initial geographical position of the vehicle based on output of a global positioning system receiver installed in the vehicle;

querying a local map cache stored within the vehicle, the local map cache storing a local map of assets comprising, for each asset;

a location of the asset, properties associated with the asset, and one or more relationships relative to other assets;

the local map cache queried to identify assets previously mapped near the initial geographical position;

extracting one or more features, and a location relative to the vehicle, of assets observed within the vicinity of the vehicle using local environment sensors installed in the vehicle;

determining a second vehicle position that is refined relative to the initial geographical position of the vehicle, by comparing the extracted features of said assets observed within the vicinity of the vehicle with asset information retrieved from the local map cache; and

identifying one or more differences between characteristics of an asset obtained in the step of querying a local map database, and characteristics of said asset determined in the step of extracting, via observation by said local environment sensors installed in the vehicle.

4. The method of claim 3, in which the step of identifying one or more differences comprises the substeps of:

identifying a missing asset observed within the vicinity of the vehicle using local environment sensors but not present within said local map cache; and

transmitting, to the remote map store, observed features and a location of the missing asset.

5. The method of claim 3, in which the step of identifying one or more differences comprises the substeps of:

identifying a missing asset present within said local map cache but not observed within the vicinity of the vehicle using local environment sensors; and

transmitting, to the remote map store, identification of the missing asset.

6. The method of claim 3, wherein said differences associated with an asset are indicative of damage or tampering.

7. The method of claim 6, further comprising transmitting, to the remote map store, said differences associated with an asset.

8. A method of localizing a vehicle by a computing platform installed within the vehicle, the method comprising:

determining an initial geographical position of the vehicle based on output of a global positioning system receiver installed in the vehicle;

querying a local map cache stored within the vehicle, the local map cache storing a local map of assets comprising, for each asset;

a location of the asset, properties associated with the asset, and one or more relationships relative to other assets;

the local map cache queried to identify assets previously mapped near the initial geographical position;

extracting one or more features, and a location relative to the vehicle, of assets observed within the vicinity of the vehicle using local environment sensors installed in the vehicle;

determining a second vehicle position that is refined relative to the initial geographical position of the vehicle, by comparing the extracted features of said assets observed within the vicinity of the vehicle with asset information retrieved from the local map cache;

receiving, from a central control server, a request to audit map data; and

evaluating differences between asset information within said local map cache and asset information observed by said local environment sensors.

9. The method of claim 8, in which the step of evaluating differences between asset information within said local map cache and asset information observed by said local environment sensors comprises confirming a discrepancy between asset information previously observed by another vehicle and asset information within a remote map store.