COLLABORATIVE MULTI SENSOR SYSTEM FOR SITE EXPLOITATION

• US 20180190014A1
• Filed: 01/03/2017
• Published: 07/05/2018
• Est. Priority Date: 01/03/2017
• Status: Abandoned Application

• Current Assignee: Honeywell International Inc.
• Original Assignee: Honeywell International Inc.
• Application Number:
  US15/397,436
• Publication Number:
  US 20180190014A1
• Time in Patent Office: Days
• CPC Class Codes: G01C 11/04 Interpretation of pictures
• CPC Class Codes: G01C 15/00 Surveying instruments or accessories not provided for in groups G01C1/00 - G01C13/00
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/87 Combinations of systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G01S 7/4808 Evaluating distance, position or velocity data
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 17/10 Constructive solid geometry [CSG] using solid primitives, e.g. cylinders, cubes

Abstract

A system communicate with a plurality of units. Each of the plurality of units includes a two dimensional camera, a three dimensional camera, a multispectral camera, and/or an inertial measurement unit. Each of the plurality of units is associated with a person, a vehicle, or a robot, and each of the units collects data relating to an environment. The system receives the data relating to the environment from the plurality of units, and uses the data from each of the plurality of units to estimate the positions of the units and to track the positions of the units. The system enables the plurality of units to communicate with each other regarding the collection of the data relating to the environment, commingles and analyzes the data from the plurality of units, and uses the commingled and analyzed data to build a three-dimensional map of the environment.

Claims

1. A system comprising:

a computer processor and a computer storage device configured to;

communicate with a plurality of units, each of the plurality of units comprising one or more of a two dimensional camera, a three dimensional camera, a multispectral camera, a sensor suite, and an inertial measurement unit;

wherein each of the plurality of units is associated with a person, a vehicle, or a robot and each of the units is operable to collect data relating to an environment;

receive the data relating to the environment from the plurality of units;

use the data from each of the plurality of units to estimate the positions of the units and to track the positions of the units;

permit the plurality of units to communicate with each other regarding the collection of the data relating to the environment;

commingle and analyze the data from the plurality of units; and

use the commingled and analyzed data to build a three-dimensional map of the environment.

2. The system of claim 1, comprising using the commingled and analyzed data to create a virtual environment.

3. The system of claim 1, wherein the communication regarding the collection of the data among the plurality of units comprises sharing three dimensional coordinate field of view boundary parameters of the environment captured by the three dimensional cameras and an approximate physical location, orientation, and directional heading of each of the persons, vehicles, or robots.

4. The system of claim 1, wherein the building of the three dimensional map comprises:

determining one or more views of the environment that are missing; and

providing instructions to the persons, vehicles, or robots such that data relating to the one or more missing views are captured.

5. The system of claim 4, wherein the determining the one or more missing views comprises an iterative and additive process and the capturing of the one or more missing views comprises model predictive control.

6. The system of claim 1, wherein the data from the two dimensional camera and sensor suite comprise two dimensional locations traversed by the person, vehicle, or robot, the data from the three dimensional camera comprise three dimensional point cloud data in world coordinates, and the data from the inertia measuring unit comprise six degrees of freedom (6DOF) tracking data.

7. The system of claim 6, wherein the commingling, analyzing, and building comprise:

performing re-localization of three dimensional landmarks in the environment;

using registered landmarks from two or more of the two dimensional camera, the three dimensional camera, and the multispectral camera in the extended Kalman filter to minimize mapping errors;

using a random sample consensus (RANSAC) process in the extended Kalman filter to identify inliers from visual feature correspondences between two or more of the two dimensional camera, the three dimensional camera, and the multispectral camera;

using camera tracking; and

using structure from motion (SfM) to estimate three dimensional coordinates for the landmarks.

8. The system of claim 1, wherein the computer processor is configured to:

create a three dimensional mesh from the three dimensional points;

map multi-spectral data for each region in the mesh; and

for every scene, use multi-spectral image boundary and field of view boundary parameters captured by the three dimensional cameras to perform a coarse registration.

9. The system of claim 1, wherein the sensor suite comprises one or more of a Bluetooth wireless low energy (BLE) device, an ultrasonic sensor, a proximity sensor, a radio frequency identification (RFID) device, an infrared radiation device, and an ultra-wide band (UWB) device.

10. A method comprising:

communicating with a plurality of units, each of the plurality of units comprising one or more of a two dimensional camera, a three dimensional camera, a multispectral camera, a sensor suite, and an inertial measurement unit;

wherein each of the plurality of units is associated with a person, a vehicle, or a robot and each of the units is operable to collect data relating to an environment;

receiving the data relating to the environment from the plurality of units;

using the data from each of the plurality of units to estimate the positions of the units and to track the positions of the units;

permitting the plurality of units to communicate with each other regarding the collection of the data relating to the environment;

commingling and analyzing the data from the plurality of units; and

using the commingled and analyzed data to build a three-dimensional map of the environment.

11. The method of claim 10, comprising using the commingled and analyzed data to create a virtual environment.

12. The method of claim 10, wherein the communication regarding the collection of the data among the plurality of units comprises sharing three dimensional coordinate field of view boundary parameters of the environment captured by the three dimensional cameras and an approximate physical location, orientation, and directional heading of each of the persons, vehicles, or robots.

13. The method of claim 10, wherein the building of the three dimensional map comprises:

determining one or more views of the environment that are missing; and

providing instructions to the persons, vehicles, or robots such that data relating to the one or more missing views are captured.

14. The method of claim 13, wherein the determining the one or more missing views comprises an iterative and additive process and the capturing of the one or more missing views comprises model predictive control.

15. The method of claim 10, wherein the data from the two dimensional camera and sensor suite comprise two dimensional locations traversed by the person, vehicle, or robot, the data from the three dimensional camera comprise three dimensional point cloud data in world coordinates, and the data from the inertia measuring unit comprise six degrees of freedom (6DOF) tracking data.

16. The method of claim 15, wherein the commingling, analyzing, and building comprise:

performing re-localization of three dimensional landmarks in the environment;

using registered landmarks from two or more of the two dimensional camera, the three dimensional camera, and the multispectral camera in the extended Kalman filter to minimize mapping errors;

using a random sample consensus (RANSAC) process in the extended Kalman filter to identify inliers from visual feature correspondences between two or more of the two dimensional camera, the three dimensional camera, and the multispectral camera;

using camera tracking; and

using structure from motion (SfM) to estimate three dimensional coordinates for the landmarks.

17. The method of claim 10, comprising:

creating a three dimensional mesh from the three dimensional points;

mapping multi-spectral data for each region in the mesh; and

for every scene, using multi-spectral image boundary and field of view boundary parameters captured by the three dimensional cameras to perform a coarse registration.

18. The method of claim 10, wherein the sensor suite comprises one or more of a Bluetooth wireless low energy (BLE) device, an ultrasonic sensor, a proximity sensor, a radio frequency identification (RFID) device, an infrared radiation device, and an ultra-wide band (UWB) device.

19. A computer readable medium comprising instructions that when executed by a processor executes a process comprising:

communicating with a plurality of units, each of the plurality of units comprising one or more of a two dimensional camera, a three dimensional camera, a multispectral camera, a sensor suite, and an inertial measurement unit;

wherein each of the plurality of units is associated with a person, a vehicle, or a robot and each of the units is operable to collect data relating to an environment;

receiving the data relating to the environment from the plurality of units;

using the data from each of the plurality of units to estimate the positions of the units and to track the positions of the units;

permitting the plurality of units to communicate with each other regarding the collection of the data relating to the environment;

commingling and analyzing the data from the plurality of units; and

using the commingled and analyzed data to build a three-dimensional map of the environment.

20. The computer readable medium of claim 19, wherein the commingling, analyzing, and building comprise:

performing re-localization of three dimensional landmarks in the environment;

using registered landmarks from two or more of the two dimensional camera, the three dimensional camera, and the multispectral camera in the extended Kalman filter to minimize mapping errors;

using a random sample consensus (RANSAC) process in the extended Kalman filter to identify inliers from visual feature correspondences between two or more of the two dimensional camera, the three dimensional camera, and the multispectral camera;

using camera tracking; and

using structure from motion (SfM) to estimate three dimensional coordinates for the landmarks.