Methods and apparatus for three-dimensional localization and mapping

• US 8,510,039 B1
• Filed: 10/05/2010
• Issued: 08/13/2013
• Est. Priority Date: 10/05/2010
• Status: Active Grant

• Current Assignee: The Boeing Co.
• Original Assignee: The Boeing Co.
• Inventors: Troy, James J., Erignac, Charles A., Murray, Paul
• Primary Examiner(s): Tarcza, Thomas
• Assistant Examiner(s): ALHARBI, ADAM MOHAMED
• Application Number:
  US12/898,412
• Time in Patent Office: 1,043 Days
• Field of Search: None
• US Class Current: 701/436
• CPC Class Codes: G01C 21/1656 with passive imaging devices, e.g. cameras
• CPC Class Codes: G05D 1/0251 extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593
• CPC Class Codes: G05D 1/027 comprising intertial navigation means, e.g. azimuth detector inertial navigation G01C21/16; inertial navigation combined with non-inertial navigation instruments G01C21/165
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30

Abstract

A system configured to enable three-dimensional localization and mapping is provided. The system includes a mobile device and a computing device. The mobile device includes an inertial measurement unit and a three-dimensional image capture device. The computing device includes a processor programmed to receive a first set of inertial measurement information from the inertial measurement unit, determine a first current position and orientation of the mobile device based on a defined position and orientation of the mobile device and the first set of inertial measurement information, receive three-dimensional image data of the environment from the three-dimensional image capture device, determine a second current position and orientation of the mobile device based on the received three-dimensional image data and the first current position and orientation of the mobile device, and generate a three-dimensional representation of an environment with respect to the second current position and orientation of the mobile device.

Claims

1. A system configured to enable three-dimensional localization and mapping;

the system comprising;

a mobile device comprising;

an inertial measurement unit; and

a three-dimensional image capture device; and

a computing device comprising a processor programmed to;

receive a first set of inertial measurement information from the inertial measurement unit;

determine a first current position and orientation of the mobile device based on a defined position and orientation of the mobile device and the first set of inertial measurement information;

receive three-dimensional image data of the environment from the three-dimensional image capture device;

determine a second current position and orientation of the mobile device based on the received three-dimensional image data and the first current position and orientation of the mobile device; and

generate a three-dimensional representation of an environment with respect to the second current position and orientation of the mobile device.

2. A system in accordance with claim 1, further comprising a display device.

3. A system in accordance with claim 2, wherein the processor is further programmed to present the three-dimensional representation of the environment on the display device.

4. A system in accordance with claim 1, wherein the processor is further programmed to receive accelerometer data and gyro data from the inertial measurement unit;

and

based on the accelerometer data and the gyro data, determine a position and an orientation of the mobile device.

5. A system in accordance with claim 1, wherein the three-dimensional image capture device is a depth camera.

6. A system in accordance with claim 5, wherein the three-dimensional image data comprises a distance from the three-dimensional image capture device to a point in the environment.

7. A system in accordance with claim 6, wherein the three-dimensional image data comprises a collection of three-dimensional points.

8. A system in accordance with claim 7, further comprising generate a shape of an object in the environment based on the collection of three-dimensional points.

9. A system in accordance with claim 8, further comprising generating a three-dimensional representation of the object based on the generated shape.

10. A method for enabling three-dimensional localization and mapping via a mobile device comprising an inertial measurement unit and a three-dimensional image capture device, the method comprising:

receiving a first set of inertial measurement information from the inertial measurement unit;

determining a first current position and orientation of the mobile device based on a defined position and orientation of the mobile device and the first set of inertial measurement information;

receiving three-dimensional image data of the environment from the three-dimensional image capture device;

determining a second current position and orientation of the mobile device based on the received three-dimensional image data and the first current position and orientation of the mobile device; and

generating a three-dimensional representation of an environment with respect to the second current position and orientation of the mobile device.

11. A method in accordance with claim 10, further comprising:

receiving accelerometer data and gyro data from the inertial measurement unit; and

based on the accelerometer data and the gyro data, determining a position and orientation of the mobile device.

12. A method in accordance with claim 10, wherein the three-dimensional image data comprises a distance from the three-dimensional image capture device to a point in the environment.

13. A method in accordance with claim 12, wherein the three-dimensional image data comprises a collection of three-dimensional points.

14. A method in accordance with claim 13, further comprising generating a shape of an object in the environment based on the collection of three-dimensional points.

15. One or more computer-readable media having computer-executable component, the components comprising:

an interface component that when executed by at least one processor causes the at least one processor to receive a first set of inertial measurement information from an inertial measurement unit, and receive three-dimensional image data of an environment from a three-dimensional image capture device;

a localization component that when executed by at least one processor causes the at least one processor to determine a first current position and orientation of the device in the environment based on a defined position and orientation of the device and the first set of inertial measurement information, determine a second current position and orientation of the device based on the received three-dimensional image data and the first current position and orientation of the mobile device; and

a mapping component that when executed by at least one processor causes the at least one processor to generate a three-dimensional representation of the environment with respect to the second current position and orientation of the device.

16. A computer-readable medium in accordance with claim 15,

wherein the interface component further causes the processor to receive accelerometer data and gyro data from the inertial measurement unit, and

wherein the localization component further causes the processor to determine a position and an orientation of the device based on the accelerometer data and the gyro data.

17. A computer-readable medium in accordance with claim 15, wherein the three-dimensional image data comprises a distance from the three-dimensional image capture device to a point in the environment.

18. A computer-readable medium in accordance with claim 17, wherein the three-dimensional image data comprises a collection of three-dimensional points.

19. A computer-readable medium in accordance with claim 18, wherein the mapping component further causes the processor to generate a shape of an object in the environment based on the collection of three-dimensional points.

20. A computer-readable medium in accordance with claim 19, wherein the mapping component further causes the processor to generate a three-dimensional representation of the object based on the generated shape of the object.