Methods for localization using geotagged photographs and three-dimensional visualization

• US 9,892,558 B2
• Filed: 02/19/2016
• Issued: 02/13/2018
• Est. Priority Date: 02/19/2016
• Status: Active Grant

• Current Assignee: The Boeing Co.
• Original Assignee: The Boeing Co.
• Inventors: Troy, James J., Karakusevic, Vladimir, Esposito, Christopher D.
• Primary Examiner(s): CRADDOCK, ROBERT J
• Application Number:
  US15/047,655
• Publication Number:
  US 20170243399A1
• Time in Patent Office: 725 Days
• CPC Class Codes: B64F 5/40 Maintaining or repairing aircraft
• CPC Class Codes: G01S 19/13 Receivers
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06Q 10/20 Administration of product repair or maintenance
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 2219/2004 Aligning objects, relative positioning of parts
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80

Abstract

Methods for identifying parts of a target object (e.g., an airplane) using geotagged photographs captured on site by a hand-held imaging device. The geotagged photographs contain GPS location data and camera setting information. The embedded image metadata from two or more photographs is used to estimate the location (i.e., position and orientation) of the imaging device relative to the target object, which location is defined in the coordinate system of the target object. Once the coordinates of the area of interest on the target object are known, the part number and other information associated with the part can be determined when the imaging device viewpoint information is provided to a three-dimensional visualization environment that has access to three-dimensional models of the target object.

Claims

1. A method for identifying a part of a target object, comprising:

(a) acquiring reference location image data, first geolocation data and first heading angle data using an imaging device directed toward the target object and located at a known reference location relative to a coordinate system of the target object;

(b) acquiring feature location image data, second geolocation data and second heading angle data using an imaging device directed toward an area on the target object and located at an unknown feature location relative to the coordinate system of the target object;

(c) computing an offset in Cartesian coordinates between the feature location and the reference location, wherein the offset is a function of coordinates in said first and second geolocation data and heading angles in said first and second heading angle data;

(d) retrieving three-dimensional model data of the target object from a three-dimensional model database;

(e) displaying a three-dimensional model image of at least a portion of the target object with a viewpoint that is a function of the offset computed in step (c) and the camera field-of-view angle, said three-dimensional model image representing said three-dimensional model data retrieved from the three-dimensional model database;

(f) displaying a feature location image of the same portion of the target object as an overlay or background to said three-dimensional model image, said feature location image representing at least portions of said feature location image data;

(g) selecting a point in the displayed feature location image;

(h) retrieving additional three-dimensional model data from the three-dimensional model database in response to said selecting; and

(i) displaying symbology representing the retrieved additional three-dimensional model data.

2. The method as recited in claim 1, wherein the additional three-dimensional model data comprises part identification data that identifies a part of the target object associated with an area of the three-dimensional model image containing the selected point.

3. The method as recited in claim 1, wherein the additional three-dimensional model data comprises coordinates of the selected point defined in the coordinate system of the target object.

4. The method as recited in claim 1, wherein step (c) comprises computing an offset transformation matrix that represents a location offset between the imaging device at the reference location and the imaging device at the feature location, wherein the offset transformation matrix is a function of at least coordinates in said first and second geolocation data and heading angles in said first and second heading angle data, and the viewpoint is a function of the offset transformation matrix.

5. The method as recited in claim 1, further comprising adjusting a position of the three-dimensional model image to align the three-dimensional model image with the feature location image on the display screen.

6. The method as recited in claim 1, wherein said displayed feature location image comprises an outline extracted from said feature location image data using edge detection.

7. The method as recited in claim 1, wherein the target object is an airplane.

8. The method as recited in claim 1, wherein a position of the reference location is collocated with an origin of the coordinate system of the target object.

9. The method as recited in claim 1, wherein steps (c)-(f), (h) and (i) are performed by the imaging device.

10. The method as recited in claim 1, further comprising sending said reference and feature location image data, said first and second geolocation data, and said first and second heading angle data from the imaging device to a computer system, wherein steps (c)-(f), (h) and (i) are performed by the computer system.

11. A method for identifying a part of a target object, comprising:

(a) acquiring reference location image data, first geolocation data and first heading angle data using an imaging device directed toward the target object and located at a known reference location relative to a coordinate system of the target object;

(b) constructing a reference location data file containing at least said reference location image data, said first geolocation data and said first heading angle data;

(c) acquiring feature location image data, second geolocation data and second heading angle data using an imaging device directed toward an area on the target object and located at an unknown feature location relative to the coordinate system of the target object;

(d) constructing a feature location data file containing at least said feature location image data, said second geolocation data and said second heading angle data;

(e) sending the reference and feature location data files from the imaging device to a computer system;

(f) computing an offset between the feature location and the reference location, wherein the offset is a function of coordinates in said first and second geolocation data and heading angles in said first and second heading angle data;

(g) retrieving three-dimensional model data of the target object from a three-dimensional model database;

(h) displaying a three-dimensional model image of at least a portion of the target object with a viewpoint that is a function of the offset computed in step (f) and the field-of-view angle of the imaging device, said three-dimensional model image representing said three-dimensional model data retrieved from the three-dimensional model database; and

(i) displaying a feature location image of the same portion of the target object as an overlay or background to said three-dimensional model image, said feature location image representing at least portions of said feature location image data,

wherein steps (f) through (i) are performed by the computer system.

12. The method as recited in claim 11, further comprising:

(g) selecting a point in said displayed feature location image;

(h) retrieving additional three-dimensional model data from the three-dimensional model database in response to said selecting; and

(i) displaying symbology representing the retrieved additional three-dimensional model data.

13. The method as recited in claim 12, wherein the additional three-dimensional model data comprises part identification data that identifies a part of the target object associated with an area of the three-dimensional model image containing the selected point.

14. The method as recited in claim 12, wherein the additional three-dimensional model data comprises coordinates of the selected point in the coordinate system of the target object.

15. The method as recited in claim 12, wherein step (f) comprises computing an offset transformation matrix that represents a location offset between the imaging device at the reference location and the imaging device at the feature location, wherein the offset transformation matrix is a function of at least coordinates in said first and second geolocation data and heading angles in said first and second heading angle data, and the viewpoint is a function of the offset transformation matrix.

16. The method as recited in claim 11, wherein said feature location image comprises an outline extracted from said feature location image data using edge detection.

17. A system comprising a digital camera, a GPS receiver, a digital compass, a display screen, a non-transitory tangible computer-readable storage medium storing three-dimensional model data of a target object, a three-dimensional visualization software application, and an offset computation software application, and a computer system configured to perform the following operations:

(a) computing an offset between a feature location of said digital camera and a reference location of said digital camera using offset computation software application, wherein the offset is a function of coordinates in geolocation data acquired by said GPS receiver and heading angle data acquired by said digital compass during capture of reference and feature location image data by said digital camera;

(b) retrieving said three-dimensional model data of the target object from said non-transitory tangible computer-readable storage medium;

(c) using said three-dimensional visualization software application to control said display screen to display a three-dimensional model image of at least a portion of the target object with a viewpoint that is a function of the offset computed in operation (a) and the field-of-view angle of the digital camera, said three-dimensional model image representing said three-dimensional model data retrieved from the three-dimensional model database; and

(d) using said three-dimensional visualization software application to control said display screen to display a feature location image of the same portion of the target object as an overlay or background to said three-dimensional model image, said feature location image representing at least portions of said feature location image data captured by said digital camera.

18. The system as recited in claim 17, further comprising a housing supporting said GPS receiver, said digital camera, said digital compass, said display screen, and said computer system.

19. The system as recited in claim 17, wherein said computer system comprises first and second processors, wherein said first processor is connected to receive geolocation data from said GPS receiver, heading angle data from said digital compass, and image data from said digital camera, and said second processor performs operations (a) through (d), the system further comprising a housing supporting said GPS receiver, said digital camera, said digital compass, said display screen, and said first processor, but not said second processor.

20. The system as recited in claim 19, wherein said housing is located in proximity to the target object, while said second processor is located at a distance from said housing.