SINGLE IMAGE POSE ESTIMATION OF IMAGE CAPTURE DEVICES

• US 20140212027A1
• Filed: 05/06/2013
• Published: 07/31/2014
• Est. Priority Date: 05/04/2012
• Status: Active Grant

• Current Assignee: Hilti Aktiengesellschaft (Hilti Martin Family Trust)
• Original Assignee: Indoor Reality, Inc.
• Inventors: Zakhor, Avideh, Hallquist, Aaron
• Granted Patent: US 9,098,229 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G06F 3/1415 with means for detecting differences between the image stored in the host and the images displayed on the displays
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]

Abstract

Methods for image based localization using an electronic computing device are presented, the methods including: capturing a local image with an image capture device; associating metadata with the local image; causing the electronic computing device to receive the local image; causing the electronic computing device to match the local image with a database image, where the database image is three-dimensional (3D); and calculating a pose of the image capture device based on a pose of the database image and metadata associated with the local image. In some embodiments, the metadata includes at least pitch and roll data corresponding with the image capture device at a time of image capture.

Claims

1. A method for image based localization using an electronic computing device, the method comprising:

capturing a local image with an image capture device;

associating metadata with the local image;

causing the electronic computing device to receive the local image;

causing the electronic computing device to match the local image with at least one database image, wherein the at least one database image is three-dimensional (3D); and

calculating a pose of the image capture device based on a pose of the at least one database image and metadata associated with the local image.

2. The method of claim 1, wherein the metadata includes at least pitch and roll data corresponding with the image capture device at a time of image capture.

3. The method of claim 2, wherein the calculating the pose comprises:

extracting the pitch and roll data from the metadata;

extracting yaw data from the metadata;

matching features between the local image and the database image; and

estimating a homography matrix to solve for translation of the image capture device with respect to the database image.

4. The method of claim 3, wherein the matching features comprises:

extracting high resolution scale invariant feature transform (SIFT) features from the local image and the database image;

performing a nearest neighbor search to establish any correspondence between the extracted SIFT features of the local image and the database image;

excluding any corresponding database image SIFT features that lack depth data; and

excluding ground plane SIFT features of the database image.

5. The method of claim 3 further comprising:

if the yaw data from metadata of the local image captured is noisy,

extracting a first plurality of vanishing points corresponding with the local image;

extracting a second plurality of vanishing points corresponding with the database image;

finding a best alignment pair between the vanishing points;

calculating a corrected yaw of the image capture device; and

calculating a normal direction of a vertical plane associated with each vanishing point in the local image that was matched to the vanishing point in the database image.

6. The method of claim 3, wherein estimating the homography matrix comprises:

recovering a 3D translation vector, wherein the 3D translation vector includes direction and magnitude., between the local image and the matched database image; and

recovering a perpendicular distance between a center of the image capture device and the vertical plane in the local image for which the first plurality of vanishing points is extracted.

7. The method of claim 1, wherein the image capture device is selected from the group consisting of:

a mobile communication device, a digital camera, and a digital camcorder.

8. The method of claim 1, wherein the image capture device includes a sensor selected from the group consisting of:

a gyroscope, an accelerometer, a compass, and a magnetometer.

12. The method of claim 1 wherein the matching features comprises:

extracting high resolution SIFT features from the local image and the database image;

performing a nearest neighbor search to establish any correspondence between the extracted SIFT features of the local image and the database image;

excluding any corresponding database images SIFT features that lack depth data; and

excluding ground plane SIFT features of the database image when the database image is a 3D image.

9. A method for image based localization using an electronic computing device, the method comprising:

causing the electronic computing device to receive a local image captured from an image capture device;

causing the electronic computing device to receive metadata associated with the local image;

causing the electronic computing device to match the local image with at least one database image, wherein the at least one database image is 3D; and

calculating a pose of the image capture device based on a pose of the at least one database image and metadata associated with the local image.

10. The method of claim 9, wherein the metadata includes at least pitch and roll data corresponding with the image capture device at a time of image capture.

11. The method of claim 10, wherein the calculating the pose comprises:

extracting the pitch and roll data from the metadata;

extracting yaw data from the metadata;

matching features between the local image and the database image; and

estimating a homography matrix to solve for translation of the image capture device with respect to the database image.

13. The method of claim 11 further comprising:

if the yaw data from the metadata of the local image captured is noisy,

extracting a first plurality of vanishing points corresponding with the local image;

extracting a second plurality of vanishing points corresponding with the database image;

finding a best alignment pair between the vanishing points;

calculating a corrected yaw of the image capture device; and

calculating a normal direction of a vertical plane associated with each vanishing point in the local image that was matched to the vanishing point in the database image.

14. The method of claim 13, wherein estimating the homography matrix comprises:

recovering a 3D translation vector, wherein the 3D translation vector includes direction and magnitude between the local image and the matched database image; and

recovering a perpendicular distance between a center of the image capture device and the vertical plane in the local image for which the first plurality of vanishing points is extracted.

15. The method of claim 9, wherein the image capture device includes a sensor selected from the group consisting of:

a gyroscope, an accelerometer, a compass, and a magnetometer.

16. A computing device program product for image based localization using a computing device, the computing device program product comprising:

a non-transitory computer readable medium;

first programmatic instructions for receiving a local image captured from an image capture device;

second programmatic instructions for receiving metadata associated with the local image;

third programmatic instructions for matching the local image with at least one database image, wherein the at least one database image is 3D; and

fourth programmatic instructions for calculating a pose of the image capture device based on a pose of the at least one database image and metadata associated with the local image.

17. The computing device program product of claim 16, wherein the metadata includes at least pitch and roll data corresponding with the image capture device at a time of image capture.

18. The computing device program product of claim 17, wherein the fourth programmatic instructions for calculating the pose comprises:

fifth programmatic instructions for extracting the pitch and roll data from the metadata sixth programmatic instructions for extracting yaw data from the metadata;

seventh programmatic instructions for matching features between the local image and the database image; and

eighth programmatic instructions for estimating a homography matrix to solve for translation of the image capture device with respect to the image database.

19. The computing device program product of claim 18, wherein the matching features comprises:

ninth programmatic instructions for extracting high resolution SIFT features from the local image and the database image;

tenth programmatic instructions for performing a nearest neighbor search to establish any correspondence between the extracted SIFT features of the local image and the database image;

eleventh programmatic instructions for excluding any corresponding database image SIFT features that lack depth data; and

twelfth programmatic instructions for excluding ground plane SIFT features of the database image when the database image is a 3D image.

20. The computing device program product of claim 18 further comprising:

if the yaw data from the metadata of the local image captured is noisy,

thirteenth programmatic instructions for extracting a first plurality of vanishing points corresponding with the local image;

fourteenth programmatic instructions for extracting a second plurality of vanishing points corresponding with the database image;

fifteenth programmatic instructions for finding a best alignment pair between the vanishing points;

sixteenth programmatic instructions for calculating a corrected yaw of the image capture device; and

seventeenth programmatic instructions for calculating a normal direction of a vertical plane associated with each vanishing point in the local image that was matched to the vanishing point in the database image.