3D Feature Descriptors with Camera Pose Information

• US 20150071524A1
• Filed: 09/30/2013
• Published: 03/12/2015
• Est. Priority Date: 09/11/2013
• Status: Active Grant

• Current Assignee: Google Technology Holdings LLC (Alphabet Inc.)
• Original Assignee: Motorola Mobility LLC (Lenovo Group Ltd.)
• Inventors: Lee, Johnny C.
• Granted Patent: US 9,646,384 B2
• Time in Patent Office: Days
• US Class Current: 382/154
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects

Abstract

A method includes determining a first two-dimensional (2D) feature descriptor from a first image captured by an imaging camera in a first pose at a time of capture of the first image, the first pose including a first observation direction of the imaging camera. The method further includes storing, at an electronic device, a 3D feature descriptor including the first 2D feature descriptor and a representation of the first pose of the imaging camera. The method additionally includes determining a second 2D feature descriptor from a second image captured by the imaging camera in a second pose at a time of capture of the second image, the second pose including a second observation direction of the imaging camera. The method also includes storing the 3D feature descriptor with the second 2D feature descriptor and a representation of the second pose of the imaging camera.

Claims

1. A method comprising:

determining a first two-dimensional (2D) feature descriptor from a first image captured by an imaging camera in a first pose at a time of capture of the first image, the first pose including a first observation direction of the imaging camera; and

storing, at an electronic device, a 3D feature descriptor including the first 2D feature descriptor and a representation of the first pose of the imaging camera.

2. The method of claim 1, further comprising:

determining a second 2D feature descriptor from a second image captured by the imaging camera in a second pose at a time of capture of the second image, the second pose including a second observation direction of the imaging camera; and

wherein storing the 3D feature descriptor includes storing the 3D feature descriptor with the second 2D feature descriptor and a representation of the second pose of the imaging camera.

3. The method of claim 1, wherein:

the representation of the first pose further includes a representation of an orientation of the imaging camera at the time of capture of the first image.

4. The method of claim 3, wherein:

the representation of the orientation of the imaging camera includes a representation of at least one of;

yaw angle of the imaging camera;

a pitch angle of the imaging camera; and

a roll angle of the imaging camera.

5. The method of claim 4, further comprising:

determining a reference orientation based on sensor output from at least one of;

an accelerometer;

a gyroscope;

a digital compass; and

a magnetometer; and

wherein at least one of the yaw angle, the pitch angle, and the roll angle is relative to the reference orientation.

6. The method of claim 1, wherein storing the 3D feature descriptor comprises:

indexing the first 2D feature descriptor in the 3D feature descriptor based on the first observation direction.

7. The method of claim 1, wherein:

the representation of the first pose further includes a representation of a distance of the electronic device from an object represented by the first 2D feature descriptor.

8. The method of claim 7, wherein storing the 3D feature descriptor comprises:

indexing the first 2D feature descriptor in the 3D feature descriptor based on the first observation direction and based on the distance.

9. The method of claim 1, wherein:

the representation of the first pose further includes at least one of;

a representation of the time of capture of the first image;

a representation of a weather condition at the time of capture of the first image; and

a representation of an ambient light condition at the time of capture of the first image.

10. The method of claim 1, further comprising:

normalizing the first 2D feature descriptor based on a specified reference distance.

11. An electronic device comprising:

an imaging camera to capture a first image;

a set of one or more sensors to determine a first pose of the imaging camera at a time of capture of the first image, the first pose including a first observation direction of the imaging camera; and

a processor to determine and store a three-dimensional (3D) feature descriptor, the 3D feature descriptor including a first two-dimensional (2D) feature extracted from the first image and a representation of the first pose of the imaging camera.

12. The electronic device of claim 11, wherein:

the imaging camera further is to capture a second image;

the set of one or more sensors further is to determine a second pose of the imaging camera at a time of capture of the second image, the second pose including a second observation direction of the imaging camera; and

the 3D feature descriptor further includes a second 2D feature descriptor extracted from the second image and a representation of the second pose of the imaging camera.

13. The electronic device of claim 11, wherein:

the representation of the first pose further includes a representation of an orientation of the imaging camera at the time of capture of the first image.

14. The electronic device of claim 11, wherein:

the representation of the first pose further includes at least one of;

a representation of the time of capture of the first image;

a representation of a weather condition at the time of capture of the first image; and

a representation of an ambient light condition at the time of capture of the first image.

15. The electronic device of claim 11, wherein:

the representation of the first pose further includes a representation of a distance of the electronic device from an object represented by the first 2D feature descriptor.

16. The electronic device of claim 15, further comprising:

a processor to index the first 2D feature descriptor in the 3D feature descriptor based on the first observation direction and based on the distance.

17. The electronic device of claim 11, further comprising:

a processor to index the first 2D feature descriptor in the 3D feature descriptor based on the first observation direction.

18. The electronic device of claim 11, further comprising:

a processor to normalize the first 2D feature descriptor based on a specified reference distance.

19. A method comprising:

for an image captured by an imaging camera of an electronic device;

determining, at the electronic device, a corresponding set of two-dimensional (2D) feature descriptors from the image; and

determining, at the electronic device, an observation direction of the imaging camera at a time of capture of the image; and

determining, at the electronic device, a set of three-dimensional (3D) feature descriptors from the set of 2D feature descriptors and corresponding observation direction, each 3D feature descriptor including at least one 2D feature descriptor and a representation of the observation direction associated with the image represented by the 2D feature descriptor; and

verifying, at the electronic device, an estimated pose of the electronic device based on the set of 3D feature descriptors.

20. The method of claim 19, wherein verifying the estimated pose comprises:

capturing an image using the imaging camera while in the estimated pose;

extracting a set of observed 2D features from the captured image;

identifying 3D feature descriptors associated with one or more objects that are expected to be visible from the estimated pose; and

verifying the estimated pose based on matches between observed 2D features of the set and corresponding 2D feature descriptors of the identified 3D feature descriptors.