3D feature descriptors with camera pose information

• US 9,646,384 B2
• Filed: 09/30/2013
• Issued: 05/09/2017
• Est. Priority Date: 09/11/2013
• Status: Active Grant

• Current Assignee: Google Technology Holdings LLC (Alphabet Inc.)
• Original Assignee: Google Technology Holdings LLC (Alphabet Inc.)
• Inventors: Lee, Johnny C
• Primary Examiner(s): Li, Ruiping
• Application Number:
  US14/040,918
• Publication Number:
  US 20150071524A1
• Time in Patent Office: 1,317 Days
• Field of Search: 382154, 382103, 382218, 382165, 382285
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 20/647 by matching two-dimensional images to three-dimensional objects

Abstract

A method includes determining a first two-dimensional (2D) feature descriptor from a first image captured by an imaging camera in a first pose at a time of capture of the first image, the first pose including a first observation direction of the imaging camera. The method further includes storing, at an electronic device, a 3D feature descriptor including the first 2D feature descriptor and a representation of the first pose of the imaging camera. The method additionally includes determining a second 2D feature descriptor from a second image captured by the imaging camera in a second pose at a time of capture of the second image, the second pose including a second observation direction of the imaging camera. The method also includes storing the 3D feature descriptor with the second 2D feature descriptor and a representation of the second pose of the imaging camera.

Claims

1. In an electronic device comprising an imaging camera coupled to at least one processor and a datastore, a method comprising:

determining, by the at least one processor, a first two-dimensional (2D) feature descriptor from a first image captured by an imaging camera in a first pose at a time of capture of the first image, the first pose including a first observation direction of the imaging camera; and

storing, at the datastore, a 3D feature descriptor having a plurality of entries, each entry configured to store a corresponding 2D feature descriptor, and wherein a first entry of the plurality of entries stores the first 2D feature descriptor and a representation of the first pose of the imaging camera;

wherein storing the 3D feature descriptor comprises indexing the first 2D feature descriptor in the plurality of entries of the 3D feature descriptor based on the first observation direction; and

wherein the representation of the first pose further includes at least one of;

a representation of a weather condition at the time of capture of the first image; and

a representation of an ambient light condition at the time of capture of the first image.

8. The method of claim 1, further comprising:

determining a second 2D feature descriptor from a second image captured by the imaging camera in a second pose at a time of capture of the second image, the second pose including a second observation direction of the imaging camera; and

wherein storing the 3D feature descriptor includes storing the 3D feature descriptor with a second entry of the plurality of entries storing the second 2D feature descriptor and a representation of the second pose of the imaging camera.

9. The method of claim 1, wherein:

the representation of the first pose further includes a representation of an orientation of the imaging camera at the time of capture of the first image.

10. The method of claim 9, wherein:

the representation of the orientation of the imaging camera includes a representation of at least one of;

yaw angle of the imaging camera;

a pitch angle of the imaging camera; and

a roll angle of the imaging camera.

11. The method of claim 10, further comprising:

determining a reference orientation based on sensor output from at least one of;

an accelerometer;

a gyroscope;

a digital compass; and

a magnetometer; and

wherein at least one of the yaw angle, the pitch angle, and the roll angle is relative to the reference orientation.

12. The method of claim 1, wherein:

the representation of the first pose further includes a representation of a distance of the electronic device from an object represented by the first 2D feature descriptor.

13. The method of claim 12, wherein storing the 3D feature descriptor further comprises:

indexing the first 2D feature descriptor in the plurality of entries of the 3D feature descriptor further based on the distance.

14. The method of claim 1, further comprising:

normalizing the first 2D feature descriptor based on a specified reference distance.

2. An electronic device comprising:

an imaging camera to capture a first image;

a set of one or more sensors to determine a first pose of the imaging camera at a time of capture of the first image, the first pose including a first observation direction of the imaging camera;

a processor to determine and store a three-dimensional (3D) feature descriptor, the 3D feature descriptor having a plurality of entries, each entry configured to store a corresponding 2D feature descriptor, and wherein a first entry of the plurality of entries stores a first two-dimensional (2D) feature extracted from the first image and a representation of the first pose of the imaging camera, wherein the processor is to index the first 2D feature descriptor in the plurality of entries of the 3D feature descriptor based on the first observation direction; and

wherein the representation of the first pose further includes at least one of;

a representation of a weather condition at the time of capture of the first image; and

a representation of an ambient light condition at the time of capture of the first image.

3. The electronic device of claim 2, wherein:

the imaging camera further is to capture a second image;

the set of one or more sensors further is to determine a second pose of the imaging camera at a time of capture of the second image, the second pose including a second observation direction of the imaging camera; and

the 3D feature descriptor further includes a second entry of the plurality of entries to store a second 2D feature descriptor extracted from the second image and a representation of the second pose of the imaging camera.

4. The electronic device of claim 2, wherein:

the representation of the first pose further includes a representation of an orientation of the imaging camera at the time of capture of the first image.

5. The electronic device of claim 2, wherein:

the representation of the first pose further includes a representation of a distance of the electronic device from an object represented by the first 2D feature descriptor.

6. The electronic device of claim 5, wherein the processor is to index the first 2D feature descriptor in the plurality of entries of the 3D feature descriptor further based on the distance.

7. The electronic device of claim 2, wherein the processor further is to normalize the first 2D feature descriptor based on a specified reference distance.