Odometry feature matching

• US 9,437,000 B2
• Filed: 02/20/2014
• Issued: 09/06/2016
• Est. Priority Date: 02/20/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Lee, Johnny, Hesch, Joel
• Primary Examiner(s): Elhag, Magdi
• Application Number:
  US14/185,616
• Publication Number:
  US 20150235099A1
• Time in Patent Office: 929 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/66 of image moments or centre of gravity
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: H04W 4/023 using mutual or relative location information between multiple location based services [LBS] targets or of distance thresholds
• CPC Class Codes: H04W 4/027 using movement velocity, acceleration information
• CPC Class Codes: H04W 4/029 Location-based management or tracking services

Abstract

Methods and systems for determining features of interest for following within various frames of data received from multiple sensors of a device are disclosed. An example method may include receiving data from a plurality of sensors of a device. The method may also include determining, based on the data, motion data that is indicative of a movement of the device in an environment. The method may also include as the device moves in the environment, receiving image data from a camera of the device. The method may additionally include selecting, based at least in part on the motion data, features in the image data for feature-following. The method may further include estimating one or more of a position of the device or a velocity of the device in the environment as supported by the data from the plurality of sensors and feature-following of the selected features in the images.

Claims

1. A method comprising:

receiving, using a processor, data from a plurality of sensors of a device;

determining, based on the data, motion data that is indicative of a movement of the device in an environment;

as the device moves in the environment, receiving image data from a camera of the device;

selecting, based at least in part on the motion data, features in the image data for feature-following; and

estimating one or more of a position of the device or a velocity of the device in the environment as supported by the data from the plurality of sensors and by feature-following of the selected features in the image data;

wherein the plurality of sensors includes an inertial measurement unit (IMU),

wherein the motion data includes data received from the IMU that is indicative of a direction of gravity being applied to the device, and

wherein selecting features in the image data for feature-following comprise;

identifying one or more vertical lines in the image data as a line that is parallel to the direction of gravity or one or more horizontal lines in the image data as a line that is perpendicular to the direction of gravity; and

providing the identified one or more vertical lines or the identified one or more horizontal lines as a given feature for feature-following.

2. The method of claim 1,

wherein the plurality of sensors further includes a depth sensor,

wherein the motion data includes data received from the depth sensor that is indicative of a plurality of depths associated with the movement of the device, and

wherein selecting features in the image data for feature-following further comprises:

determining, based on respective depths of the plurality of depths associated with the movement of the device, a plurality of cells in the image data that are divided by respective depths of the plurality of depths; and

based on a given cell of the plurality of cells including a number of features less than a threshold number of features, selecting features for feature-following that correspond to the given cell.

3. The method of claim 1,

wherein selecting features in the image data for feature tracking further comprises:

determining, based on the image data, a plurality of sections of the image data; and

based on a given section of the plurality of sections including a number of features less than a threshold number of features, selecting features for feature-following that correspond to the given section.

4. The method of claim 3, wherein respective sections of the plurality of sections are defined based on quadrants associated with the image data.

5. The method of claim 3,

wherein the camera comprises a wide-angle view camera, and

wherein respective sections of the plurality of sections are defined based on angles associated with the image data.

6. The method of claim 1, further comprising:

receiving, using the processor, environmental data from the plurality of sensors of the device, wherein the environmental data includes shape data that defines one or more shapes of objects in the environment; and

selecting, based at least in part on the shape data, the features in the image data for feature-following.

7. The method of claim 1, further comprising:

determining, based on the motion data, an uncertainty of movement of the device that defines a level of uncertainty of a portion of the movement of the device; and

selecting, based on the uncertainly of movement of the device, the features in the image data for feature-following.

8. The method of claim 7,

wherein the uncertainty of movement of the device comprises an uncertainty of a roll or a pitch of the device, and

wherein selecting the features in the image data for feature-following comprises selecting the identified one or more horizontal lines for feature-following.

9. The method of claim 7,

wherein the uncertainty of movement of the device comprises an uncertainty of a yaw of the device, and

wherein selecting the features in the image data for feature-following comprises selecting the identified one or more vertical lines for feature-following.

10. A non-transitory computer readable memory configured to store instructions that, when executed by a device, cause the device to perform functions comprising:

receiving, at the device, data from a plurality of sensors of the device;

determining, based on the data, motion data that is indicative of a movement of the device in an environment;

as the device moves in the environment, receiving image data from a camera of the device;

selecting, based at least in part on the motion data, features in the image data for feature-following;

estimating one or more of a position of the device or a velocity of the device in the environment as supported by the data from the plurality of sensors and by feature-following of the select features in the image data; and

wherein the plurality of sensors includes an inertial measurement unit (IMU),

wherein the motion data includes data received from the IMU that is indicative of a direction of gravity being applied to the device, and

wherein selecting features in the image data for feature-following comprise;

identifying one or more vertical lines in the image data as a line that is parallel to the direction of gravity or one or more horizontal lines in the image data as a line that is perpendicular to the direction of gravity; and

providing the identified one or more vertical lines or the identified one or more horizontal lines as a given feature for feature-following.

11. The non-transitory computer readable memory of claim 10,

wherein the plurality of sensors includes a depth sensor,

wherein the motion data further includes data received from the depth sensor that is indicative of a plurality of depths associated with the movement of the device, and

wherein selecting features in the image data for feature-following further comprises:

determining, based on respective depths of the plurality of depths associated with the movement of the device, a plurality of cells in the image data that are divided by respective depths of the plurality of depths; and

based on a given cell of the plurality of cells including a number of features less than a threshold number of features, selecting features for feature-following that correspond to the given cell.

12. The non-transitory computer readable memory of claim 10,

wherein selecting features in the image data for feature-following further comprises:

determining, based on the image data, a plurality of sections of the image data; and

based on a given section of the plurality of sections including a number of features less than a threshold number of features, selecting features for feature-following that correspond to the given section.

13. The non-transitory computer readable memory of claim 10, further comprising:

receiving, using the device, environmental data from the plurality of sensors of the device, wherein the environmental data includes shape data that defines one or more shapes of objects in the environment; and

selecting, based at least in part on the shape data, the features in the image data for feature-following.

14. The non-transitory computer readable memory of claim 10, further comprising:

determining, based on the motion data, an uncertainty of movement of the device that defines a level of uncertainty of a portion of the movement of the device; and

selecting, based on the uncertainly of movement of the device, the features in the image data for feature-following.

15. A device comprising:

one or more processors; and

data storage configured to store instructions that, when executed by the one or more processors, cause the device to perform functions comprising;

receiving data from a plurality of sensors of the device;

determining, based on the data, motion data that is indicative of a movement of the device in an environment;

as the device moves in the environment, receiving image data from a camera of the device;

selecting, based at least in part on the motion data, features in the image data for feature-following; and

estimating one or more of a position of the device or a velocity of the device in the environment as supported by the data from the plurality of sensors and by feature-following of the selected features in the image data;

wherein the plurality of sensors includes an inertial measurement unit (IMU),

wherein the motion data includes data received from the IMU that is indicative of a direction of gravity being applied to the device, and

wherein the function of selecting features in the image data for feature-following comprise;

identifying one or more vertical lines in the image data as a line that is parallel to the direction of gravity or one or more horizontal lines in the image data as a line that is perpendicular to the direction of gravity; and

providing the identified one or more vertical lines or the identified one or more horizontal lines as a given feature for feature-following.

16. The device of claim 15, wherein the functions further comprise:

receiving, using the one or more processors, environmental data from the plurality of sensors of the device, wherein the environmental data includes shape data that defines one or more shapes of objects in the environment; and

selecting, based at least in part on the shape data, the features in the image data for feature-following.

17. The device of claim 15, wherein the functions further comprise:

determining, based on the motion data, an uncertainty of movement of the device that defines a level of uncertainty of a portion of the movement of the device; and

selecting, based on the uncertainly of movement of the device, the features in the image data for feature-following.