Odometry feature matching

• US 9,990,547 B2
• Filed: 08/15/2016
• Issued: 06/05/2018
• Est. Priority Date: 02/20/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Lee, Johnny, Hesch, Joel
• Primary Examiner(s): Elhag, Magdi
• Application Number:
  US15/236,881
• Publication Number:
  US 20170018092A1
• Time in Patent Office: 659 Days
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/66 of image moments or centre of gravity
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: H04W 4/023 using mutual or relative location information between multiple location based services [LBS] targets or of distance thresholds
• CPC Class Codes: H04W 4/027 using movement velocity, acceleration information
• CPC Class Codes: H04W 4/029 Location-based management or tracking services

Abstract

Methods and systems for determining features of interest for following within various frames of data received from multiple sensors of a device are disclosed. An example method may include receiving data from a plurality of sensors of a device. The method may also include determining, based on the data, motion data that is indicative of a movement of the device in an environment. The method may also include as the device moves in the environment, receiving image data from a camera of the device. The method may additionally include selecting, based at least in part on the motion data, features in the image data for feature-following. The method may further include estimating one or more of a position of the device or a velocity of the device in the environment as supported by the data from the plurality of sensors and feature-following of the selected features in the images.

Claims

1. A method comprising:

receiving, using a processor, data from a plurality of sensors of a device;

determining, based on the data, motion data that is indicative of a movement of the device in an environment;

as the device moves in the environment, receiving image data from a camera of the device;

selecting, based at least in part on the motion data, features in the image data for feature-following, wherein selecting features in the image data for feature-following comprises;

determining, based on the image data, a plurality of sections of a first image of the image data; and

selecting for each of the plurality of sections a threshold number of features for feature-following; and

estimating one or more of a position of the device or a velocity of the device in the environment as supported by the data from the plurality of sensors and by feature-following of the selected features in the image data.

2. The method of claim 1,

wherein the plurality of sensors includes an inertial measurement unit (IMU),

wherein the motion data includes data received from the IMU that is indicative of a direction of gravity being applied to the device, and

wherein selecting features in the image data for feature-following further comprises:

identifying one or more vertical lines in the image data as a line that is parallel to the direction of gravity; and

providing the identified vertical line as a given feature for feature-following.

3. The method of claim 1,

wherein the plurality of sensors includes an inertial measurement unit (IMU),

wherein the motion data includes data received from the IMU that is indicative of a direction of gravity being applied to the device, and

wherein selecting features in the image data for feature-following further comprises:

identifying one or more horizontal lines in the image data as a line that is perpendicular to the direction of gravity; and

providing the identified horizontal line as a given feature for feature-following.

4. The method of claim 1,

wherein the plurality of sensors includes a depth sensor,

wherein the motion data includes data received from the depth sensor that is indicative of a plurality of depths associated with the movement of the device, and

wherein selecting features in the image data for feature-following further comprises:

determining, based on respective depths of the plurality of depths associated with the movement of the device, a plurality of cells in the image data that are divided by respective depths of the plurality of depths; and

selecting for each of the plurality of cells a threshold number of features for feature-following.

5. The method of claim 1, wherein respective sections of the plurality of sections are defined based on quadrants associated with the image data.

6. The method of claim 1,

wherein the camera comprises a wide-angle view camera, and

wherein respective sections of the plurality of sections are defined based on angles associated with the image data.

7. The method of claim 1, further comprising:

receiving, using the processor, environmental data from the plurality of sensors of the device, wherein the environmental data includes shape data that defines one or more shapes of objects in the environment; and

wherein selecting features in the image data for feature following further comprises selecting features in the image data for feature-following further based at least in part on the shape data.

8. The method of claim 1, further comprising:

determining, based on the motion data, an uncertainty of movement of the device that defines a level of uncertainty of a portion of the movement of the device; and

wherein selecting features in the image data for feature-following comprises selecting features in the image data for feature-following further based on the uncertainty of movement of the device.

9. The method of claim 8,

wherein the uncertainty of movement of the device comprises an uncertainty of a roll or a pitch of the device, and

wherein selecting the features in the image data for feature-following comprises selecting horizontal lines in the image data for feature-following.

10. The method of claim 8,

wherein the uncertainty of movement of the device comprises an uncertainty of a yaw of the device, and

wherein selecting the features in the image data for feature-following comprises selecting vertical lines in the image data for feature-following.

11. A non-transitory computer readable memory configured to store instructions that, when executed by a device, cause the device to perform functions comprising:

receiving, at the device, data from a plurality of sensors of the device;

determining, based on the data, motion data that is indicative of a movement of the device in an environment;

as the device moves in the environment, receiving image data from a camera of the device;

selecting, based at least in part on the motion data, features in the image data for feature-following, wherein selecting features in the image data for feature-following comprises;

determining, based on the image data, a plurality of sections of a first image of the image data; and

selecting for each of the plurality of sections a threshold number of features for feature-following; and

estimating one or more of a position of the device or a velocity of the device in the environment as supported by the data from the plurality of sensors and by feature-following of the selected features in the image data.

12. The non-transitory computer readable memory of claim 11,

wherein the plurality of sensors includes an inertial measurement unit (IMU),

wherein the motion data includes data received from the IMU that is indicative of a direction of gravity being applied to the device, and

wherein selecting features in the image data for feature-following further comprises:

identifying one or more vertical lines in the image data as a line that is parallel to the direction of gravity or one or more horizontal lines in the image data as a line that is perpendicular to the direction of gravity; and

providing the identified vertical line or the identified horizontal line as a given feature for feature-following.

13. The non-transitory computer readable memory of claim 11,

wherein the plurality of sensors includes a depth sensor,

wherein the motion data includes data received from the depth sensor that is indicative of a plurality of depths associated with the movement of the device, and

wherein selecting features in the image data for feature-following further comprises:

determining, based on respective depths of the plurality of depths associated with the movement of the device, a plurality of cells in the image data that are divided by respective depths of the plurality of depths; and

selecting for each of the plurality of cells a threshold number of features for feature-following.

14. The non-transitory computer readable memory of claim 11, further comprising:

receiving environmental data from the plurality of sensors of the device, wherein the environmental data includes shape data that defines one or more shapes of objects in the environment; and

wherein selecting features in the image data for feature following further comprises selecting features in the image data for feature following further based at least in part on the shape data.

15. The non-transitory computer readable memory of claim 11, further comprising:

determining, based on the motion data, an uncertainty of movement of the device that defines a level of uncertainty of a portion of the movement of the device; and

selecting features in the image data for feature-following further comprises selecting features in the image data for feature-following further based on the uncertainty of movement of the device.

16. A device comprising:

one or more processors; and

data storage configured to store instructions that, when executed by the one or more processors, cause the device to perform functions comprising;

receiving data from a plurality of sensors of the device;

determining, based on the data, motion data that is indicative of a movement of the device in an environment;

as the device moves in the environment, receiving image data from a camera of the device;

selecting, based at least in part on the motion data, features in the image data for feature-following, wherein selecting features in the image data for feature-following comprises;

determining, based on the image data, a plurality of sections of a first image of the image data; and

selecting for each of the plurality of sections a threshold number of features for feature-following; and

estimating one or more of a position of the device or a velocity of the device in the environment as supported by the data from the plurality of sensors and by feature-following of the selected features in the image data.

17. The device of claim 16, wherein the functions further comprise:

receiving, using the one or more processors, environmental data from the plurality of sensors of the device, wherein the environmental data includes shape data that defines one or more shapes of objects in the environment; and

wherein selecting features in the image data for feature-following further comprises selecting features in the image data for feature-following further based at least in part on the shape data.

18. The device of claim 16, wherein the functions further comprise:

determining, based on the motion data, an uncertainty of movement of the device that defines a level of uncertainty of a portion of the movement of the device; and

wherein selecting features in the image data for feature-following further comprises selecting features in the image data for feature-following further based at least in part on the uncertainty of movement of the device.

19. The device of claim 16, wherein respective sections of the plurality of sections are defined based on quadrants associated with the image data.

20. The device of claim 16,

wherein the camera comprises a wide-angle view camera, and

wherein respective sections of the plurality of sections are defined based on angles associated with the image data.