Methods and systems for vision-based motion estimation

• US 10,339,389 B2
• Filed: 09/03/2014
• Issued: 07/02/2019
• Est. Priority Date: 09/03/2014
• Status: Expired due to Fees

• Current Assignee: Sharp Kabushiki Kaisha (Hon Hai Precision Industry Co., Ltd.)
• Original Assignee: Sharp Laboratories of America Incorporated (Hon Hai Precision Industry Co., Ltd.)
• Inventors: Xu, Xinyu, Liao, Miao, van Beek, Petrus J. L.
• Primary Examiner(s): Moyer, Andrew M
• Assistant Examiner(s): Rosario, Dennis
• Application Number:
  US14/476,524
• Publication Number:
  US 20160063330A1
• Time in Patent Office: 1,763 Days
• CPC Class Codes: G05D 1/0253 extracting relative motion information from a plurality of images taken successively, e.g. visual odometry, optical flow determining position or orientation from images G06T7/70
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle

Abstract

Aspects of the present invention are related to methods and systems for vision-based computation of ego-motion.

Claims

1. A visual odometry method for estimating vehicle motion, said method comprising:

providing a vehicle equipped with a camera rigidly mounted and calibrated with respect to the vehicle;

moving the vehicle from a starting position in a global coordinate frame to a first location across a planar ground plane;

receiving an incoming image on the camera;

performing feature detection on said incoming image to identify a plurality of regions, wherein each region in said plurality of regions is associated with a key point in an incoming image coordinate frame;

selecting a feature descriptor for each region in said plurality of regions, thereby producing a plurality of feature descriptors for said incoming image coordinate frame;

performing feature matching between said plurality of feature descriptors for said incoming image coordinate frame and a plurality of feature descriptors selected for a previous image coordinate frame, thereby producing a plurality of feature correspondences;

for each feature correspondence in said plurality of feature correspondences, aligning key points from said previous image coordinate frame to said incoming image coordination frame;

projecting said key points to a previous world coordinate frame and an incoming world coordinate frame at the planar ground plane, thereby producing a plurality of pairs of world coordinates at the planar ground plane;

estimating vehicle motion from said plurality of pairs of world coordinates at the planar ground plane;

minimizing an accumulation of errors by selecting a key pose as follows;

comparing a rotation angle of said motion estimate to an angle threshold and a distance traveled by the vehicle to a distance threshold; and

when said angle comparison meets a first criterion or said distance comparison meets a second criterion;

projecting said key pose to a current camera pose in said global coordinate frame;

determining a motion trajectory from said current camera pose; and

updating said plurality of feature descriptors selected for a previous world coordinate frame to said plurality of feature descriptors for said incoming world coordinate frame, wherein

said motion estimate comprises a rotation matrix and a translation vector;

said rotation matrix is associated with no more than three degrees of freedom and an in-plane rotation that is in-plane with respect to the ground plane;

said motion estimate comprises a rigid transformation T<sub>k,k−

1</sub>

where said rigid transformation T<sub>k,k−

1</sub> consists of the rotation matrix R<sub>k,k−

1</sub> between a previous time (k−

1) and a current time (k), and the translation vector t<sub>k−

k−

1</sub> between the previous time and the current time, such that

2. The method as described in claim 1, wherein said performing feature detection comprises using a feature detector selected from the group consisting of an edge detector, a corner detector, a blob detector, a peak detector, a SIFT key-point detector, a SURF key-point detector, a FAST key-point detector, a grid-based FAST key-point detector, an ORB key-point detector, a MSER key-point detector, a BRIEF key-point detector, a BRISK key-point detector, a FREAK key-point detector and a STAR key-point detector.

3. The method as described in claim 1, wherein said feature descriptor is a feature descriptor selected from the group consisting of a block of pixel values, a block of normalized pixel values, a block of gradient values, a block of adjusted pixel values, a SIFT feature descriptor, a SURF feature descriptor, an ORGB feature descriptor, a BRIEF feature descriptor, a BRISK feature descriptor and a FREAK feature descriptor.

4. The method as described in claim 1, wherein said performing feature matching comprises a fast, nearest-neighbor search with a k-dimensional tree.

5. The method as described in claim 1, wherein said performing feature matching comprises pruning a plurality of candidate matches.

6. The method as described in claim 1, wherein said computing the motion estimate comprises using a motion estimator selected from the group consisting of an Orthogonal-Procrustes-Analysis-based motion estimator and an Absolute-Orientation-based motion estimator.

7. The method as described in claim 1, wherein performing feature detection comprises performing feature detection using a first mathematical model;

and,

wherein estimating vehicle motion comprises rejecting outliers not accounted for in said first mathematical model.

8. The method as described in claim 7, wherein said rejecting outliers comprises performing RANSAC motion estimation.

9. The method as described in claim 1, wherein:

said first criterion is said rotation angle is greater than said angle threshold; and

said second criterion is said distance traveled is greater than said distance threshold.

10. The method as described in claim 1, wherein said generating a current camera pose comprises concatenating said estimated motion to a previous camera pose.

11. The method of claim 1 wherein estimating said motion using said rigid transformation T_k,k−

1 comprises minimizing the mean-squared-error (MSE), denoted as E, as follows;

12. The method of claim 1 wherein minimizing the accumulation of errors comprises decreasing the accumulation of errors in response to an action selected from the group consisting of increasing the angle threshold or increasing the distance threshold.

13. A vehicle odometry system comprising:

a vehicle in motion from a starting position in a global coordinate frame to a first location;

a camera rigidly mounted and calibrated with respect to the vehicle;

a processor component; and

a non-transitory computer-readable medium encoded with a computer program code for causing said processor component to execute a method for estimating vehicle motion, said method comprising;

receiving an incoming image;

performing feature detection on said incoming image to identify a plurality of regions, wherein each region in said plurality of regions is associated with a key point in an incoming image coordinate frame;

selecting a feature descriptor for each region in said plurality of regions, thereby producing a plurality of feature descriptors for said incoming image coordinate frame;

performing feature matching between said plurality of feature descriptors for said incoming image coordinate frame and a plurality of feature descriptors selected for a previous image coordinate frame, thereby producing a plurality of feature correspondences;

for each feature correspondence in said plurality of feature correspondences, aligning key points from said previous image coordinate frames to said incoming image coordinate frame;

projecting said keys points to a previous world coordinate frame and an incoming world coordinate frame at the planar ground plane, thereby producing a plurality of pairs of world coordinates at the planar ground plane;

estimating vehicle motion from said plurality of pairs of world coordinates at the planar ground plane;

minimizing an accumulation of errors by selecting a key pose as follows;

comparing a rotation angle of said motion estimate to an angle threshold and a distance traveled by the vehicle to a distance threshold; and

when said angle comparison meets a first criterion or said distance comparison meets a second criterion;

projecting said key pose to a current camera pose in said global coordinate frame;

determining a motion trajectory from said current camera pose;

updating said plurality of feature descriptors computed for a previous world coordinate frame to said plurality of feature descriptors for said incoming world coordinate frame, wherein

said motion estimate comprising a rotation matrix and a translation vector, said rotation matrix being associated with no more than three degrees of freedom and an in-plane rotation that is in-plane with respect to the ground plane;

said motion estimate comprises a rigid transformation T<sub>k,k−

1</sub>; and

said rigid transformation T<sub>k,k−

1</sub> consists of the rotation matrix R<sub>k,k−

1</sub> between a previous time and a current time, and the translation vector t<sub>k−

k−

1</sub> between the previous time and the current time, such that

14. The system as described in claim 13, wherein:

said camera is mounted on a front end or a back end of said vehicle and above a body chassis of said vehicle; and

when said camera is mounted on said front end of said vehicle, said camera is tilted to have a field of view of a ground path directly in front of said vehicle and when said camera is mounted on said back end of said vehicle, said camera is tilted to have a field of view of said ground path directly behind said vehicle.

15. A non-transitory computer-readable medium encoded with a computer program code for causing a processor to execute a visual odometry method for estimating vehicle motion, said code comprising the steps of:

receiving an incoming image from a camera rigidly mounted and calibrated with respect to a vehicle in motion from a starting position in a global coordinate frame to a first location;

performing feature detection on said received, incoming image to identify a plurality of regions, wherein each region in said plurality of regions is associated with a key point in an incoming image coordinate frame;

selecting a feature descriptor for each region in said plurality of regions, thereby producing a plurality of feature descriptors for said incoming image coordination frame;

performing feature matching between said plurality of feature descriptors for said incoming image and a plurality of feature descriptors computed for a previous image coordination frame, thereby producing a plurality of feature correspondences;

for each feature correspondence in said plurality of feature correspondences, aligning key points from said previous image coordinate frames to said incoming image coordination frame;

projecting said key points to a previous world coordinate frame and an incoming world coordinate frame at the planar ground plane, thereby producing a plurality of pairs of world coordinates at the planar ground plane;

estimating vehicle motion from said plurality of pairs of world coordinates at the planar ground plane;

minimizing an accumulation of errors by selecting a key pose as follows;

comparing a rotation angle of said motion estimate to an angle threshold and a distance traveled by the vehicle to a distance threshold; and

when said angle comparison meets a first criterion or said distance comparison meets a second criterion;

projecting said key pose to a current camera pose in said global coordinate frame;

determining a motion trajectory from said current camera pose;

updating said plurality of feature descriptors computed for a previous world coordinate frame to said plurality of feature descriptors for said incoming world coordinate frame, wherein

said motion estimate comprising a rotation matrix and a translation vector, said rotation matrix being associated with no more than three degrees of freedom and an in-plane rotation that is in-plane with respect to the ground plane;

said motion estimate comprises a rigid transformation T<sub>k,k−

1</sub>; and

said rigid transformation T<sub>k,k−

1</sub> consists of the rotation matrix R<sub>k,k−

1</sub> between a previous time and a current time, and the translation vector t<sub>k−

k−

1</sub> between the previous time and the current time, such that

16. The non-transitory computer-readable medium as described in claim 15, wherein said processor is integrated on said vehicle.