Radar aided visual inertial odometry outlier removal

• US 10,514,456 B2
• Filed: 11/29/2017
• Issued: 12/24/2019
• Est. Priority Date: 11/29/2017
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Niesen, Urs
• Primary Examiner(s): Dang, Duy M
• Application Number:
  US15/826,556
• Publication Number:
  US 20190162842A1
• Time in Patent Office: 755 Days
• Field of Search: 382103, 382 14
• CPC Class Codes: B60W 10/18 including control of braking systems
• CPC Class Codes: B60W 10/20 including control of steering systems
• CPC Class Codes: B60W 2420/408 Radar; Laser, e.g. lidar
• CPC Class Codes: B60W 2554/00 Input parameters relating to objects
• CPC Class Codes: B60W 2710/18 Braking system
• CPC Class Codes: B60W 2710/20 Steering systems
• CPC Class Codes: B60W 30/09 Taking automatic action to avoid collision, e.g. braking and steering
• CPC Class Codes: G01C 21/3602 Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera
• CPC Class Codes: G01S 13/42 Simultaneous measurement of distance and other co-ordinates indirect measurement G01S13/46
• CPC Class Codes: G01S 13/58 Velocity or trajectory determination systems; Sense-of-movement determination systems
• CPC Class Codes: G01S 13/72 for two-dimensional tracking, e.g. combination of angle and range tracking, track-while-scan radar
• CPC Class Codes: G01S 13/726 Multiple target tracking
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/931 of land vehicles
• CPC Class Codes: G01S 2013/9323 Alternative operation using light waves
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/248 : involving reference images or patches
• CPC Class Codes: G06T 7/74 : involving reference images or patches

Abstract

Various embodiments disclose a device with one or more processors which may be configured to translate a RADAR velocity map in at least one image plane of at least one camera, to form a three-dimensional RADAR velocity image. The 3D RADAR velocity image includes a relative velocity of each pixel in the one or more images, and the relative velocity of each pixel is based on a RADAR velocity estimate in the three-dimensional RADAR velocity map. The one or more processors may be configured to determine whether visual features correspond to a moving object based on the relative velocity of each pixel determined, and may be configured to remove the visual features that correspond to a moving object, prior to providing them as an input into a state updater, in a RADAR-aided visual inertial odometer.

Claims

1. A device comprising:

one or more processors configured to;

obtain one or more images from at least one camera;

translate a radio detection and ranging (RADAR) velocity map in at least one image plane of at least one camera, to form a three-dimensional RADAR velocity image, wherein the 3D RADAR velocity image includes a relative velocity of each pixel in the one or more images, wherein the relative velocity of each pixel is based on a RADAR velocity estimate in the three-dimensional RADAR velocity map;

detect visual features in the one or more images;

determine whether the visual features correspond to a moving object based on the relative velocity of each pixel determined;

remove the visual features that correspond to a moving object, prior to providing them as an input into a state updater, in a RADAR-aided visual inertial odometer; and

refine an estimated position, based on the removal of the visual features that correspond to the moving object.

2. The device of claim 1, further comprising at least one RADAR sensor configured to provide the RADAR velocity map.

3. The device of claim 1, further comprising at least one camera configured to provide at least one image, wherein the image is located in the at least image plane of the at least one camera.

4. The device of claim 1, wherein the remaining visual features after the visual features that correspond to the moving object are removed, are provided to a RADAR-based feature associator.

5. The device of claim 4, wherein the refined estimated position of the device is based on the one or more processors configured to:

estimate at least a three-dimensional position of the device in a world reference frame, based on a three-dimensional position of tracked visual features, across two or more images, in the world reference frame;

detect one or more two-dimensional positions of visual features in a current image;

associate the one or more two-dimensional positions of the visual features of the current image with predicted three-dimensional positions of the visual features of the current image;

select two-dimensional positions of new visual features based on the association in the RADAR-based feature associator; and

provide new visual features based on the association in the RADAR-based feature associator.

6. The device of claim 1, further comprising a device controller including a steer controller configured to steer the device based on the removal of the visual features that correspond to a moving object, prior to providing them as an input into the state updater, in the RADAR-aided visual inertial odometer.

7. The device of claim 1, further comprising a device controller including a break controller configured to slow down the device based on the removal of the visual features that correspond to a moving object, prior to providing them as an input into the state updater, in the RADAR-aided visual inertial odometer.

8. The device of claim 1, further comprising a use alerter including a visual converter configured to convert, based on the removal of the visual features that correspond to a moving object, prior to providing the visual features as an input into the state updater, the refined estimated position of the device of the device into a visual representation of the refined estimated position of the device.

9. The device of claim 8, further comprising a display device configured to display, based on the removal of the visual features that correspond to a moving object, prior to providing the visual features as an input into the state updater, the visual representation of at least one of:

(a) the refined estimated position, (b) the refined estimated orientation, and (c) the refined estimated velocity of the device.

10. The device of claim 4, further comprising a use alerter including an audible converter configured to convert, based on the removal of the visual features that correspond to a moving object, prior to providing the visual features as an input into the state updater, at least one of:

(a) the refined estimated position, (b) the refined estimated orientation, and (c) the refined estimated velocity of the device into an audible representation of at least one of;

(a) the refined estimated position, (b) the refined estimated orientation, and (c) the refined estimated velocity of the device.

11. The device of claim 10, further comprising one or more loudspeakers configured to play-out the audible representation of at least one of:

(a) the refined estimated position, (b) the refined estimated orientation, and (c) refined estimated velocity of the device.

12. The device of claim 4, further comprising one or more antennas and a transceiver, coupled to the one or more antennas, configured to transmit at least on of:

(a) the refined estimated position, (b) the refined estimated orientation, and (c) the refined estimated velocity of the device,

based on the removal of the visual features that correspond to a moving object, prior to providing them as an input into the state updater, in the RADAR-aided visual inertial odometer.

13. The device of claim 12, wherein the one or more antennas and a transceiver are configured to receive, from a second device, at least one of a position, orientation, and velocity of the second device, for the device'"'"'s path planning,

based on the removal of the visual features that correspond to a moving object, prior to providing them as an input into the state updater, in the RADAR-aided visual inertial odometer.

14. The device of claim 1, further comprising a path planner configured to plan a navigation route,

based on the removal of the visual features that correspond to a moving object, prior to providing them as an input into the state updater, in the RADAR-aided visual inertial odometer.

15. A method comprising:

obtaining one or more images from at least one camera;

translating a radio detection and ranging (RADAR) velocity map in at least one image plane of at least one camera, to form a three-dimensional RADAR velocity image, wherein the 3D RADAR velocity image includes a relative velocity of each pixel in the one or more images, wherein the relative velocity of each pixel is based on a RADAR velocity estimate in the three-dimensional RADAR velocity map;

detecting visual features in the one or more images;

determining whether the visual features correspond to a moving object based on the relative velocity of each pixel determined;

removing the visual features that correspond to a moving object, prior to providing them as an input into a state updater, in a RADAR-aided visual inertial odometer; and

refining at least one of;

(a) an estimated position, (b) an estimated orientation, and (c) velocity of a device, based on the removal of the visual features that correspond to the moving object.

16. The method of claim 15, after the visual features that correspond to the moving object are removed, providing the remaining visual features to a RADAR-based feature associator.

17. The method of claim 15, wherein the refined estimated position and the refined estimated velocity of the device is based on:

estimating at least a three-dimensional position of the device in a world reference frame, based on a three-dimensional position of tracked visual features, across two or more images, in the world reference frame;

detecting one or more two-dimensional positions of visual features in a current image;

associating the one or more two-dimensional positions of the visual features of the current image with predicted three-dimensional positions of the visual features of the current image;

selecting two-dimensional positions of new visual features based on the association in the RADAR-based feature associator; and

providing new visual features based on the association in the RADAR-based feature associator.

18. The method of claim 15, further comprising steering the device based on the removal of the visual features that correspond to a moving object, prior to providing them as an input into a state updater in the RADAR-aided visual inertial odometer.

19. The method of claim 15, further comprising slowing down the device based on the removal of the visual features that correspond to a moving object, prior to providing them as an input into a state updater in the RADAR-aided visual inertial odometer.

20. The method of claim 15, further comprising displaying a visual representation of at least one of:

(a) a refined estimated position, (b) a refined estimated orientation, and (c) a refined estimated velocity of the device.

21. The method of claim 15, further comprising playing-out, one or more loudspeakers, an audible representation of at least one of:

(a) refined estimated position, (b) a refined estimated orientation, and (c) a refined estimated velocity of the device.

22. The method of claim 15, further comprising transmitting at least one of:

(a) refined estimated position, (b) a refined estimated orientation, and (c) a refined estimated velocity of the device,

based on the removal of the visual features that correspond to a moving object, prior to providing them as an input into the state updater, in the RADAR-aided visual inertial odometer.

23. The method of claim 15, further comprising receiving, from a second device, at least one of a position, orientation, and velocity of the second device, for the device'"'"'s path planning,

based on the removal of the visual features that correspond to a moving object, prior to providing them as an input into the state updater in the RADAR-aided visual inertial odometer.

24. The method of claim 15, further comprising planning a navigation route,

based on the removal of the visual features that correspond to a moving object, prior to providing them as an input into the state updater in the RADAR-aided visual inertial odometer.

25. An apparatus comprising:

means for obtaining one or more images from at least one camera;

means for translating a radio detection and ranging (RADAR) velocity map in at least one image plane of the at least one camera, to form a three-dimensional RADAR velocity image, wherein the 3D RADAR velocity image includes a depth estimate of each pixel in the one or more images, and a relative velocity of each pixel in the one or more images, where the relative velocity of each pixel is based on a RADAR velocity estimate in the three-dimensional RADAR velocity map;

means for detecting visual features in the one or more images;

means for determining whether the visual features correspond to a moving object based on the relative velocity of each pixel determined;

means for removing the visual features that correspond to a moving object, prior to providing them as an input into a state updater, in a RADAR-aided visual inertial odometer; and

means for refining at least one of;

(a) an estimated position, (b) an estimated orientation, and (c) velocity of the device, based on the removal of the visual features that correspond to the moving object.

26. A non-transitory computer-readable storage medium having stored thereon instructions that, when executed, cause one or more processors of a device to:

obtain one or more images from at least one camera;

translate a radio detection and ranging (RADAR) velocity map in at least one image plane of the at least one camera, to form a three-dimensional RADAR velocity image, wherein the 3D RADAR velocity image includes a depth estimate of each pixel in the one or more images, and a relative velocity of each pixel in the one or more images, where the relative velocity of each pixel is based on a RADAR velocity estimate in the three-dimensional RADAR velocity map;

detect visual features in the one or more images;

determine whether the visual features correspond to a moving object based on the relative velocity of each pixel determined;

remove the visual features that correspond to a moving object, prior to providing them as an input into a state updater, and

refine at least on of (a) an estimated position, (b) an estimated orientation, and (c) velocity of the device, based on the removal of the visual features that correspond to the moving object.