Real-time range map generation

• US 9,378,554 B2
• Filed: 10/09/2014
• Issued: 06/28/2016
• Est. Priority Date: 10/09/2014
• Status: Active Grant

• Current Assignee: Caterpillar Incorporated
• Original Assignee: Caterpillar Incorporated
• Inventors: Chang, Hong, Friend, Paul Russell
• Primary Examiner(s): Chawan, Sheela C
• Application Number:
  US14/510,869
• Publication Number:
  US 20160104289A1
• Time in Patent Office: 628 Days
• Field of Search: 382/118, 382/117, 382/199, 382/209, 382/217, 382/276, 382/285, 382/294, 382/312, 382/243, 382/154, 396/159, 396/52, 396/55, 396/281, 396/287, 396/290, 396/103, 396/89, 250/559.29, 250/559.3, 250/200, 250/216, 250/229, 250/231.1, 250/231.13, 250/231.14, 250/231.16
• US Class Current: 1/1
• CPC Class Codes: G01S 17/89 for mapping or imaging
• CPC Class Codes: G05D 1/0248 in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: H04N 13/207 using a single 2D image sensor

Abstract

A system, method, and non-transitory computer-readable storage medium for range map generation is disclosed. The method may include receiving an image from a camera and receiving a 3D point cloud from a range detection unit. The method may further include transforming the 3D point cloud from range detection unit coordinates to camera coordinates. The method may further include projecting the transformed 3D point cloud into a 2D camera image space corresponding to the camera resolution to yield projected 2D points. The method may further include filtering the projected 2D points based on a range threshold. The method may further include generating a range map based on the filtered 2D points and the image.

Claims

1. A range map generation system comprising:

a range detection unit;

a camera; and

a controller configured to execute instructions to perform operations including;

receiving an image from the camera;

receiving a 3D point cloud from the range detection unit;

transforming the 3D point cloud from range detection unit coordinates to camera coordinates;

projecting the transformed 3D point cloud into a 2D camera image space corresponding to the camera resolution to yield projected 2D points;

filtering the projected 2D points based on a range threshold; and

generating a range map based on the filtered 2D points and the image.

2. The system of claim 1, wherein transforming the 3D point cloud from range detection unit coordinates to camera coordinates includes utilizing a rotation matrix and a translation vector to transform the 3D point cloud from range detection unit coordinates to camera coordinates.

3. The system of claim 1, wherein projecting the transformed 3D point cloud includes utilizing intrinsic calibration information of the camera to project the transformed 3D point cloud into the 2D camera image space.

4. The system of claim 3, wherein the intrinsic camera calibration includes a focal length of the camera and center of projection.

5. The system of claim 1, wherein each of the projected 2D points have a range associated therewith, and filtering the projected 2D points includes:

setting a minimum range and a maximum range, and

determining filtered 2D points by determining the projected 2D points whose associated range falls within the minimum and maximum range.

6. The system of claim 1, wherein generating the range map includes identifying a feature in the image and associating one of the filtered 2D points with the identified feature.

7. The system of claim 6, wherein associating one of the filtered 2D points with the identified feature includes:

determining a pixel position of the identified feature in the 2D camera image space, and

determining the one filtered 2D point using a matching window around the pixel position of the identified feature.

8. A computer-implemented method for range map generation, the method comprising:

receiving an image from a camera;

receiving a 3D point cloud from a range detection unit;

transforming the 3D point cloud from range detection unit coordinates to camera coordinates;

projecting the transformed 3D point cloud into a 2D camera image space corresponding to the camera resolution to yield projected 2D points;

filtering the projected 2D points based on a range threshold; and

generating a range map based on the filtered 2D points and the image.

9. The method of claim 8, wherein transforming the 3D point cloud from range detection unit coordinates to camera coordinates includes utilizing a rotation matrix and a translation vector to transform the 3D point cloud from range detection unit coordinates to camera coordinates.

10. The method of claim 8, wherein projecting the transformed 3D point cloud includes utilizing intrinsic calibration information of the camera to project the transformed 3D point cloud into the 2D camera image space.

11. The method of claim 10, wherein the intrinsic camera calibration includes a focal length of the camera and center of projection.

12. The method of claim 8, wherein each of the projected 2D points have a range associated therewith, and filtering the projected 2D points includes:

setting a minimum range and a maximum range, and

determining filtered 2D points by determining the projected 2D points whose associated range falls within the minimum and maximum range.

13. The method of claim 8, wherein generating the range map includes identifying a feature in the image and associating one of the filtered 2D points with the identified feature.

14. The method of claim 13, wherein associating one of the filtered 2D points with the identified feature includes:

determining a pixel position of the identified feature in the 2D camera image space, and

determining the one filtered 2D point using a matching window around the pixel position of the identified feature.

15. A non-transitory computer-readable storage medium storing instructions that enable a computer to implement a method for range map generation, the method comprising:

receiving an image from a camera;

receiving a 3D point cloud from a range detection unit;

transforming the 3D point cloud from range detection unit coordinates to camera coordinates;

projecting the transformed 3D point cloud into a 2D camera image space corresponding to the camera resolution to yield projected 2D points;

filtering the projected 2D points based on a range threshold; and

generating a range map based on the filtered 2D points and the image.

16. The non-transitory computer-readable storage medium of claim 15, wherein transforming the 3D point cloud from range detection unit coordinates to camera coordinates includes utilizing a rotation matrix and a translation vector to transform the 3D point cloud from range detection unit coordinates to camera coordinates.

17. The non-transitory computer-readable storage medium of claim 15, wherein projecting the transformed 3D point cloud includes utilizing intrinsic calibration information of the camera to project the transformed 3D point cloud into the 2D camera image space.

18. The non-transitory computer-readable storage medium of claim 17, wherein the intrinsic camera calibration includes a focal length of the camera and center of projection.

19. The non-transitory computer-readable storage medium of claim 15, wherein each of the projected 2D points have a range associated therewith, and filtering the projected 2D points includes:

setting a minimum range and a maximum range, and

determining filtered 2D points by determining the projected 2D points whose associated range falls within the minimum and maximum range.

20. The non-transitory computer-readable storage medium of claim 15, wherein generating the range map includes identifying a feature in the image and associating one of the filtered 2D points with the identified feature.