Scan colorization with an uncalibrated camera

• US 10,237,532 B2
• Filed: 03/07/2017
• Issued: 03/19/2019
• Est. Priority Date: 03/07/2017
• Status: Active Grant

• Current Assignee: Trimble AB (Trimble Inc.)
• Original Assignee: Trimble AB (Trimble Inc.)
• Inventors: Monnier, Fabrice, Chaperon, Thomas, Tremblay, Guillaume
• Primary Examiner(s): Gilliard, Delomia L
• Application Number:
  US15/451,834
• Publication Number:
  US 20180262737A1
• Time in Patent Office: 742 Days
• Field of Search: None
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2219/2012 Colour editing, changing, or manipulating; Use of colour codes
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: H04N 13/15 for colour aspects of image signals
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 2013/0077 Colour aspects
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

A method of colorizing a 3D point cloud includes receiving the 3D point cloud, receiving a 2D color image acquired by a camera, creating a 2D intensity image of the 3D point cloud based on intrinsic and extrinsic parameters of the camera, generating a set of refined camera parameters by matching the 2D intensity image and the 2D color image, creating a depth buffer for the 3D point cloud using the set of refined camera parameters, determining a foreground depth for each respective pixel of the depth buffer, and coloring the point cloud by, for each respective point of the 3D point cloud: upon determining that the respective point is in the foreground, assigning a color of a corresponding pixel in the 2D color image to the respective point; and upon determining that the respective point is not in the foreground, not assigning any color to the respective point.

Claims

1. A method of colorizing a three-dimensional (3D) point cloud, the method comprising:

receiving the 3D point cloud, the 3D point cloud acquired by a 3D imaging device positioned at a first location and including 3D coordinates and intensity data for a set of points representing surfaces of one or more objects;

receiving a two-dimensional (2D) color image of the one or more objects acquired by a camera positioned at an approximately known second location and an approximately known orientation, the camera having an approximately known set of intrinsic optical parameters;

creating a 2D intensity image of the 3D point cloud using the intensity data by projecting the 3D point cloud onto a 2D image area from a view point of the second location based on the set of intrinsic optical parameters, the second location, and the orientation of the camera, the 2D image area having a size that is same as a size of the 2D color image;

generating a set of refined camera parameters by matching the 2D intensity image and the 2D color image;

projecting the set of points of the 3D point cloud onto the 2D image area using the set of refined camera parameters;

creating a depth buffer for the 3D point cloud, the depth buffer having an image area of a size that is same as the size of the 2D color image and a pixel size that is same as a pixel size of the 2D color image, the depth buffer including depth data for the set of points in the 3D point cloud as the set of points are projected onto pixels of the depth buffer using the set of refined camera parameters;

determining a foreground depth for each respective pixel of the depth buffer by detecting a closest point among a subset of the set of points corresponding to the respective pixel; and

coloring the point cloud by, for each respective point of the set of points;

comparing a depth of the respective point with a foreground depth of a corresponding pixel in the depth buffer;

upon determining that the depth of the respective point is within a predetermined threshold value from the foreground depth of the corresponding pixel, assigning a color of a corresponding pixel in the 2D color image to the respective point; and

upon determining that the depth of the respective point differs from the foreground depth of the corresponding pixel by an amount greater than the predetermined threshold value, not assigning any color to the respective point.

2. The method of claim 1, wherein the set of intrinsic optical parameters of the camera includes one or more of a focal length, an optical center, a principal point, and distortion parameters.

3. The method of claim 1, wherein the camera is attached to the 3D imaging device, and the second location and the orientation of the camera are determined from mechanical design information of the camera with respect to the 3D imaging device.

4. The method of claim 1, wherein generating the set of refined camera parameters comprises:

detecting a first set of features in the 2D intensity image;

detecting a second set of features in the 2D color image; and

generating the set of refined camera parameters so that, when the set of refined camera parameters is applied to the 2D intensity image, the first set of features in the 2D intensity image matches with the second set of features in the 2D color image.

5. The method of claim 4, wherein each of the first set of features and the second set of features includes a set of edges.

6. The method of claim 4, wherein the set of refined camera parameters includes a translation parameter, a rotation parameter, and a scaling parameter.

7. The method of claim 1, wherein the 3D imaging device comprises a laser scanner.

8. The method of claim 1, wherein the camera comprises a thermal camera.

9. The method of claim 1, wherein the camera comprises a panoramic camera.

10. The method of claim 1, wherein the second location and the orientation of the camera are determined using one or more of a global navigation satellite system (GNSS), an inertial measurement unit (IMU), a compass, and a pressure sensor.

11. The method of claim 1, further comprising storing the 3D point cloud including the 3D coordinates and the intensity data for the set of points, and color data for a subset of the set of points that are assigned colors.

12. A method of camera calibration with respect to a three-dimensional (3D) imaging device, the method comprising:

receiving a 3D point cloud acquired by the 3D imaging device positioned at a first location and including 3D coordinates and intensity data for a set of points representing surfaces of one or more objects;

receiving a two-dimensional (2D) color image of the one or more objects acquired by a camera positioned at an approximately known second location and an approximately known orientation, the camera having an approximately known set of intrinsic optical parameters;

creating a 2D intensity image of the 3D point cloud using the intensity data by projecting the 3D point cloud onto a 2D image area from a view point of the second location based on the set of intrinsic optical parameters of the camera, the second location, and the orientation of the camera, the 2D image area having a size that is same as a size of the 2D color image; and

performing image matching between the 2D intensity image and the 2D color image to generate a set of refined camera parameters for projecting the 3D point cloud onto the 2D image area, wherein the set of refined camera parameters includes a translation parameter, a rotation parameter, and a scaling parameter.

13. The method of claim 12, wherein performing image matching comprises:

detecting a first set of edges of the one or more objects in the 2D intensity image to create a first edge image;

detecting a second set of edges of the one or more objects in the 2D color image to create a second edge image; and

estimating the translation parameter, the rotation parameter, and the scaling parameter such that, when applied to the first edge image, the first edge image matches with the second edge image.

14. The method of claim 12, wherein the set of intrinsic optical parameters of the camera includes one or more of a focal length, an optical center, a principal point, and distortion parameters.

15. A method of camera calibration with respect to a three-dimensional (3D) imaging device, the method comprising:

receiving a 3D point cloud acquired by the 3D imaging device, the 3D point cloud including 3D coordinates and intensity data for a set of points representing surfaces of one or more objects;

receiving a plurality of two-dimensional (2D) color images of the one or more objects acquired by a camera positioned at a plurality of corresponding locations and a plurality of corresponding orientations, the camera having an approximately known set of intrinsic optical parameters;

generating a plurality of sets of refined camera parameters by, for each respective 2D color image of the plurality of 2D color images;

creating a respective 2D intensity image of the 3D point cloud using the intensity data by projecting the 3D point cloud onto a 2D image area from a view point of the camera based on a respective location and respective orientation, and the set of intrinsic optical parameters of the camera, the 2D image area having a size that is same as a size of the 2D color image; and

performing image matching between the respective 2D intensity image and the respective 2D color image to generate a respective set of refined camera parameters for projecting the 3D point cloud onto the 2D image area; and

selecting one set of the refined camera parameters from the plurality of sets of refined camera parameters as a final set of refined camera parameters according to a predetermined criteria.

16. The method of claim 15, wherein each set of the plurality of sets of refined camera parameters includes a translation parameter, a rotation parameter, and a scaling parameter.

17. The method of claim 15, wherein the set of intrinsic optical parameters of the camera includes one or more of a focal length, an optical center, a principal point, and distortion parameters.

18. The method of claim 15, wherein selecting the one set of the refined camera parameters is performed using a clustering algorithm.

19. A method of colorizing a three-dimensional (3D) point cloud, the method comprising:

receiving the 3D point cloud, the 3D point cloud acquired by a 3D imaging device positioned at a first location and including 3D coordinates for a set of points representing surfaces of one or more objects;

receiving a two-dimensional (2D) color image of the one or more objects acquired by a camera positioned at a second location different from the first location and having an orientation;

receiving a set of camera parameters for projecting the 3D point cloud onto a 2D image area from a view point of the second location, the 2D image area having a size that is same as a size of the 2D color image;

projecting the set of points of the 3D point cloud onto the 2D image area using the set of camera parameters;

creating a depth buffer for the 3D point cloud, the depth buffer having an image area of a size that is same as the size of the 2D color image and a pixel size that is same as a pixel size of the 2D color image, the depth buffer including depth data for the set of points in the 3D point cloud as the set of points are projected onto pixels of the depth buffer using the set of camera parameters;

determining a foreground depth for each respective pixel of the depth buffer by detecting a closest point among a subset of the set of points corresponding to the respective pixel; and

coloring the point cloud by, for each respective point of the set of points;

comparing a depth of the respective point with a foreground depth of a corresponding pixel in the depth buffer;

upon determining that the depth of the respective point is within a predetermined threshold value from the foreground depth of the corresponding pixel, assigning a color of a corresponding pixel in the 2D color image to the respective point; and

upon determining that the depth of the respective point differs from the foreground depth of the corresponding pixel by an amount greater than the predetermined threshold value, not assigning any color to the respective point.

20. The method of claim 19, wherein creating the depth buffer, determining the foreground depth, and coloring the point cloud are performed using a graphics processing unit (GPU).

21. The method of claim 19, wherein determining the foreground depth is performed using a Mathematical Morphology algorithm.

22. The method of claim 19, wherein the 3D point cloud includes intensity data for the set of points, the camera has a set of intrinsic optical parameters, and the set of camera parameters is obtained by:

creating a 2D intensity image of the 3D point cloud using the intensity data by projecting the 3D point cloud onto a 2D image area from a view point of the second location based on the intrinsic optical parameters of the camera, the second location, and the orientation of the camera, the 2D image area having a size that is same as a size of the 2D color image; and

performing image matching between the 2D intensity image and the 2D color image to generate the set of camera parameters.