Backfilling Points in a Point Cloud

• US 20140132733A1
• Filed: 11/09/2012
• Published: 05/15/2014
• Est. Priority Date: 11/09/2012
• Status: Active Grant

• Current Assignee: The Boeing Co.
• Original Assignee: The Boeing Co.
• Inventors: Mundhenk, Terrell Nathan, Owechko, Yuri, Kim, Kyungnam
• Granted Patent: US 9,811,880 B2
• Time in Patent Office: Days
• US Class Current: 348/47
• CPC Class Codes: G06T 17/00 Three dimensional [3D] modelling, e.g. data description of 3D objects
• CPC Class Codes: G06T 2210/56 Particle system, point based geometry or rendering
• CPC Class Codes: G06T 3/4007 based on interpolation, e.g. bilinear interpolation image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: H04N 13/128 Adjusting depth or disparity
• CPC Class Codes: H04N 13/25 using two or more image sensors with different characteristics other than in their location or field of view, e.g. having different resolutions or colour pickup characteristics; using image signals from one sensor to control the characteristics of another sensor

Abstract

An apparatus, system, and method for increasing points in a point cloud. In one illustrative embodiment, a two-dimensional image of a scene and the point cloud of the scene are received. At least a portion of the points in the point cloud are mapped to the two-dimensional image to form transformed points. A fused data array is created using the two-dimensional image and the transformed points. New points for the point cloud are identified using the fused data array. The new points are added to the point cloud to form a new point cloud.

Claims

1. An apparatus comprising:

an image processing system configured to map at least a portion of points in a point cloud of a scene to a two-dimensional image of the scene to form transformed points;

create a fused data array using the two-dimensional image and the transformed points;

identify new points for the point cloud using the fused data array; and

add the new points to the point cloud to form a new point cloud.

2. The apparatus of claim 1, wherein the image processing system comprises:

a fusion manager configured to map the at least a portion of the points in the point cloud to the two-dimensional image to form the transformed points and configured to create the fused data array using the two-dimensional image and the transformed points.

3. The apparatus of claim 2, wherein the fused data array comprises matched elements associated with filled data vectors comprising non-null depth values and unmatched elements associated with unfilled data vectors comprising null depth values.

4. The apparatus of claim 3, wherein the image processing system further comprises:

a depth value generator configured to identify new depth values to replace at least a portion of the null depth values.

5. The apparatus of claim 4, wherein the image processing system further comprises:

a point cloud manager configured to identify the new points for the point cloud using the new depth values.

6. The apparatus of claim 5, wherein the point cloud manager is further configured to add the new points to the point cloud to form the new point cloud.

7. The apparatus of claim 1, wherein the fused data array is comprised of elements, wherein each of the elements is associated with a data vector comprising at least one of a pixel location, a depth value, and original pixel data.

8. The apparatus of claim 1 further comprising:

a first sensor system configured to generate the two-dimensional image; and

a second sensor system configured to generate the point cloud.

9. The apparatus of claim 8, wherein the first sensor system is a camera system and wherein the second sensor system is a light detection and ranging system.

10. The apparatus of claim 9, wherein the image processing system is configured to map the at least a portion of the points in the point cloud to the two-dimensional image using pose information for the camera system.

11. An image processing system comprising:

a fusion manager configured to map at least a portion of points in a point cloud of a scene to a two-dimensional image of the scene to form transformed points and configured to create a fused data array using the two-dimensional image and the transformed points,

wherein the fused data array includes matched elements associated with filled data vectors comprising non-null depth values and unmatched elements associated with unfilled data vectors comprising null depth values;

a depth value generator configured to identify new depth values to replace at least a portion of the null depth values; and

a point cloud manager configured to identify new points for the point cloud using the new depth values and add the new points to the point cloud to form a new point cloud.

12. A computer-implemented method for increasing a number of points in a point cloud, the computer-implemented method comprising:

receiving a two-dimensional image of a scene and the point cloud of the scene;

mapping at least a portion of the points in the point cloud to the two-dimensional image to form transformed points;

creating a fused data array using the two-dimensional image and the transformed points;

identifying new points for the point cloud using the fused data array; and

adding the new points to the point cloud to form a new point cloud.

13. The computer-implemented method of claim 12, wherein mapping the at least a portion of the points in the point cloud to the two-dimensional image to form the transformed points comprises:

identifying pose information for a camera system;

transforming a three-dimensional reference coordinate system for the point cloud to a three-dimensional camera-centric coordinate system using the pose information to identify camera-centric coordinates for the points in the point cloud; and

mapping the at least a portion of the points in the point cloud having the camera-centric coordinates to pixel locations in the two-dimensional image.

14. The computer-implemented method of claim 12, wherein creating the fused data array using the two-dimensional image and the transformed points comprises:

forming the fused data array, wherein the fused data array is comprised of elements having a one-to-one correspondence with pixels in the two-dimensional image; and

associating data vectors with the elements in the fused data array, wherein the data vectors include filled data vectors comprising non-null depth values and unfilled data vectors comprising null depth values.

15. The computer-implemented method of claim 14, wherein identifying the new points for the point cloud using the fused data array comprises:

generating new depth values to replace at least a portion of the null depth values; and

identifying the new points for the point cloud using the new depth values.

16. The computer-implemented method of claim 15, wherein generating the new depth values to replace the at least a portion of the null depth values comprises:

centering a window at a location of an element in the fused data array, wherein the element is associated with an unfilled data vector comprising a null depth value;

identifying support elements in a portion of the fused data array overlapped by the window; and

generating a new depth value to replace the null depth value using the support elements and a linear estimation algorithm.

17. The computer-implemented method of claim 16, wherein identifying the support elements in the portion of the fused data array overlapped by the window comprises:

determining whether matched elements in the portion of the fused data array overlapped by the window meet selected criteria;

responsive to the matched elements meeting the selected criteria, scoring each of the matched elements with a goodness score;

sorting the matched elements by the goodness score; and

selecting a portion of the matched elements as the support elements.

18. The computer-implemented method of claim 16 further comprising:

repeating the steps of centering the window at the location of the element in the fused data array, identifying the support elements in the portion of the fused data array overlapped by the window, and generating the new depth value to replace the null depth value using the support elements and the linear estimation algorithm for each location in the fused data array.

19. The computer-implemented method of claim 15, wherein identifying the new points for the point cloud using the new depth values comprises:

identifying camera-centric coordinates within a three-dimensional camera-centric coordinate system for the new points using the new depth values; and

transforming the camera-centric coordinates to point locations within a three-dimensional reference coordinate system for the point cloud to form the new points.

20. The computer-implemented method of claim 15, wherein generating the new depth values to replace the at least a portion of the null depth values comprises:

processing the fused data array using windows to generate the new depth values.