FUSION OF A 2D ELECTRO-OPTICAL IMAGE AND 3D POINT CLOUD DATA FOR SCENE INTERPRETATION AND REGISTRATION PERFORMANCE ASSESSMENT

• US 20100207936A1
• Filed: 02/13/2009
• Published: 08/19/2010
• Est. Priority Date: 02/13/2009
• Status: Active Grant

• Current Assignee: Harris Corporation (L3Harris Technologies, Inc.)
• Original Assignee: Harris Corporation (L3Harris Technologies, Inc.)
• Inventors: Pooley, Donald, Smith, Anthony O'Neil, Minear, Kathleen
• Granted Patent: US 8,179,393 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration

Abstract

Method and system for combining a 2D image with a 3D point cloud for improved visualization of a common scene as well as interpretation of the success of the registration process. The resulting fused data contains the combined information from the original 3D point cloud and the information from the 2D image. The original 3D point cloud data is color coded in accordance with a color map tagging process. By fusing data from different sensors, the resulting scene has several useful attributes relating to battle space awareness, target identification, change detection within a rendered scene, and determination of registration success.

Claims

1. A method for combining a 2D image with a 3D image for improved visualization of a common scene, comprising:

analyzing said 2D image to identify selected content based characteristics of a plurality of areas in said common scene;

selectively assigning to each of said plurality of areas a color map tag corresponding to a content based characteristic of said area;

selectively assigning a different color map to each of a plurality of areas of said 3D image in accordance with said color map tags;

forming a virtual 3D image from said 2D image by assigning a Z value to pixels in said 2D image based on a ground contour in said 3D image;

determining color values for points in the virtual 3D image based on a desired color map and creating a fused image by overlaying said 3D image and said virtual 3D image.

2. The method according to claim 1, further comprising evaluating a performance or quality of said registration step by visually inspecting said fused image to determine if features in said common scene are properly aligned.

3. The method according to claim 1, wherein said content based characteristics are selected from the group consisting of urban content, natural content, water content, and man-made structure content.

4. The method according to claim 3, wherein said man-made structure content is selected from the group consisting of buildings, houses, roadways, and vehicles.

5. The method according to claim 1, wherein said 3D image is comprised of a plurality of points comprising a 3D point cloud where each point is defined in accordance with an X, Y, and Z coordinate axis value, and said 2D image is comprised of a plurality of pixels having a position defined exclusively in accordance with values defined by said X and Y axis.

6. The method according to claim 5, further comprising assigning each said color map to one or more points having X, Y, and Z coordinate values within areas in said 3D image based on said color map tags assigned to corresponding X, Y coordinate values of said plurality of areas identified in said 2D image.

7. The method according to claim 1, further comprising removing a portion of said 3D image comprising ground surface data prior to said fusing step.

8. The method according to claim 5, wherein said step of assigning said Z values further comprises:

assigning for each 2D pixel having said X, Y coordinate value, assigning a Z value from the ground contour of the said 3D image that has the same X, Y coordinate value; and

interpolating or estimating a Z value if there is no data point in said 3D image that has the same X, Y coordinate value as a particular 2D pixel.

9. The method according to claim 1, wherein a plurality of said color maps are selected to mimic colors or hues that are commonly associated with said content based characteristic of the area for which the color map is used.

10. The method according to claim 1, further comprising registering said 2D image and said 3D image.

15. The system according to claim 1, wherein said 3D image is comprised of a plurality of points comprising a 3D point cloud where each point is defined in accordance with an X, Y, and Z coordinate axis value, and said 2D image is comprised of a plurality of pixels having a position defined exclusively in accordance with values defined by said X and Y axis.

16. The system according to claim 15, wherein said computer is programmed to assign each said color map to one or more points having X, Y, and Z coordinate values within areas in said 3D image based on said color map tags assigned to corresponding X, Y coordinate values of said plurality of areas identified in said 2D image.

18. The system according to claim 15, wherein said step of assigning said Z values further comprises:

assigning for each 2D pixel having said X, Y coordinate value, assigning a Z value from the ground contour of the said 3D image that has the same X, Y coordinate value; and

interpolating or estimating a Z value if there is no data point in said 3D image that has the same X, Y coordinate value as a particular 2D pixel.

11. A system for combining a 2D image with a 3D image for improved visualization of a common scene, comprising:

a computer programmed with a set of instructions for

analyzing said 2D image to identify selected content based characteristics of a plurality of areas in said common scene;

selectively assigning to each of said plurality of areas a color map tag corresponding to a content based characteristic of said area;

selectively assigning a different color map to each of a plurality of areas of said 3D image in accordance with said color map tags;

forming a virtual 3D image from said 2D image by assigning a Z value to pixels in said 2D image, each said Z value determined based on ground contour in said 3D image;

determining color values for point in the virtual 3D image based on a desired color map on a color values of a corresponding pixel in said 2D image; and

creating a fused image by overlaying said 3D image and said virtual 3D image.

12. The system according to claim 11, further comprising evaluating a performance or quality of said registration step by visually inspecting said fused image to determine if features in said common scene are properly aligned.

13. The system according to claim 11, wherein said content based characteristics are selected from the group consisting of urban content, natural content, water content, and man-made structure content.

14. The system according to claim 13, wherein said man-made structure content is selected from the group consisting of buildings, houses, roadways, and vehicles.

17. The system according to claim 11, wherein said computer is programmed to remove a portion of said 3D image comprising ground surface data prior to said fusing step.

19. The system according to claim 11, wherein a plurality of said color maps mimic colors or hues that are commonly associated with said content based characteristic of the area for which the color map is used.

20. The system according to claim 11, wherein said computer is programmed to register said 2D image and said 3D image.

21. A method for combining a 2D image with a 3D image for improved visualization of a common scene, comprising:

registering a 2D image with a 3D image;

forming a virtual 3D image from said 2D image by assigning a Z value to pixels in said 2D image based on a ground contour in said 3D image;

creating a fused image by overlaying said 3D image and said virtual 3D image in accordance with said registration.